Hu Jia Garden was built by Hu Lian, an anti-Portuguese hero in Ming Dynasty, during Jiajing period. It has a history of 500 years and is famous for Hu's "three scholars in one door".

Wu Cheng'en, the author of The Journey to the West and nephew of Hu Lian, praised Hu Jia Garden as "the first noble house in Changhuai" and later called it "the first garden in Huaihai".

In the early years of Emperor Kangxi in Qing Dynasty, the garden was spread to Hu Jianjing, a descendant of Hu Lian and a mentor of Emperor Kangxi, and was expanded again, with many exotic flowers and grasses newly planted, which was quite garden-like.