According to the inscription, Huang San Temple was built in the early years of Hongwu in Ming Dynasty (1370). There are only two temples in the original version, and the scale is very small. During the Jiajing period of Ming Dynasty, Huangsantang was rebuilt and expanded, and the East-West Room, Bell and Drum Tower and White Pavilion were newly built.

According to "Rebuilding Temple Monument", in the fourth year of Qing Qianlong, the abbot Wang Duofang raised funds and built Sanqing Pavilion, Wenchang Pavilion, Pavilion, East and West Corners and Observation Deck in the backyard. The west turret corresponds to Wuyun Cave, the east turret corresponds to Yueyang Mountain, and the windows of the Bell and Drum Tower meet the Tatsumi Peak, forming a beautiful triangle.

Since the reign of Jiajing (1530), the affairs of Huang San Temple have been presided over by Taoist priests. According to the records of Taoist tombs in West Temple, it was the 20th generation by 1949, and Hu Lijun and Zhang Zongxian were the last monks to leave Huang San Temple.

After four years of Qianlong, it took Wang more than ten years to complete the construction of the third courtyard and the repair of the temple.

Huang San Temple was converted into a school in 195 1.

The Three Emperors Hall was demolished as a dangerous building in the late 1970s.

In recent years, with the support of town and village leaders, the three pavilions have been restored and the Three Emperors Hall has been rebuilt.