Hoan Kiem Lake was already considered the most beautiful lake in Ha Noi when Ngoc Son Temple was built on an island in the 19th century. Initially, the temple was called Ngoc Son Pagoda and was later renamed Ngoc Son Temple, since temples are dedicated to saints.



Saint Van Xuong, considered to be one of the brightest stars in Viet Nam's literary and intellectual circles, was worshipped there. National hero Tran Hung Dao is also worshipped after he led the Vietnamese people to victory over the Yuan aggressors.
The temple as it is today is the result of renovations made by Nguyen Van Sieu in 1864. A Confucian scholar, Nguyen Van Sieu had a large pen-shaped tower built at the entrance of the temple. On the upper section of the tower, also called Thap But, are three Chinese characters Ta Thanh Thien, which literally means "to write on the blue sky" is to imply the height of a genuine and righteous person's determination and will; Dai Nghien, meaning "ink stand", is carved from stone resembling a peach placed on the back of the three frogs on top of the gate to the temple; and The Huc, meaning "where rays of morning sunshine touch".

On the way to the temple there are several parallel sentences (cau doi), written on the walls. These cau doi were part of traditional word puzzles played by educated individuals.