Cabo Raj Bhavan stands at the confluence of the rivers Mandovi and Zuari at the top of a cliff. It lies at the tip of a peninsula near Dona Paula, nine kilometres from Panjim. An imposing structure, it was built in 1540 and was tasked with the defence of the Goa harbour. When completed it was large enough to house the entire population of the citadel. It lies in ruins today and what remains of it are just three large cisterns. Before the fort was erected, a chapel dedicated to the Blessed Virgin Mary and a convent to the Franciscans were built.

Later, this came to be the official residence of the Governor of Goa. It is one of the most elegant of the Raj Bhavans in India and special prior appointment is required to visit here.