Alexander Hay bought a block of land, near the Hindmarsh River in Victor Harbor, of some 160 acres in the early 1870s. In 1875 he stocked it with sheep and later, on suggestions of his second wife Agnes Gosse, decided to build a house on it which would overlook not only the land but also the sea. William McMinn, of Overland Telegraph fame, was contracted to to design the house. McMinn also designed Prince Alfred College, the General Post Office, Adelaide Children's Hospital, Marble Hill and Adelaide University. The foundations for Mount Breckan were laid in April 1879 and early in 1881 Alexander Hay and his family moved in.
The main building consisted of a great hall, measuring 11 x 16 metres, 22 rooms, including seven bedrooms, master bedroom, dining room, drawing room, ball room, bath rooms, halls, library, school room, vestibules, dressing rooms, a large staircase and a 25 metre high tower. Outside there was a tennis court and several other buildings some of which had been erected to house the construction workers on site.
In 1908, ten years after the death of Alexander Hay, Mount Breckan was destroyed by fire. The insurance policy only covered a very small amount of the damage In July 1909 Mrs Hay and a daughter were lost in the ill fated Waratah on a voyage to England. Subsequently the property was acquired by W.F. Connell and rebuilt as the Mount Breckan Club, a high class guest house. By 1913 it had 38 rooms and a golf course. At the same time the estate became smaller and smaller as a total of 57 housing blocks were carved from it and sold.
The Commonwealth Government bought Mount Breckan for the use as a training and rehabilitation centre for the Air Force. From 1962-1978 the building was used as a Bible College with as many as a hundred students attending each year.
In 1996 the property was bought by the Baron and Baroness of Ballintober who have since restored it to its former glory. Today the main part of the building is used as their private residence but an accommodation and function complex has been established to cover the maintenance of the building.