Named in 1838 by Governor Gawler after HMS Victor, commanded by Captain Richard Crozier, who surveyed the area in 1837. That same year a whaling station was established on Granite Island, managed by Captain Blenkinsopp. The first ship to load at Victor Harbor was the Goshawk taking on a cargo of whale oil in 1838. Although one of the very first harbours in South Australia, the town did not come into being until 1863 when it was surveyed as a private town by L.J. Hyndman.
During the early days of settlement, Victor Harbour was considered as the site for the colony's capital by several of its influential citizens, including Governor Hindmarsh. As an ex navy man Hindmarsh was anxious that sailors should report any parts of the coastline which might offer protection for ships.
In 1838 it was reported that the land was extremely rich, and the site most picturesque, and well calculated for a town. It was bounded by two rivers from seventeen to thirty metres wide, and navigable for boats three to five kilometres. We consider this site the most eligible that we have seen so far in the colony for the first town. However six months later another report stated that the plan for a proposed town was utterly useless and absurd.
The first thirty-four settlers arrived with the Rev Ridgeway W. Newland in 1839 and settled at Yelki, near the Bluff. Newland was regarded as a man of good standing and character. Life was very hard for these early pioneers and they had to overcome many problems. They were forced to live in tents for nearly two years before the first houses were built. Land for farming, covered with giant blue gums, was hard to clear. As early as 1840 Lutheran Missionary H. Meyer had established a school for the local Aborigines, to give them some European Education'. He was later transferred to Bethany in the Barossa Valley.
During the early 1840s, Newland cultivated his land with the help of his family and some Aborigines. They ploughed, sowed and reaped and had made enough progress for the Adelaide Observer to conclude that the Aboriginal race was capable of a high degree of civilised life.
From its early days the town had close connections with Goolwa and the River Murray. After 1850 river steamers carried wool and wheat up and down the river to Goolwa but could not make it through the river mouth to the sea. Instead goods had to be transported to the nearest sea port which was Victor Harbor.
Port facilities created employment with many workers needed to load and unload the cargo from ships, trains or bullock wagons. Once there was a small community other services followed rapidly. Soon there were the usual churches, hotel, school, post office and police station. In August 1863 two bridges, one over the Hindmarsh and the other across the Inman River, were opened making it much easier for people to visit the town.
During that year several stone houses were built and a year later a telegraph station and large railway sheds to cater for the traffic on the original horse drawn railway. With increasing traffic a new jetty and a breakwater were built but when the town of Morgan was connected by rail to Adelaide in 1880, Victor Harbor ceased to be a port.
Even so, Victor Harbor continued to grow despite the loss of the river trade. With the hinterland now well established, farmers and graziers came to Victor to buy or sell their goods. When connected by rail to Adelaide the town and harbour became a tourist attraction which has kept on growing to such an extent that today Victor Harbor is one of the major tourist destinations in South Australia. Several well to do Adelaide families built their summer residence at Victor Harbor. The two best known are Mount Breckan and Adare.