Lavender Bay was named after the boatswain (bosun) George Lavender from the prison hulk Phoenix, which was moored there for many years. The bay is dual-named Gooweebahree, (also sometimes written as Quiberee) in the Dharug language of the local inhabitants, the Cammeraygal people of the Eora nation. The colonists also called it Hulk Bay and sometimes Phoenix Bay. George Lavender lived on 14 acres (5.7ha) adjacent to the property of Billy Blue.
In the 2021 Census, there were 870 people in Lavender Bay. 59.4% of people were born in Australia and 75.5% of people spoke only English at home. The most common responses for religion were No Religion 41.6% and Catholic 23.9%.
Landmarks
A wharf is located in the bay which provides access to private vessels.
The Lavender Bay Baths (1910), also known as Cavill's Baths, a tidal swimming pool, were once popular with swimmers, located in the area beside the ferry wharf.Wendy's Secret Garden
Wendy Whiteley created a garden in 1992 adjacent to her home, in an overgrown railway siding between Clark Park and the old railway line. Known as Wendy's Whiteley's Secret Garden, it is now operated by a trust and is open to the public.
Cricketer Don Bradman lived in the harbourfront Bay View Street, and was one of the first few Australians to get a private telephone number while living in Bay View Street.
Artist Norman Lindsay lived at 'Heidelberg' at 9 Bay View Street
Artist Brett Whiteley lived in a house overlooking the Bay with his wife Wendy Whiteley; Wendy continued to live there as of 2019[update]. In 2018 the NSW Government granted heritage status to the Whiteleys' house, its views of Sydney Harbour, and the Secret Garden.
Artist Peter Kingston lived at 3 Walker Street. Many of his most well known artworks depict the view of the Bay from his home.