"All of a sudden, these young western people were dancing down Oxford Street, looking very zany, bit crazy perhaps, singing with their full hearts, worshipping God and thinking they're going to change the world…it was an alien beginning," says devotee Gauri Das.
That Bhaktivedanta Manor became so popular was due largely to the influx of ethnic Indians expelled from east Africa who made their home in north London during the early 1970s.
Kripamoya Das says many found solace in visiting the manor: "All those people who didn't have a place to worship now discovered they could have a place to worship, just a few miles into the country.
"They were refugees, and how wonderful it was for them to see the compassion of Lord Krishna in this beautiful shrine when they were in exile - when they had lost everything."
The growth in popularity resulted in new problems, with complaints about noise leading to the closure of the temple for public worship in 1994.
A lengthy legal battle against Hertsmere Borough Council culminated with more than 30,000 Hindus marching on Parliament, with Bhaktivedanta Manor finally told it was allowed to continue worship in 1996.
About 40 monks now live on the 78-acre site, which includes a theological college, organic farm, herb gardens, a cow sanctuary, primary school and kitchens providing meals for the homeless.
So what is life like at Bhaktivedanta Manor?
"All the monastics living at the temple get up at 3.30 am, there's a morning ceremony of lamps, and they sing devotional songs and meditate, then we have a philosophical lecture with a vegetarian meal, then it's a working day for everyone" says Gauri Das.
"Around nine o'clock all the school children who study here arrive, along with coaches from visiting schools who come to see this place as a devotional centre of excellence."
The farm is run by volunteers with about 200 devotees preparing for major Hindu festivals which attract thousands of worshippers.
Hare Krishna is also known as the "kitchen religion" in that all visitors are offered a meal.
Each day, 1,000 meals are prepared at Bhaktivedanta and taken to central London to feed the homeless.
On festival days, volunteers will prepare more than 30,000 meals.
The temple has attracted attracted high profile guests including Boy George, Chrissie Hynde , Russell Brand and Prime Minister David Cameron.
And 40 years after it opened, the spiritual retreat once viewed by critics as a hippie commune, now hosts Janmashtami, the largest Hindu festival in Europe, which marks the birth of Krishna, a popular God in the Hindu Pantheon.
"I think its part of the British fabric now," says Gauri Das.
"We are considered to be a major influence within the Hindu community in the UK."