Prime Minister Anthony Albanese has hit back after the opposition lashed the Indigenous Voice to Parliament as a “re-election vanity project” for Labor.

Mr Albanese said Australians would make their own judgment on whether the Coalition was being constructive on the Voice.

In a speech to the Liberal Party’s Western Australia branch on Friday, deputy opposition leader Sussan Ley labelled it a vanity project for Mr Albanese, and nothing more than a “political wedge”.

“Anthony Albanese would rather see the Liberal Party say no and this referendum fail, than the Liberal Party say yes and this referendum succeed,” she said.

“[He] has tied constitutional recognition of our first Australians, which everyone across the parliament supports, to a concept called the Voice, which he cannot explain.

“Sadly, I suspect he is using this referendum as a platform for an early election.”

Opposition Leader Peter Dutton is yet to reveal what position the Liberal Party will take, and has repeatedly raised concerns about a lack of detail.

He met Indigenous groups while visiting the Northern Territory this week. Mr Dutton said stakeholders he met on Thursday backed the Voice but he wanted more details from the government to ensure it would help rather than hinder Indigenous communities.

“We want that detail before we arrive at a position,” he told Sky News.

But Mr Albanese said he’d had six meetings with Mr Dutton on the issue – and accused the opposition of using complaints about a lack of detail to sow division.

“They have an opportunity to ask questions about detail. That does not happen,” he said.

“It is not a right of veto. It won’t be a funding body, it won’t run programs. They know that that’s the case. Why do they know it? Because actually a lot of this process has occurred on their watch.

“In 2017, they were the government. They then set up a parliamentary committee … It came to a unanimous parliamentary committee report. They then had, under [former minister] Ken Wyatt, a process where they produced a 270-page document suggesting options.

“That went to cabinet twice. They chose not to progress that at all. I am progressing it. I am being constructive. People will look at those comments and draw their own conclusions.”

Indigenous lawyer and academic Noel Pearson said the referendum was a “once in a nation’s lifetime” opportunity for reconciliation.

“We will be crying about this matter for another 200 years if we don’t do it this year,” he told Sky.

“Constitutional recognition is about the words in the constitution … that enable Indigenous voices to be able to speak to the parliament, speak to the government, about the policies and laws that affect people.”

Meanwhile, a $5 million donation has kick-started the launch of the yes campaign for the Voice.

Hundreds of advocates and volunteers drawn from faith-based groups to business organisations this week gathered for workshops on how they could win the referendum, which will be held between October and December.

Yes campaign director, Quandamooka man Dean Parkin, said the campaign would start taking the conversation to people to bring the country together for a successful yes vote.

Australians for Indigenous Constitutional Recognition co-chair Rachel Perkins said the “campaign now has tremendous momentum as we head towards a referendum later this year that offers a chance for a moment of national unity”.

Paul Ramsay Foundation director and Kuku Yalanji woman Natalie Walker said the foundation aimed to make a “lasting contribution” to positive social change, with the voice a “critical step” in doing that.

The Nationals have announced they will formally oppose the voice, despite a split within the party over the decision.

Northern Territory CLP Senator Jacinta Nampijinpa Price will head a no campaign funded by the conservative lobby group Advance Australia.

– with AAP