A key Indigenous leader has issued a grave warning on the Indigenous Voice to parliament as the Coalition steps up its calls for more details.

Also on Monday, shadow attorney-general Julian Leeser threatened to withdraw his support for the Voice unless more details were forthcoming.

“The government is in danger of losing me because I just don’t think they’re listening,” Mr Leeser told ABC radio.

“I’m really trying to get them to listen to the reasonable concerns that people are raising.”

But founder of the Cape York Institute and From the Heart Advisory Group member Noel Pearson called the demand for detail a concerning diversion.

Mr Pearson said detail would be in legislation made by the parliament but the referendum was about the constitution.

“It’s the parliamentarians who have the responsibility to come up with the detail,” he told ABC Radio.

“The Australian people are being asked to vote on a constitutional amendment. All of the power lies with the parliament and it’s a complete diversion to demand these details.”

“I think that Julian Leeser [and Opposition Leader] Peter Dutton … may be just choosing to play a spoiling game. I hope they are not.

“We’ve got to understand what is it stake here. What is at stake is the chance for referendum, and if this referendum is kiboshed through game playing and a spoiling game by the opposition, we will lose the opportunity forever.”

Tweet from @sunriseon7

Prime Minister Anthony Mr Albanese said details on the question were released last July and he had personally met Mr Dutton to discuss the issue more than five times.

“I released what Australians will potentially be voting for, a draft question and draft constitutional change,” he told ABC News.

“There hasn’t been any suggested changes to that draft from the Coalition and they have had more than six months.”

Mr Albanese said the government would not push through legislation to establish the voice to parliament if the referendum failed. But he said he was focused on ensuring its success.

Mr Albanese said following a successful referendum – to be held as soon as August – parliament would decide how the advisory group would operate.

“The voice won’t be a funding body, it won’t be an administrative body … it will simply be an advisory group,” he told Seven’s Sunrise.

“Its advice doesn’t have to be taken but it does have to be heard.”

Meanwhile, senior cabinet minister Tanya Plibersek cautioned Australians not to confuse the status of the proposed Indigenous voice to parliament.

“This is a voice, not a veto,” she told Seven.

“It will be a chance for Australians to talk about the sort of nation we want to be in the future … and get more practical delivery of healthcare and education and so on for First Nations Australians.”

Ms Plibersek said the conversation about the proposal was well underway and a picture of what it might look like was gathering detail.

The National Party confirmed last year it would not support the referendum, a move that spurred MP Andrew Gee to leave the party and join the crossbench.

Nationals Leader David Littleproud said his party did not have malicious intent in not supporting the voice but believed it would add more bureaucracy to closing the gap.

“We have members of parliament that actually have lived experience on these communities … the biggest disadvantage is in rural and remote areas,” he told Sky News.

“What we’re saying is another layer of bureaucracy will not change this.”

– with AAP