Shadow Attorney-General Julian Leeser has announced he will quit the frontbench over concerns with Peter Dutton’s stance on the Indigenous Voice to Parliament.

Mr Leeser had worked with Indigenous leaders such as Pat Dodson on an earlier version of the Voice to Parliament over the past decade and had been conspicuously absent when the Opposition Leader announced the Liberal Party position on the referendum.

“I believe the time for the Voice has come,” Mr Leeser told reporters in Sydney.

“The Voice is not about special privileges.

“It’s about recognising that there are indigenous brothers and sisters, and it is about recognising that indigenous Australians are brothers and sisters, and we have left them behind in our shared national project.”

Mr Leeser said his position was not a reflection on Opposition Leader Peter Dutton’s leadership.

“I’m resigning without rancour but on a point of principle,” he said.

“I want to be able to say to my children that [their] father stood up for something that he believes in.

“That’s really important and that’s what all of us as parliamentarians should do.”

Mr Leeser said he would campaign for changes to the referendum wording to be put to the Australian people to ensure it was “on a surer footing” with conservatives wary of constitutional change.

But he said he would ultimately support the ‘yes’ case in any case.

“I will support the referendum being put this year,” he said.

“We need to find common ground as Australians.

“No great nation has ever been built by dividing good nations engaged in the search for a common purpose and common ground.”

Shortly before the release of the government’s Voice proposal, Mr Leeser gave a speech to the Press Club arguing against the wording of the proposed amendment to the constitution, which the government will put to the Australian people in the referendum scheduled for the latter stages of this year.

He said on Tuesday he would continue to pursue the model he argued for at the Press Club in order to assuage doubts held by constitutional conservatives and bring them over to the ‘yes’ campaign.

“I’m looking closely at proposed amendments to the government’s model,” he said.

“That’s why I’ve put forward the Press Club, a model that provides a pathway for the funding of local and regional bodies, a model that seeks to amend the government’s proposal in order to give the referendum the best chance of success.

“But ultimately, I will support the referendum because I’m a supporter of the Voice.”

Mr Leeser’s decision comes after former Indigenous Australians minister Ken Wyatt – a member of the referendum working group – quit the Liberal Party in outrage at its decision to officially oppose the Voice to parliament. Outspoken backbench MP Bridget Archer has said publicly she has also considered splitting from the party.

At least three leading Liberal moderates – Simon Birmingham, Paul Fletcher and Marise Payne – also reportedly spoke against the idea at last week’s snap partyroom meeting.

Mr Dutton confirmed last week that the Liberals would argue against an Indigenous voice to advise executive government, which will be enshrined in the constitution if this year’s referendum succeeds. They have instead called for a legislated regional and local voice.

Liberal shadow cabinet ministers are bound to the party position and a free vote has not been granted. Backbenchers are not tied to the decision.

Several senior Liberals are arguing for people to be able to campaign according to their conscience, as was the case in the republic referendum.