Opposition Leader Peter Dutton denies the resignation of his spokesman on the Indigenous Voice to Parliament is a sign of wider dissatisfaction in the Liberal party room.

Mr Dutton retained his composure while fielding questions about shadow attorney-general Julian Leeser’s departure, which he said had been driven by Mr Leeser’s personal involvement for more than a decade in developing a model for constitutional recognition.

But Mr Leeser’s bombshell announcement on Tuesday that he will quit the Coalition frontbench will increase the pressure on three other Liberal moderates who are also understood to have reservations about the party’s official opposition to the Voice, announced by Mr Dutton last week.

“If anybody in this debate is dividing the country, it’s [Prime Minister] Anthony Albanese,” Mr Dutton, who emerged two hours later to react to Mr Leeser’s announcement, said.

“Julian’s position in this is pretty unique.

“I’m very happy with the position that we’ve got.

“I believe that represents the overwhelming view of our party room.”

Liberal senator Andrew Bragg said in a statement following Mr Leeser’s resignation:

“Julian Leeser has done more than any other constitutional conservative to advance the Indigenous Voice.

“This referendum is too important to play politics and it is not good enough to oppose the referendum on process grounds.

“I want the referendum to be successful and I believe a yes vote is now more likely because of Julian’s conviction.”

Five Liberal MPs, including Senator Bragg, spoke out against Mr Dutton’s opposition to the Voice, and his intention to actively campaign against the referendum, when it was carried by the Liberal party room last week.

A further three members of the frontbench are understood to have been opposed: Paul Fletcher, Marise Payne and Simon Birmingham.

Mr Leeser quit the frontbench, although he will remain in the Liberal Party. He said his position was not a reflection on Mr Dutton’s leadership.

“I’m resigning without rancour but on a point of principle,” he said.

“I want to be able to say to my children that [their] father stood up for something that he believes in.

“That’s really important and that’s what all of us as parliamentarians should do.”

Mr Leeser said he would campaign for changes to the referendum wording to be put to the Australian people, to ensure it was “on a surer footing” with conservatives wary of constitutional change.

But he said he would ultimately support the ‘yes’ case in any case.

“I will support the referendum being put this year,” he said.

“We need to find common ground as Australians.

“No great nation has ever been built by dividing good nations engaged in the search for a common purpose and common ground.”

Labor figures have welcomed Mr Leeser’s move. Foreign Minister Penny Wong said Mr Dutton’s decision to campaign against the Voice was disappointing.

“Julian Leeser has shown real strength today. He put his principles ahead of partisan politics and we welcome that,” she said.