Simon Birmingham, the opposition foreign affairs spokesman, said he wouldn’t quit the Coalition frontbench but it was “equally not” his intention to campaign for the “no” case.

The South Australian senator is a key moderate Liberal, who – as a frontbencher – is bound to the party position on the Voice. Despite that, he told Sky News on Wednesday afternoon he had no intention of following shadow attorney-general and Indigenous Australians spokesman Julian Leeser and quitting shadow cabinet.

He is the latest Liberal MP to take a different view of the Voice to Opposition Leader Peter Dutton, who announced the party’s formal opposition last week – along with his intention to campaign against it.

On Tuesday, One Nation leader Pauline Hanson called for Birmingham to step down if he could “not provide leadership the Coalition must have to effectively oppose the Voice”.

Leeser quit shadow cabinet on Tuesday, and moved to the backbench, so he can support the yes campaign on the Voice.

He remains a member of the Liberal Party, and defended Mr Dutton publicly on Wednesday – despite his decision.

“The Liberal Party is not in tatters. I support the leadership of Peter Dutton, I support the party – we just have a difference of opinion on this issue,” Leeser told Today on Wednesday.

“I’ve been a very long supporter of the Voice, I’ve been there since its early creation, I set up an organisation to encourage people to support the Voice before I was a member of parliament.

“I felt as a matter of conscience and as a matter of my own ethics – that you’ve got to stand for something, even if it costs you – that I needed to resign, to take a different position.”

He said he would work with the government to ensure the best chances of success by bringing more conservatives onside, saying polls putting support for the Voice at slightly more than 50 per cent were not good enough.

Leeser will also push to amend Prime Minister Anthony Albanese’s proposed wording but will support the Voice being embedded in the constitution regardless.

“The Voice will make a positive difference in people’s lives and the downside risk is small,” he told Sky News on Wednesday.

Moderate Liberal MP Bridget Archer said Leeser’s stance as an “authentic conservative” would help other conservative-minded people vote ‘yes’.

“This is an issue that should transcend the political cut and thrust,” she told ABC TV.

“[Leeser] campaigning actively for a yes vote will be enormously helpful, particularly to Liberal-minded or conservative-minded voters who are considering what they are going to do with the referendum.”

Andrew Bragg, another Liberal moderate, said the yes campaign had been bolstered by Leeser’s flip.

Leeser will push to strip out the second proposed clause referencing representations to the parliament and executive government in a bid to allay the concerns of Australians worried about the voice overreaching.

“The way it’s drafted currently has some risk and it’s better to remove that to encourage more Australians to vote for it,” he said.

But senior Labor minister Penny Wong said the clause would ensure Indigenous people had input into policy that affected them.

“If we look to our history, much of what we would now say were some of the wrong decisions that governments took in relation to First Nations peoples and communities were taken by executive governments,” she told Sky News.

Deputy Liberal leader Sussan Ley said the party would continue to push for a focus on local, community-led efforts instead of a national voice in the constitution.

“We believe in a local and regional voice because we want this to be, if you’d like, bottom-up, not top-down,” she told Seven’s Sunrise.

Ley would not speculate on Leeser’s replacement.

Victorian conservative Michael Sukkar and NSW moderate Paul Fletcher have been touted for the shadow attorney-general portfolio. Fletcher is understood to have been one of several so-called “Liberal moderates” – including Birmingham – who objected last week to Dutton’s decision to formally campaign for a no vote.

Former Liberal Indigenous Australians minister Ken Wyatt, the first Aboriginal person elected to the House of Representatives, hit back at opposition claims the voice would be “elitist” and Canberra-focused.

“It’s not a Canberra voice. It’s not elite. It’s people from the grassroots,” he said.

– The New Daily with AAP