Finance Minister Simon Birmingham has urged members of parliament to listen to expert medical advice, as controversy swirls over rogue MP Craig Kelly’s appearance on a podcast hosted by disgraced celebrity chef and COVID sceptic Pete Evans.

Australia’s peak body representing GPs called Mr Kelly’s latest stunt “very unhelpful”.

Labor leader Anthony Albanese has described Mr Kelly’s constant discredited claims about the coronavirus and health responses as “dangerous”, while Nationals MP Barnaby Joyce said people shouldn’t listen to the backbencher.

“If you’re going to take medical advice from Craig, you may as well take it from me,” Mr Joyce told Sky News.

Mr Kelly gave an interview for Mr Evans’ podcast on Monday. Mr Evans, who has been removed from Spotify and banned from Facebook for “repeated violations” of misinformation policies, was dropped from multiple commercial arrangements and a book deal in November for sharing a meme featuring a neo-Nazi symbol.

Mr Evans called Mr Kelly a “beautiful and beyond courageous man” and a “true hero”, after a 90-minute conversation, to be published on Tuesday. Mr Kelly defended his appearance with Mr Evans, telling The New Daily in an interview “you’ve got to be able to talk to people you don’t agree with”.

“I’m sure many people would think ‘I don’t agree with what The New Daily says, why the hell are you speaking to The New Daily?’” Mr Kelly said.

• Related: ‘Enough is enough’: Constituents fight to oust rogue Liberal MP Craig Kelly

Dr Karen Price, president of the Royal Australian College of General Practitioners, said Mr Kelly’s appearance with Mr Evans was disappointing and called for elected politicians to be more responsible.

“It is unacceptable that Craig Kelly is persisting in disseminating misinformation concerning COVID-19 and to appear on this podcast with a disgraced former celebrity chef is very unhelpful,” she said.

“We are still in the middle of a pandemic that is leaving many people feeling understandably anxious and exhausted. Those like Pete Evans are preying on that anxiety to peddle false information and he should not be enabled by anyone, especially a federal member of Parliament.

It is unacceptable that Craig Kelly is persisting in disseminating misinformation concerning #COVID19 and to appear on this podcast with Pete Evans is very unhelpful. As @RACGP President I urge all public figures to act responsibly.https://t.co/auVILwGlI2

— Royal Australian College of GPs (RACGP) President (@RACGPPresident) February 1, 2021

Dr Price claimed Mr Evans had “spread alarming misinformation, including content linked to the Qanon conspiracy theory and COVID-19 deniallist claims”.

In the Coalition’s regular partyroom meeting on Tuesday morning, Mr Kelly took the floor to defend himself against political criticism. He again tried to make the case for hydroxychloroquine, an anti-malarial drug which chief medical officer Professor Paul Kelly and the Therapeutic Goods Administratio have been categorically refuted as possible COVID treatments.

Labor’s shadow health minister, Mark Butler, moved a motion condemning Mr Kelly in the House of Representatives immediately after parliament resumed on Tuesday. The motion was gagged by the government and ultimately failed by a small margin.

Speaking outside a Canberra church on Tuesday morning, following a service to mark the beginning of Parliament’s new term, Mr Albanese took aim at the rogue backbencher.

“Craig Kelly’s actions are dangerous. And what worries me isn’t just that his comments are dangerous, that they endanger people’s health, that those mixed messages will undermine our response to COVID,” he said.

“Yesterday [Monday] at the National Press Club, the Prime Minister was asked a very clear question to distance himself from Craig Kelly, and he said he was doing a fantastic job as the Member for Hughes. Craig Kelly is not doing a fantastic job. Craig Kelly is doing a dangerous job. Craig Kelly is a problem.”

Mr Morrison had said on Monday that Mr Kelly was “not my doctor”, and told people “don’t go to Facebook to find out about the vaccine. Go to official government websites”.

Senator Birmingham, the government’s Senate leader, took a slightly harder line than the PM, specifically saying his fellow MPs should listen to experts.

“My message to everyone, be you a member of the public or a member of parliament, stick to the health advice, the expert medical health advice. That’s what we should all be doing,” Senator Birmingham said on Tuesday.

When asked if elected officials should be associating with Evans, Senator Birmingham said he would “urge everyone to associate themselves with [chief medical officer] Professor Paul Kelly, with our health experts.”

“They’re the advice everybody should be relying on. whether you’re a member of the public, the media or the parliament,” he said.

Speaking on Sky, Mr Joyce was also circumspect.

“Craig’s allowed to say what Craig wants. The people of the electorate of Hughes have the capacity… it’s their judgment, not ours,” he said.

“I agree with something the Prime Minister said – Craig is not a doctor.”

Another Labor MP, Tim Watts, said Mr Kelly’s Facebook activity should be taken seriously.

“This is not a trivial issue. It’s not something that should be laughed off,” he said.

“Unfortunately, we’ve seen in the United States recently, what happens when political leaders turn the other cheek and ignore conspiracy theories and extremism online. They become emboldened.”

Mr Watts, Labor’s shadow assistant minister for cybersecurity, said he welcomed Evans being deleted from Facebook and added “everybody has an obligation during this pandemic to play their part to promote good quality information for the Australian public”.

Despite a growing community campaign to oust him from his seat of Hughes, Mr Kelly told TND that he looked forward to running as the Liberal candidate at the next election.