John Setka has warned he will hit Anthony Albanese with a legal challenge if he tries to expel him from the party and is threatening to withdraw millions of dollars in funding.

The Construction Forestry Maritime Mining and Energy union leader believes his threat at a recent meeting to withdraw funding from the ALP is the real reason the Labor Party is trying to boot him from the party and the union.

It’s the same meeting he is accused of disparaging anti-violence campaigner Rosie Batty – a claim he denies.

“What I said was no more money to the ALP. We are freezing everything. Not one more cent,” he told The New Daily.

“The $12 million the ACTU spent, they might as well have gone down the racetrack and gone to the Crown casino and got a better return. It’s pretty bad.

“They f—-d it all. Their policies. Everyone tip-toed around and did everything we had to do. Millions and millions of dollars. And we are in the gun now.”

The Victorian branch of the CFMMEU that Mr Setka leads would have conservatively donated $4 million to the ALP and the ACTU to fight the election over the past three years.

Mr Setka said that if Mr Albanese tried to expel him from the party without due process when the ALP executive meets in July he will take him on in the courts.

Labor sources have backed Mr Setka’s view that there is plenty of case law to support a legal challenge.

“Look, my view is I will challenge it. It’s up to the members because it would be a costly exercise,” Mr Setka said.

“I am a little bit old school, I actually think you should have actually done something before you get cooked on a spit about it.

“The bottom line is I am for fair play. They’ve accused me of bagging Rosie Batty. Now, it’s clear that wasn’t said. Well, good luck to them. It’s going to be a long, drawn-out thing.”

John Setka denies denigrating Australian of the Year Rosie Batty. Photo: AAP

Mr Albanese has cited his comments about Ms Batty as the reason for his expulsion, along with previous controversies.

But on Friday, the ACTU’s Sally McManus conceded that while Mr Setka did mention Ms Batty during a discussion of men and family violence laws that he did not disparage her.

Sources also claim the view that he did not criticise Ms Batty was backed by CFMMEU leaders Michael O’Connor and Tony Maher, who have remained silent on the matter.

“Right now, without sounding like a sook, because Albanese is going to expel me, look, I would say no, we are not going to give them a cent,” Mr Setka said.

“I said that at the start of that meeting, we are not giving one more cent to the ALP. That’s a big threat to a lot of people. Even in our national office.

“That was the first item. Our union and the ACTU invested so much money in that ReachTel. Gee, how good was that? So, when someone gets up and says we are not giving them any more money then it starts the whole crescendo.

“Why should we give them any more money? That’s the bigger threat. So, you’ve got to read between the lines.”

Mr Setka is separately facing criminal charges for using a carriage service to harass a woman. He has previously indicated he will plead guilty to sending the text messages, which included calling the woman “a dog” and a drunk.

“I am a pretty mentally tough person. A lot of the hypocrites bagging me have done 10 times worse,” he said.

“My lieutenants now, after what Albanese and Sally McManus said about me standing down from the union, we’re not giving them anything.”

However, Mr Setka said he was not proposing to pull out of the ACTU.

“We actually pay them money. So, pretty rude of them to say I should go. I wouldn’t pull out of the ACTU. Sally McManus is a really good person. She’s a committed unionist. I just think she listens to the ALP hacks too much,” he said.

The Electrical Trades Union, which is backing Mr Setka, has also fallen out with the ALP previously.

In 2010 the union fell out with Kevin Rudd and subsequently donated $780,766 to the Australian Greens Victoria branch for Adam Bandt’s campaigns since 2010.

After coming back into the fold, the ETU Victorian branch alone then donated $1 million to the ALP last year and $200,000 to the Greens.

John Setka and his wife Emma Walters speak to media in Melbourne last week. Photo: AAP

The last time the ALP tried to boot union leaders from the party was under Kevin Rudd’s leadership.

In 2007, he expelled the CFMEU’s Joe McDonald but the expulsion proved a protracted affair before he ultimately rejoined the ALP.

Two years later, then as Prime Minister, Mr Rudd came after the Tasmanian Electrical Trades Union’s Kevin Harkins.

Mr Rudd later privately confessed that “he got his Kevins mixed up” when he publicly attacked Kevin Harkins and meant to refer to West Australian unionist, the CFMEU’s Kevin Reynolds.

In 2009, he described Mr Harkins as a “well-known pugilist”, which left Tasmanians baffled because he had no history of violence.

On Sunday, Australian Workers Union national secretary Daniel Walton joined ACTU secretary Sally McManus in calling for Mr Setka to resign.

“When you find yourself charged with a substantial number of domestic violence matters, when there are a number of allegations in terms of comments, some of which are fiercely fought against, those expectations have clearly been breached and the most appropriate thing to do as a trade union leader is to step down and resign,” Mr Walton told Sky News.