Construction union official John Setka says crossbench senators who don’t like his campaign against proposed anti-union laws might need to “toughen up”.

The CFMEU Victorian secretary also denies threatening independent senators Jacqui Lambie and Rex Patrick.

But Senator Patrick believes Mr Setka’s comments go beyond normal campaigning and he has referred them to the Australian Federal Police.

“Mr Setka foreshadowed members of his organisations crossing paths with myself and Senator Griff at some future time and engaging in abuse,” Senator Patrick told the upper house on Wednesday.

The Senate crossbench is considering voting for proposed Coalition government legislation that would make it easier to ban rogue unions and their officials.

Mr Setka told a recent shop stewards meeting the senators would “wear the consequences” of their decision, and they can be expect to be abused on the street in 20 years time.

“Someone is going to point the finger and say, ‘There’s them f—ers that voted for that bill that f—ed up not just construction workers but all workers in Australia’,” Mr Setka said in the meeting last week.

Mr Setka played down the comments on ABC radio on Wednesday.

“There’s been no threat made,” he said.

“Maybe they should toughen up a little bit because it’s called campaigning.

“They’re pretty bad laws we’re looking at.

“If they don’t like what we’re saying, well I don’t like what they’re saying.”

Senator Lambie has asked Mr Setka to step down from his union job. He says it wasn’t because of his conviction for harassing his wife, but rather because the senator thinks he’s become a “target”.

The Tasmanian’s vote will be crucial in deciding if the union-busting laws pass.

“My vote will never be determined by who bullies me the most,” Senator Lambie said.

But Mr Setka says while he was in a “bad place” when police were pressing blackmail charges against him – which were later dropped – he and his wife received counselling and were doing well.

“We’re not proud of the things we did and said to each other,” he said.

Opposition Leader Anthony Albanese wants Mr Setka kicked out of the Labor Party, but the union boss says he has no reason to step aside.

“This all started when I said I was not going to give any more money to the ALP,” he said.

Mr Setka says his union is delivering some of the best wages in the country, workers are safer, buildings are done ahead of schedule, and there are no strikes.

“Why are we so bad?” he asked.

Former Labor leader Bill Shorten said Mr Setka’s campaign against the proposed laws was counter-productive.

“I think attacking the crossbench senators and criticising them when you want their votes, to be honest, it may work in the construction world but it doesn’t work in the political world,” he told Sky News.

-AAP