Victorian Speaker Ken Smith has told state parliament he will quit the post.

As Mr Smith announced he would be stepping down, he took a swipe at independent Frankston MP Geoff Shaw who holds the balance of power in the parliament.

“I have no confidence in the member for Frankston and I believe he is unworthy of being a member of parliament,” Mr Smith told the first sitting of the Victorian parliament for 2014.

Mr Smith said he would not be bullied into making any deals with Mr Shaw, who he said never once came to air his grievances with him.

He said he believed Mr Shaw declined to support him after he refused a number of “outrageous” demands put to him by Mr Shaw that went against policy and regulation.

“I will not be doing any deals with the member for Frankston, nor will I be bullied or held to ransom to comply with his demands,” he told parliament.

“I have no confidence in the member for Frankston, and actually I believe he is unworthy of being a member of this parliament.”

Mr Smith also lashed out at the Labor opposition, whom he accused of plotting with Mr Shaw to pressure him to step down by creating chaos in parliament.

He accused Opposition Leader Daniel Andrews of showing unparliamentary behaviour and “typical union tactics” to collude with Mr Shaw.

Mr Smith suspended parliament for an hour so members can consider a new Speaker.

“I’m proud to be given the honour of being the Speaker of the Victorian parliament and I inform the house I will standing down at 3pm today,” he said.

Mr Smith, the lower house Speaker, had been under pressure since balance-of-power MP Mr Shaw announced late last year he no longer supported him.

The revelation led to chaotic scenes in the parliament as the opposition repeatedly tried to force a vote of no confidence in him.

As a result, Mr Smith shut down parliament for two days and suspended two Labor MPs for the remainder of the year.