The Prime Minister has urged one of the men he dumped from Cabinet not to leave Parliament and held open the possibility of a return to the ministry.

Between Christmas and New Year, Prime Minister Malcolm Turnbull spoke with former small business minister Bruce Billson and told him to reconsider his decision to quit politics in the wake of his demotion.

In the conversation, the Prime Minister reflected on his own about face on leaving politics in 2010, and held open the possibility that 50-year-old Mr Billson had time to work his way back into Cabinet.

Mr Billson told the ABC his mind was made up to leave.

• Bruce Billson to quit politics
• Bruce Billson may quit politics
• Malcolm Turnbull unconvinced on GST hike

“My family has given our all to this job and we have voted for our family,” he said.

The Prime Minister’s office has been approached for a comment.

News of the conversation comes as the Minister for Veterans Affairs, Stuart Robert, faces increasing pressure to stand down over an undeclared trip to China in support of a Liberal donor’s business interests.

If he did, he would be the third Turnbull government minister to fall since the Cabinet was appointed in late September.

The decision to dump Mr Billson from Cabinet after September’s leadership coup came as a surprise to many observers, given he was an energetic and well-regarded minister who was widely seen to be one of the government’s most effective advocates.

At the time, there were complaints from some of his colleagues he was “talking himself out of a job” in the ministry by insisting on staying put in small business.

Mr Turnbull will have to reshuffle his ministry at some stage to replace former cities minister Jamie Briggs, who resigned in December after being accused of acting inappropriately on an overseas trip.

The timing of any reshuffle appears to hang on when the Deputy Prime Minister Warren Truss announces his retirement.

That has been expected since late last year but Mr Truss appears to be in no hurry to leave and there is furious jockeying inside the National Party over his replacement.

Forces are assembling to try and block what was once seen as the natural rise of Agriculture Minister Barnaby Joyce.

The Special Minister of State, Mal Brough, has also stepped aside while the police investigate his role in a long-running saga over the events that led to the fall of former speaker Peter Slipper.

If he is cleared, he will return.

It all adds up to a quandary for Mr Turnbull and his conversation with Mr Billson is being interpreted by some in the Coalition as a sign that he made a mistake when he cast his first Cabinet.

– ABC