Workers have downed tools at an inner Sydney construction site where a man fell to his death.

Construction was on hold on Friday morning and a counsellor was on hand.

Emergency services were called to the Barangaroo development on Thursday morning following reports a man, believed to be aged 30, had fallen from scaffolding.

NSW Paramedics tried to revive the man but he died at the scene.

A stop-work meeting was underway at the Lend Lease site on Friday morning, but a large number of workers have now begun leaving the development.

It is understood they have been given the choice to stay or go.

There were emotional scenes outside, with one worker screaming at waiting media before police rushed in to pull him away.

Construction, Forestry, Mining and Energy Union NSW secretary Brian Parker said the union was concerned about safety and supervision at the worksite.

“The company are getting paid a lot of money under federal and state government grants to mentor and to supervise young workers on this particular job, and he’s fallen to his death some 30 metres,” Mr Parker told Macquarie Radio on Friday.

A police spokeswoman on Friday said the death was not being treated as suspicious or work-related.

WorkCover NSW has been notified.

(Readers seeking support and information about suicide prevention can contact Lifeline on 13 11 14, or MensLine Australia 1300 78 99 78.)