The towering seven-storey walls of Sydney’s historic hat factory were demolished on Tuesday as onlookers gasped at the painstaking operation.

The 1910s-era building in Surry Hills was destroyed in a raging fire last week.

A specialist excavator on Tuesday began knocking apart the brick walls as people watched from outside a fenced-off exclusion zone.

The building was sprayed with water before the 60-tonne excavator hammered at the exterior walls, sending bricks crashing onto the street below.

Cadaver detection dogs were on standby as police faced the grim possibility that human remains could be buried in the rubble.

Two “rough sleepers” known to frequent the empty building remain unaccounted for.

A canine sniff of the site will be undertaken when it is considered safe.

The demolition plan was set on Tuesday morning but crews were open to reworking their strategy if they encountered problems, NSW Fire and Rescue Superintendent Adam Dewberry said.

Meanwhile NSW Police have spoken with a fourth boy over the incident after three other young teens handed themselves in to police after the fire last week.

“Detectives have identified and spoken to four teenage boys as they continue to appeal for anyone else to come forward,” NSW Police said on Tuesday.

“No charges have been laid at this time.”

Tweet from @9NewsSyd

Earlier, two large excavators picked over rubble as Fire and Rescue’s major structural collapse urban search and rescue unit truck arrived, and a surveillance drone flew overhead.

The street was laid with a carpet of thick steel pads to protect infrastructure below ground from damage by falling debris.

The site needs to be made safe before it can be thoroughly inspected and residents in neighbouring buildings can return home.

“We want to make sure everyone remains safe but we also don’t want to cause any damage to adjoining structures and utilities that are in and around and under the road,” Mr Dewberry said.

“We’ve got gas mains underneath, we’ve got electricity and water so we just need to make sure we don’t cause any more problems.”

The area will then be handed over to NSW Police so arson squad investigators and cadaver-detection dogs can examine the scene for any human remains or evidence as the rubble is removed.

“While police have not received any reports of missing people, investigators are unable to definitively say there is no one inside,” police said.

The brick and timber building and a neighbouring structure, formerly home to karaoke bar Ding Dong Dang, was known to regularly housing 15 rough sleepers.

Police have not been able to make contact with two of those people.

The blaze lit up the city sky on Thursday night, closing streets, diverting public transport and displacing more than 100 people who live in the vicinity.

Welfare services will continue to update displaced residents and keep them informed.

The brick hat factory was built in 1912 and operated until 1954 when proprietor RC Henderson went into liquidation.

The property was later leased to individual tenants in the 1960s and ’70s, and plans were afoot to use the brick structure for a $40 million redevelopment of the site into a 123-room hotel.