Police say they “cannot rule out” the possibility of finding bodies in the rubble of a Sydney building destroyed by a massive blaze.

The warning came as work began on Monday to demolish the derelict hat factory after in Surry Hills, after an inferno gutted it and displaced more than 100 people.

“While police have not received any reports of missing people, investigators are unable to definitively say there is no one inside,” NSW Police said in a statement.

The heritage-listed Henderson hat factory building was known to regularly house 15 rough sleepers. Police have since contacted 13 of those people to confirm their safety, but two remained unaccounted for on Monday.

“We can’t rule out that there were people under the rubble. We can’t rule it out,” Detective Superintendent Gordon Arbinja said.

“I don’t have their identities.

“I just have a description of them, but we need to do the best we can do to account for these people.”

At least 100 people living in the area have registered as being displaced since the fire lit up the city skyline last Thursday, closing streets and diverting public transport.

Police are appealing for anyone who saw suspicious activity in the vicinity in the lead-up to the fire to come forward.

Teenagers spotted running from the building shortly after the blaze started handed themselves in last Thursday night.

Officers were questioning a fourth boy on Monday, and said they still wanted to talk to three or four more.

Detective Superintendent Arbinja said “it’s possible” charges would be laid against some of the youngsters, including a 12-year-old and two 13-year-old boys, who are already assisting detectives with their investigation.

“We’ll go through the whole investigation and then we’ll work out what the best and most appropriate measure is to solve this matter,” he said.

An exclusion zone remains in place in Surry Hills as the old factory’s unstable walls continue to pose a threat to public safety.

Excavators were clearing rubble at street level on Monday, with heavy demolition machinery expected to arrive later.

From Tuesday, it will begin to take apart what remains of the seven-storey, heritage-listed brick building.

Fire and Rescue NSW Superintendent Adam Dewberry said the demolition would start as soon as the light allowed.

“The whole thing is about getting in as early as possible,” he said.

Safety remained a priority after concerns more parts of the structure would collapse over the weekend, he said.

One of the factory’s walls rained bricks on Randle Street as the inferno raged on Thursday night. It was one of the largest fires seen in central Sydney for about a decade.

Firefighters then warned another wall moved slightly inwards on Friday night, highlighting the importance of the exclusion zone affecting neighbouring apartments.

The fire also continues to burn under the rubble.

“Specialist firefighters remain on scene monitoring the stability of the remaining structure and smouldering from deep within the rubble,” NSW Police said.

“The priorities for the emergency operation are to render the fire ground safe to allow on-site investigations, co-ordinate for the safe demolition of the remaining structures, limit the impacts on traffic and public transport and, importantly, house displaced residents and ensure the timely return to their properties.”

Specialist crime scene officers will conduct further investigations, including using cadaver-detection dogs and forensic imaging as sections of the rubble are removed.

Police said they expected the exclusion zone in the surrounding area to remain in place for several more days.

There had been plans to turn the heritage-listed building into a $38-million hotel complex.

-with AAP