South Korean rescue teams, including elite navy SEAL divers, raced Wednesday to find nearly 300 people missing after a ferry sank with 459 on board, mostly high school students bound for a holiday island.

Lee Gyeong-Og, the vice minister of security and public administration, said 164 people had been rescued, leaving 292 “unaccounted for”. His office said there were three confirmed deaths, including a female crew member and a student.

The final death toll was expected to be much higher, after the 6,825-tonne ship listed sharply, capsized and finally sank within two hours of sending a distress signal at 9am (10am AEST).

“I’m afraid there’s little chance for those trapped inside to still be alive,” said one senior rescue team official, speaking by phone from the scene.

Dramatic television aerial footage showed terrified passengers wearing life jackets clambering into inflatable boats as water lapped over the rails of the vessel as it sank 20 kilometres off the southern island of Byungpoong.

Some could be seen sliding down the steeply inclined side of the ferry and into the water, as rescuers, including the crew of what appeared to be a small fishing boat, struggled to pull them to safety.

Several rescued passengers said they had initially been ordered to stay in their seats, before the ferry suddenly listed to one side, triggering panic.

“The crew kept telling us not to move and to stay seated,” one male survivor told the YTN news channel.

“Then it suddenly shifted over and people slid to one side and it became very difficult to get out,” he added.

Of the 429 passengers on board the ferry bound for the popular southern resort island of Jeju, more than 300 were students travelling with 14 teachers from a high school in Ansan just south of Seoul.

“I feel so pained to see students on a school trip … face such a tragic accident. I want you to pour all your energy into this mission,” President Park Geun-Hye said on a visit to the main disaster agency situation room in Seoul.

Many of the survivors were plucked out of the water by fishing and other commercial vessels who were first on the scene before a flotilla of coastguard and navy ships arrived, backed by more than a dozen helicopters.

Lee said 178 divers, including a team of South Korean navy SEALS, were searching the submerged vessel.

An amphibious assault ship, the USS Bonhomme Richard which was on routine patrol west of the Korean peninsula, was being sent to help.

The cause of the accident was not immediately clear, although rescued passengers reported the ferry coming to a sudden, shuddering halt – indicating it may have run aground.

The weather was described as “fine” with moderate winds and sea swell.