Indigenous constitutional recognition might be a key focus for Tony Abbott in Arnhem Land, but his government still intends to push back a referendum.

Indigenous Affairs Minister Nigel Scullion has reaffirmed 2017 is the government’s pick for a vote to include indigenous Australians in the constitution, as he and the prime minister continue discussions with Yolngu elders.

Next year was probably too soon and 2016 clashed with the federal election, he said.

• Employment at top of PM’s Arnhem Land agenda
• Senator Lambie calls for indigenous-only seats in parliament

 “It should and will be held at a date when all Australians support this in the same way that they supported the 1967 referendum,” Senator Scullion told reporters in northeast Arnhem Land on Monday.

If held in three years, the referendum would mark 50 years since Australia voted to give indigenous people the right to vote.

Mr Abbott’s advisory panel looking into public support for a referendum is likely to recommend a 2017 vote.

The prime minister also has flagged concerns that public awareness is not yet high enough for a successful vote.

Senator Scullion said constitutional recognition had been on the agenda for the 24 hours he and Mr Abbott had been in Nhulunbuy.

The local Yolngu people are clear any change must progress their people. “One thing they’ve said is ‘we can’t go back, we can’t lose anymore’,” he said.

The Yolngu people have told the minister they want progression in land rights and are willing to work with government.

Senator Scullion and Mr Abbott will take part in talks on constitutional recognition throughout the week as the government operates from a temporary tent village on the outskirts of Nhulunbuy.

Mr Abbott flew to Arnhem Land hours after announcing the deployment of a 600-strong military force to the Middle East.

The decision, which came after an official request from the US government, is likely to divert some of the prime minister’s attention away from indigenous affairs.

He is expected to hold teleconferences with officials in Canberra on Monday to discuss the deployment as part of an international effort to take on Islamic State militants.

Mr Abbott remains committed to his pre-election promise to spend a whole week in indigenous communities but says he can fly back to Canberra if necessary.