Prime Minister Tony Abbott has said constitutional change to recognise Indigenous Australians should not be rushed, despite promising swift action a year ago.

In a week-long visit to Arnhem Land in September last year, Mr Abbott said a timeframe on Indigenous constitutional recognition would be finalised within weeks.

But the process has largely been stalled.

• The week that Tony Abbott lost control
• Tony Abbott blasts ‘racist’ campaign
• Tony Abbott: every day is a test for government

“I can understand why people are impatient, I really can, ” Mr Abbott said.

“It’s a natural instinct when something needs to be done to want to do it today. But it’s more important to get it right.

“And if we were to rush this, and get it wrong, if we were to rush this and come up with something that Indigenous people felt had been foisted on them, or rush it and come up with something that was defeated at a referendum, that would be tragic.

“I’m confident that the process is on track.”

Mr Abbott arrived on Mer Island – also known as Murray Island – in the Torres Strait on Monday, where he began his annual trip to a remote Indigenous community by visiting the grave site of land rights campaigner Eddie Koiki Mabo.

Mr Abbott was the first prime minister to visit the grave in far north Queensland.

He said the referendum calls came at a time when Indigenous Australians were “absolutely front and centre, heart and soul of modern Australia”.

One of Eddie Mabo’s children, Gail Mabo, personally took Mr Abbott to visit her father’s grave site.

The significance of Mr Abbott’s visit could be seen on the face of Ms Mabo, who fought back tears as she thanked the PM for acknowledging her father.

“This is a space that is very spiritual, it is spiritual in the sense that this is where he wanted to be,” she said after the brief ceremony.

“This is the space he was fighting for.”

Mr Mabo led the landmark High Court case that in 1992 handed indigenous Australians land rights and rejected the legal doctrine of terra nullius – or land belonging to no one.

Mr Mabo died of cancer six months before the High Court ruled in his favour and never witnessed the outcome or the rights it handed to his people.

The gravesite visit was Mr Abbott’s first official business of his trip to the Torres Strait to honour his promise to run government from a remote indigenous community for one week each year.

Indigenous Affairs Minister Nigel Scullion and Parliamentary Secretary to the Prime Minister Alan Tudge will accompany Mr Abbott for the week, along with secretary of the Department of Prime Minister and Cabinet Michael Thawley.

A string of senior ministers are expected to join him for shorter periods including Attorney-General George Brandis, Health Minister Sussan Ley and Social Services Minister Scott Morrison.

– with AAP, ABC