Emergency services were called to Little Bay beach near Malabar on Wednesday afternoon where human remains were found in the water, police say.
Authorities in the New South Wales state have not named the victim, and an investigation is under way.
Little Bay and several nearby beaches are now closed.
At the time of the attack at 16:30 local time (05:30 GMT) on Wednesday, there were dozens of people swimming, paddle boarding and fishing on the nearby rocks, according to Sky News Australia.
"Some guy was swimming and a shark came and attacked him," one witness told Nine news.
"We heard a yell and turned around - it looked like a car had landed in the water," he said.