The Waratahs got off to a flyer on Saturday night, leading 14-0, and looked in great shape to back up their win over the Highlanders, which ended 40 straight losses for Australian teams against Kiwi foes.
The game was closer than the scoreline suggests, with just two points separating the teams with five minutes remaining.
But Chiefs playmaker Damian McKenzie scored his second try after the final siren to claim a bonus-point win and ensure the Waratahs left empty-handed.
The loss continued another miserable record - no Australian team has been able to win on New Zealand soil in more than three years, with a Michael Cheika- coached Waratahs outfit the last to do so against the Hurricanes.
NSW big guns Kurtley Beale, Israel Folau and skipper Michael Hooper stood up, and the team was desperately unlucky not to at least pick up a bonus point.
They now hold just a one-point lead over the Melbourne Rebels in the Australian conference.
The Waratahs can at least take confidence from their recent results with the finals looming, in recent weeks falling to the Crusaders by two points and the Blues by three.
"We're getting used to starting well against New Zealand teams and then the Chiefs did very well to battle back, as we knew they would," Waratahs coach Daryl Gibson said.
"It's disappointing. We lost some big moments out there through some good pressure from them and some tactically poor play from us.
"That's three New Zealand games we've lost and we've had an opportunity to win them, and learning from those moments is the big thing this team is going to carry forward."
While there were plenty of highlights, NSW let themselves down with a poor tactical kicking game. Turnovers through pushed passes also hurt them.
Big winger Taqele Naiyaravoro was a repeat offender.
Trailing 19-14 at halftime, NSW looked as if they'd level early in the second half but the final pass from Beale to Will Miller was ruled forward and with the referee playing advantage, they settled for a penalty.
The second half remained a fierce battle as the teams traded blows, with the Waratahs levelling through a converted Beale try in the 52nd minute.
Winger Toni Pulu scored his second of the night in the 57th minute from a clever Brad Weber kick and the home team again pulled away.
The Chiefs' defence, led by menacing lock Brodie Retallick, continued to stand up to anything the Waratahs threw at them.
McKenzie iced a penalty in the 75th minute and scored the final try to cap the win.
The Waratahs next take on the Queensland Reds in Brisbane before the June break for internationals.