The move marked a sharp turnaround from as recently as last week, when some officials were still insisting they could organise the Games safely with some fans.
It all but strips the global sporting showpiece, which is due to start on 23 July and run until 8 August, of its last vestiges of pomp and public spectacle.
Prime Minister Yoshihide Suga said it was essential to prevent Tokyo, where the highly contagious Delta variant of COVID-19 variant was spreading, from becoming a flashpoint of new infections.
People will also be asked not to gather for events on public roads, such as the triathlon, though officials said some venues outside the greater Tokyo metropolitan area would allow small numbers of spectators
"It is regrettable that we are delivering the Games in a very limited format, facing the spread of coronavirus infections," Tokyo 2020 President Seiko Hashimoto said, following talks between government officials, Tokyo organisers and Olympic and Paralympic representatives.
"I am sorry for those who purchased tickets."
Once seen as an opportunity for Japan to shine on a global stage after a devastating earthquake, tsunami and nuclear disaster over the past decade, the Games were postponed last year due to the pandemic and have been hit by massive budget overruns.
Much of Japan's population is still not vaccinated against COVID-19, stoking fears that an influx of thousands of athletes and officials will fuel more infections.
Medical experts have said for weeks that holding the Games without spectators would be the least risky way of staging the event.
Organisers were still uncertain how much revenue would be lost due to the absence of spectators, Tokyo 2020 chief executive Toshiro Muto said, adding they were looking to further cut staff.
Torch relays have been scaled back or taken off of public roads, and promotional events dropped. Tokyo - which had been counting on a record boom in tourism - has experienced none of the buzz and excitement that normally characterise host cities.
Japan's drive to hold the Olympics even as the epidemic worsened alienated much of the public and sparked concern among sponsors about a consumer backlash.
Suga said COVID-19 infections were on the rise in Tokyo, due in part to the Delta variant, warning that that it could hit the rest of the country.
"We absolutely must avoid Tokyo being the starting point again of another spread of the infection," he told a news conference.
Tokyo's neighbouring prefectures of Kanagawa, Saitama and Chiba will also not allow spectators at their Olympic events, the government said.