Those who did not sing properly did not deserve to represent the country, said Yoshiro Mori, a former prime minister with a history of gaffes.
He was speaking to 300 athletes at a send-off ahead of the August Olympic Games to be held in Rio de Janeiro.
Reports say attendees had just sung Kimigayo, Japan's national anthem.
"When you go up to the podium, please do not be mumbling but sing the national anthem," Mr Mori told those gathered on Sunday.
