Old magazine articles resurfaced in which Keigo Oyamada, 52, boasted of the bullying and said he had no regrets.
After the scandal broke, the performer apologised. But the outrage in Japan has now led to his resignation.
It is the latest setback for the games, which are opening on Friday under the shadow of the coronavirus outbreak.
"I have become painfully aware that accepting the offer of my musical participation in the Tokyo 2020 Olympics and Paralympics lacked consideration to a lot of people," said Mr Oyamada, who is widely known as Cornelius.
