Mori, 83, apologised at a hastily called news conference, but when pressed on whether he really thought women talked too much, he said: "I don't listen to women that much lately so I don't know".
Mori, whose term as prime minister was marked by a string of gaffes and blunders, made the comments at a Japanese Olympic Committee board of trustees meeting this week.
"If we increase the number of female board members, we have to make sure their speaking time is restricted somewhat, they have difficulty finishing, which is annoying," said Mori, according to local media.
