That remark was met with widespread condemnation, and on Wednesday Duterte tried to defend himself.
At the end of a long speech to naval officers and their families, Duterte said he was just being sarcastic as he took aim at people who criticized his rape remarks but particularly Clinton.
He then accused American soldiers of raping women in the Philippines and Japan, without giving details, before returning to Clinton.
'It is a crime actually committed by soldiers, mostly Americans in Okinawa,' he said, referring to several rape cases involving U.S. servicemen stationed in Japan, most recently last year.
"But, we never heard of a Filipino. But I am just warning them that anything they do, I have to answer for it. I take full responsibility for your foolishness. I speak sarcastically."
'You Americans, like Chelsea, be careful because you live in a glass house,' he said.
'I repeat, when President Clinton was fucking Lewinsky, what was your statement or your reaction?'
It was not the first time Duterte has made a joke about rape.
He caused outrage in the lead-up to his presidential election win last year when he recalled a 1989 prison riot in which an Australian missionary was killed, and inmates had lined up to rape her.
In what was intended as a joke, Duterte said the victim was "beautiful" and as mayor of Davao city where the riot took place, he should have been first in line. He later apologized and said he meant no disrespect to women or rape victims.
The Australian and American ambassadors to Manila voiced their disapproval at those comments, but Duterte reacted furiously then while insisting he had been taken out of context.