Brazil's real rose as much as 1.2% before trading flat, while Mexico's peso rose almost 1%. Chile's peso firmed 1.1%, while Peru's sol snapped a three-day losing streak.
The dollar's bounce on Tuesday put pressure on the euro, which dropped once again below the $1.20 mark on Wednesday, hitting its lowest against the buck in over two weeks.
The dollar index, which measures the greenback against a basket of peer currencies, rose as high as 91.436, its highest since April 19.
Though Yellen acknowledged the higher growth "could cause some very modest increases in interest rates," the United States needs the investments "to be competitive and to be productive."
Nikkei share average lost 0.81% to 29,844.81 by 0208 GMT, after hitting the 30,000 mark for the first time in more than two weeks on Monday. The broader Topix fell 0.74% to 1,968.80
"The market fundamental is strong. But ultimately the stimulus packages in the US and the direction of the long term interest rates could determine the move of the Japanese market."