The 19-year-old captured the decisive break at 5-5 in the second set and served out the match nervelessly under the lights of Rod Laver Arena, stunning the two-time Wimbledon champion.
Keys' win made it four American women in the last 16 at Melbourne Park, the most since five made it to the fourth round of the 2003 Australian Open.
She will play compatriot Madison Brengle for a place in the quarter-finals.