The 30-year-old German, who is seeking his sixth straight pole position and hoping to complete a hat-trick of successive wins, was one-tenth of a second faster than the Briton as the pair scrapped for honours ahead of the last race of the season.
Rosberg clocked a best lap in 1min 41.983sec, ahead of Hamilton who recorded 1:42.121 with Mexican Sergio Perez third for Force India, four-tenths adrift of Hamilton, in hot conditions at the Yas Marina circuit.
Australian Daniel Ricciardo was fourth for Red Bull, who earlier confirmed they have agreed a deal for an unnamed engine for next season, ahead of four-time champion German Sebastian Vettel of Ferrari and Russian Daniil Kvyat in the second Red Bull.
Finland's Kimi Raikkonen was seventh in the second Ferrari ahead of German Nico Hulkenberg in the second Force India and, surprisingly, two-time champion Spaniard Fernando Alonso in his McLaren Honda.
Venezuelan Pastor Maldonado was 10th for Lotus, whose participation in the event was only confirmed after F1 commercial ringmaster Bernie Ecclestone paid their expenses. The struggling Lotus team are expected to be taken over by Renault for next season.
Earlier in the day, Hamilton had topped the opening free practice session and given an on-track answer to his critics after negative comments were made about his lifestyle.