Asad Shafiq made 42 and Sarfraz Ahmed was unbeaten on 66, the pair adding 93 for the sixth wicket after the tourists were reduced to 140-5 soon after tea.
Left-arm spinner Ranga Herath picked up five wickets for 98 runs.
Earlier, Sri Lanka were dismissed for 320 in their first innings 45 minutes before lunch after the last two wickets added 59 runs to the overnight score of 261-8.
Opener Upul Tharanga top-scored for Sri Lanka with 92. Left-arm seamers Junaid Khan (five for 87) and Wahab Riaz (3-88) shared eight wickets.
Sri Lanka won the first Test in Galle by seven wickets.