The Turks went into the game looking to claw back a 1-0 first leg deficit but Arslan scored the only goal of the game at the Ataturk Olympic Stadium and then, following a goalless half hour of extra-time, struck the decisive penalty before Dejan Lovren blazed his effort for Liverpool over the bar.
With a tough clash against Manchester City in the Premier League at Anfield on Sunday, it was the nightmare scenario for Liverpool manager Brendan Rogers.
Not only did they play an energy-sapping 120 minutes of football but also had the mentally draining penalty shoot-out and a morale-sagging defeat to contend with before making the long trip home from Istanbul.
Besiktas had levelled on aggregate in the 72nd minute with a stunning strike from Arslan, assisted by a deft back heel from Senegalese striker Demba Ba, who had been the scourge of Liverpool last season when his goal, following a slip from Steven Gerrard, set Chelsea on the way to a 2-0 win at Anfield that wiped out the Reds' advantage over eventual champions Man City in the league.
With penalties level on 4-4 after all the takers had held their nerve, Arslan deftly slotted in. Lovren stepped up for Liverpool but his effort went high, leaving Besiktas fans running onto the pitch in joy.
Title-holders Sevilla are also through to the last 16 following a 3-2 win at Borussia Moenchengladbach on Thursday.
Machin Vitolo scored a brace as the Spanish La Liga strugglers earned a 4-2 aggregate win after winning the first leg 1-0 in Andalusia.
Borussia didn't go quietly, though, twice fighting back from a goal down -- Martin Stranzl's own goal opened the scoring -- through Granit Xhaka and Thorgan Hazard before Vitolo's second ensured there would be no dramatic turnaround for the Germans.
Sevilla actually had goalkeeper Sergio Rico to thank for keeping them on level terms until Xhaka was sent-off 20 minutes from time.
Even with 10 men the Germans looked a threat but Vitolo killed off their challenge with the winner 11 minutes from time.