The Social Democrats (SPD) and their likely allies, the fast-growing Greens, gained the edge over Merkel's coalition of Christian Democrats (CDU) and Free Democrats (FDP) in opinion polls before the vote in Baden-Wuerttemberg.
In another vote in neighbouring Rhineland-Palatinate, the SPD was expected to retain power with the Greens' support.
Voter turnout on a warm and mild early spring day in both states was higher than four years ago, election officials said.
The CDU, which has governed Baden-Wuerttemberg without interruption since 1953, has been hurt by Merkel's policy U-turns on nuclear Power, Libya and the euro zone debt crisis.
"Our chances to win were never better," said SPD candidate Nils Schmid after casting his ballot in Baden-Wuerttemberg.
The race has turned into a referendum on Merkel.
"She no longer looks like a strong leader for her party who can get the job done," Gero Neugebauer, political scientist at Berlin's Free University, told Reuters.
Baden-Wuerttemberg, on the French and Swiss borders is known for its car industry, medieval university towns and the Black Forest. Losing it would weaken Merkel's grip on her party, which suffered a bitter state election defeat in Hamburg last month.
It would be ironic as Baden-Wuerttemberg is one of Germany's richest states with its lowest unemployment rate 4.7 percent.
CDU state premier Stefan Mappus warned of disorder in the southwestern state if the conservatives lose control.
"What's at stake is the future of Baden-Wuerttemberg, we've got the lowest unemployment and highest wages in Germany," Mappus told Bild am Sonntag newspaper.
"There would be chaos if the SPD and Greens win."