"Four people were found dead: two women, an 11-year-old boy and a seven-month baby," Silvester Temmar, head of Maluku Tenggara Barat district in the east of the country, told AFP.
"The remaining 100 passengers have been rescued alive and no one is reported missing."
He said the wooden boat, with a capacity of about 50 passengers, was heavily overloaded and began to sink when it was hit by bad weather.
The disaster came just four days after another boat overloaded with 250 migrants, mostly from Afghanistan and Iran, heading for Australia sank in bad weather 40 nautical miles off eastern Java early Saturday.
Only 47 survivors were rescued from that vessel, and rescuers plucked about 40 bloated and rotting bodies from the sea on Wednesday.
Temmar said that the boat that sank Wednesday was filled with villagers going home for Christmas.
"People were heading to their home village to celebrate Christmas at the weekend," he said.
"It sank less than a mile from the shore near the coastal village of Lat Dalam. Local people rushed to help the survivors," he added.
Boat accidents are commonplace in Indonesia, an archipelago of some 17,000 islands that relies heavily on sea transport. Disasters involving desperate migrants trying to reach Australia are also common.
Many try to head for Australia's Christmas Island, which is closer to Indonesia than Australia, and where nearly 50 would-be migrants are believed to have died in wild seas during a shipwreck in December 2010.