Authorities said they could not verify how many residents were still missing after the 6.4-magnitude quake which hit the popular tourist city late Tuesday.
Rescue efforts were focused early Wednesday on the Yun Tsui residential building, which also housed a restaurant, shops and a hostel. The quake left the 12-storey building leaning to one side, its lower floors pancaked.
The Hualien local fire service said 147 of the 213 residents from the building remained unaccounted for.
One local who lives nearby told AFP how he watched the tower block partially collapse.
"I saw the first floor sink into the ground. Then it sunk and tilted further and the fourth floor became the first floor," said Lu Chih-son, 35, who saw 20 people rescued from the building.
"My family were unhurt, but a neighbour was injured in their head and is bleeding. We dare not go back home now. There are many aftershocks and we are worried the house is damaged," he told AFP.