At the close, the benchmark S&P/ASX 200 was 19.6 points down at 4,082.3, following falls in the United States and Europe to end last week.
The index had earlier been up more than one percent on strong earnings reports and takeover announcements.
"Today's skittish price action is evidence of a market that doesn't know what to do next," said IG Markets analyst Ben Potter.
"There are plenty of bears out there calling for further significant falls, but at the same time there are just as many optimists suggesting equities at current levels are a fantastic buying opportunity.
"Although we’re seeing some stocks react to earnings reports and stock specific news stories, it really is all about the macro picture at the moment."
Among major companies, BHP ended marginally higher at Aus$37.53 but fellow miner Rio Tinto slipped 1.2 percent to Aus$68.58.
The Australian dollar was trading at 103.94 US cents late in the day, from 103.75 at the close on Friday.
Copyright AFP (Agence France-Presse), 2011