Netanyahu, who heads the right-wing Likud party, said he was returning the mandate back to Israel's president, Reuven Rivlin after "working unceasingly ... to establish a broad unity government".
Rivlin said he will now hold consultations with the various political parties to tell them he intends to task Netanyahu's centrist rival Benny Gantz with the job of putting together a new government.
Gantz's Blue and White party was the largest to emerge from the Sept. 17 ballot, though he also has no clear path to a parliamentary majority.
Should he also fail to bring in enough partners it would almost certainly lead to another general election, the third since April.