The list of 12 titles for the literary prize, picked from a total of 145 books, included four debut novels and three small independent publishers, but it shunned a string of big-name writers.
Mantel, from Britain, was listed for her novel "Bring up the Bodies", a sequel to her 2009 Booker winner "Wolf Hall", while her compatriot writer Will Self was nominated for the first time for his novel "Umbrella".
There was no place however for heavy-hitters including Martin Amis, whose novel "Lionel Asbo" was published this year but has never won the prize.
"We did not set out to reject the old guard but, after a year of sustained critical argument by a demanding panel of judges, the new has come powering through," said chairman of the judges Peter Stothard.
"Goodness, madness and bewildering urban change are among the themes of this year's longlist."
The longlist included Indian writer Jeet Thayil, South African Andre Brink and Malaysian author Tan Twan Eng. The other nine authors are British while a total of seven men and five women make up the list.
Michael Frayn at 78 is the eldest to be included while British writer Ned Beauman is the youngest at 27.
The shortlist of six authors will be announced on September 11 while the winner will be announced at a dinner at London's Guildhall on October 16, picking up a £50,000 ($77,400, 63,700-euro) prize.
The Booker is awarded each year for the best original full-length novel written in the English language by a citizen of the Commonwealth, the Republic of Ireland or Zimbabwe.
Contenders must have been published in the past year and write in English. This is the 44th year of the prize, which began in 1969.
The long list is:
Nicola Barker -- The Yips
Ned Beauman -- The Teleportation Accident
Andre Brink -- Philida
Tan Twan Eng -- The Garden of Evening Mists
Michael Frayn -- Skios
Rachel Joyce -- The Unlikely Pilgrimage of Harold Fry
Deborah Levy -- Swimming Home (And Other Stories)
Hilary Mantel -- Bring up the Bodies
Alison Moore -- The Lighthouse
Will Self -- Umbrella
Jeet Thayil -- Narcopolis
Sam Thompson -- Communion Town