The ruling, in a case brought by Iranian and Afghan families detained for more than a year after their asylum applications were refused, is the latest clash between EU authorities and Prime Minister Viktor Orban's government.
The Luxembourg-based court ruled that holding people at the Roszke transit zone - a container camp built into a fence along the Serbian border - amounted in legal terms to detention.
In its judgment, handed down Thursday, the court said migrants could not be detained without "a reasoned decision ordering that detention and without the need for and proportionality of such a measure having been examined".
It also said that the detention of asylum seekers "may not under any circumstances exceed four weeks", counting from the date they lodged their application for asylum.
The Hungarian Helsinki Committee (HHC), a human rights organisation that represented the families, welcomed the ruling, saying it meant the transit zones amounted to "unlawful detention".
"All those kept in the transits beyond four weeks must be released. If they are still in the asylum procedure, they should either be placed in an open reception facility or, after an individualised assessment, in formal asylum detention," Andras Lederer of the HHC told AFP.