The incident happened on Saturday at the Mentao refugee camp, which hosts some 6,500 people who have fled neighbouring Mali, it said.
"According to refugee accounts, security forces burst into the camp in search of armed elements involved in an attack on soldiers earlier that day on (the) nearby Djibo-Ouagadougou road," the UN High Commissioner for Refugees (UNHCR) said in a statement.
"The incident had left one security personnel dead and another missing."
Security personnel went from house to house, forcing men and boys out of their homes, "beating them with sticks, belts and ropes", it said.
Four are being treated for broken limbs.
Refugees accused of complicity with the gunmen were told to leave the camp within 72 hours "or face death".
Mentao is located 20 kilometres (12 miles) south of Djibo in northern Burkina Faso -- a border region battered by a jihadist offensive in the Sahel.
"The actions of the security forces as reported to us are totally unacceptable," said Millicent Mutuli, the regional UNHCR chief.
"Refugees living in Mentao camp should be protected."
More 800 people in Burkina Faso have been killed and nearly 860,000 have fled their homes since armed Islamists in Mali extended their campaign across the border five years ago.
The beleaguered security forces in those countries, as in Niger, have been frequently accused of abuse.
On April 20, Human Rights Watch accused the Burkinabe security forces of the extra-judicial killing of 31 unarmed detainees in Djibo in a "brutal mockery of a counterterrorism operation" on April 9.
On May 30, the UN peacekeeping mission in Mali, MINUSMA, said 101 extra-judicial killings had been carried out by Malian security forces in the first three months of the year, and 34 by the Nigerien army.