Less than a year before Britain is due to leave the EU, the Office for National Statistics said 2.292 million nationals of other EU countries were working in Britain in the first three months of 2018, 1.2 percent fewer than a year before.
The fall was driven mostly by workers from the eight eastern European countries which joined the EU in 2004, whose numbers have fallen by 9.1 percent over the past year.
But the number of Romanians and Bulgarians - who have only been able to work freely in Britain since 2014 - continued to rise sharply, increasing by 19.8 percent over the year.
Looking at EU-born workers rather than EU nationals - a measure preferred by some academics as it includes long-term immigrants who acquire British citizenship - the number of workers rose by 0.3 percent, the smallest increase since 2010.
A desire to control migration into Britain was one of the main drivers of the vote to leave the EU in the 2016 referendum, although business groups have said clamping down on migration will probably hurt the economy.
In March, a report commissioned by the British government warned that restricting migration into Britain will likely lead to weaker economic growth.
Tuesday's figures chimed with separate migration data that have shown a sharp decrease in the number of people moving to Britain from the EU.
"If this trend continues, it will sound alarm bells for employers especially in low-skilled sectors," Gerwyn Davies, senior labour market analyst at the Chartered Institute of Personnel Development, said.
Food manufacturing and social care were among the parts of the economy that were most likely to be affected by a drop in EU workers, especially those from eastern Europe, he said.
This was echoed by the Food and Drink Federation, a trade body whose members employ 117,000 EU workers.
"The government must move swiftly to set up the new registration system and provide clear guidance to workers and businesses so that our valued EU workforce and those arriving during transition have the security they need," it said.
While the number of EU nationals employed has decreased over the last year, overall employment growth in Britain has remained strong, easily outpacing forecasts in the three months to March.
"It seems that perhaps organisations have grasped the challenge and have become more adept at recruiting from the domestic workforce. The question of course is whether the quality of applicants is sufficiently good," Davies said.
The latest migration data are due on May 24. In February, the ONS said net migration of EU into Britain almost halved in the 12 months to September.
(Reporting by Andy Bruce; Editing by Hugh Lawson)