Beijing-based Megvii did not win approval for its initial public offering (IPO) at a hearing with the Hong Kong Stock Exchange's Listing Committee on Thursday, the people with knowledge of the matter told Reuters.
Megvii was given additional questions it must respond to by the committee, according to the sources, with one adding the decision followed a lengthy discussion of the application.
The meeting was a regularly scheduled committee hearing, during which the IPOs of several other companies were discussed and approved.
The 27-strong committee contains a mix of bankers, lawyers, accountants and investors, and its approval is essential for any IPO candidate.
Megvii and the three main banks working with it on the IPO - Citigroup, Goldman Sachs and JPMorgan - declined to comment, as did the Hong Kong Stock Exchange. The sources spoke on condition of anonymity because the information is not public.
Concerns about Megvii included questions about its suitability to seek an IPO, said the sources, after the Trump administration last month barred the company from buying U.S. parts and components without U.S. government approval.
Megvii was put on the blacklist along with seven other Chinese companies for their alleged involvement in human rights violations related to Beijing's repression of Muslim minority populations in the Xinjiang Uighur Autonomous Region.
Megvii said at the time it strongly objected to being blacklisted, and there were "no grounds" for the designation.
The company, backed by e-commerce giant Alibaba, and its banking advisers are in the process of answering the listing committee's questions, said two of the people.
Separate sources previously told Reuters the company had looked to go public in Hong Kong in the fourth quarter and planned to raise between $500 million and $1 billion.
The biggest market for raising funds though IPOs in 2018, Hong Kong has seen a slowdown this year amid anti-government demonstrations, though activity has increased since September.
Alibaba this week raised HK$88 billion ($11.3 billion) in a landmark Hong Kong secondary listing.
Megvii, known for its facial recognition platform Face++, was founded in 2011 by Chief Executive Yin Qi and two friends from Tsinghua University. If the IPO goes ahead, it will be the first Chinese AI firm to go public.
The company provides AI technology to governments and companies including Alibaba, Ant Financial and Huawei.
In May, a $750 million fundraising valued the company at slightly over $4 billion and attracted investors including Macquarie Group and the Abu Dhabi Investment Authority.
By Scott Murdoch and Julie Zhu