Last month's surprise cash-and-shares approach threatened to upend the LSE's $27 billion plan to buy data and analytics firm Refinitiv. The Hong Kong exchange had said the LSE would have to ditch the Refinitiv deal for its offer to go ahead.
LSE shares slid 6% by 0848 GMT, close to their lowest since Hong Kong Exchanges and Clearing Ltd (HKEX) announced its approach on Sept. 11. Shares in the Hong Kong bourse were up 2.3% near the end of its trading day.
HKEX chief executive Charles Li wrote in a blog post: "We still believe the strategic rationale for the combination of our two businesses is compelling and would create a world-leading market infrastructure group."
"Despite a huge amount of work and discussions with a broad set of regulators and extensive shareholder discussions, the level of engagement from LSEG led us to conclude that the continued pursuit of a combination of the two businesses would not be in the best interests of our own shareholders," he said.
LSE said in a statement it remained "committed to and continues to make good progress on its proposed acquisition of Refinitiv".
Investors were due to vote on the Refinitiv deal next month and the transaction was expected to close in the second half of 2020, the LSE said, subject to regulatory approvals.
Refinitiv is 45%-owned by Thomson Reuters which owns Reuters News.
On the HKEK offer, one LSE investor told Reuters it would have been "very, very hard to get the LSE board to engage as they rebutted the offer on grounds of strategy, not price." The investor spoke on condition of anonymity due to the sensitivity of the matter.
Analysts had viewed HKEX's chance of success as slim after it was rejected by the LSE just two days from the HKEX going public with its interest. Political turmoil engulfing Hong Kong and perceptions of Beijing's growing influence over the city, were seen as another key obstacle to any deal.
Subsequent efforts by HKEX officials to engage with LSE shareholders had also met with resistance. Some investors told Reuters the HKEX would have to raise its offer by at least 20%, mostly in cash, to tempt LSE shareholders.
"The price tag from the Hong Kong exchange perspective was getting a bit too high, so it's good for the shareholders that they decided to walk away," said Hao Hong, head of research at broker BOCOM International.
Since the HKEX announced its interest in mid-September, its shares had fallen 8%, compared with a 5% drop in the Hang Seng benchmark index.
The failure of the bold move leaves open the question of what Li might try next to fulfill the HKEX's strategy of being "China-anchored and globally connected".
The HKEX has been the world's largest capital-raising venue in five of the past 10 years, but has been working to diversify its equities focus, launching a bond trading platform with China and buying the London Metal Exchange in 2012 for 1.4 billion pounds ($2.2 billion).
"HKEX will continue to try other things. Charles Li has done a lot of deals, most notably the London Metal Exchange. It may not be a stock exchange, but other related areas," said Bocom's Hong.
Under British takeover rules, the HKEX had until Oct. 9 to make a binding offer for LSE. The withdrawal of the approach means it cannot bid again for the LSE for at least six months unless the LSE's management agreed to an offer, another group made a bid for the London exchange operator, or other events were deemed to be a material change in the LSE's circumstances.
"If the Refinitiv deal surprisingly fails to get approval, I think we could see HKEX come again," said China Galaxy Securities analyst Chi Man Wong.
"The (LSE) shareholder meeting (to approve the Refinitiv purchase) has been tentatively set for November but there is no firm date. If that deal fails then HKEX will be there."
Analysts at Bank of America said investors had been worried that the HKEX's pursuit of a deal could have led the exchange to raise fresh funds - weighing on the price of existing shares.
"We think there may still be questions as to why HKEX proposed the acquisition when it seemed somewhat challenging to bring to fruition, but we believe investors will likely focus more on the fundamentals of the company and the market in Hong Kong," they said in a research note.
(Reporting by Sumeet Chatterjee and Scott Murdoch in Hong Kong, additional reporting by Sinead Cruise in London and Devika Syamnath in Bengaluru; Writing by Jennifer Hughes; Editing by Kenneth Maxwell and Edmund Blair)
By Sumeet Chatterjee and Scott Murdoch