The investors were worried that Yunfeng Capital might not put their interests first, said David Yu, a long-time friend and business associate of Ma, speaking to Reuters in a rare interview at his Hong Kong office on July 3.
Yu said the independence of Yunfeng from Ma was "the number one concern institutional investors had".
He named several high-profile investors that came on board after getting assurances Ma played no direct role in Yunfeng's investment decisions.
However, four other institutional players told Reuters they opted out because they said they were worried Ma's role as one of China's most active private investors could conflict with his responsibility to Alibaba shareholders. They declined to be identified because their talks with Yunfeng were private.
"You can look at Yunfeng and you can say there are clearly a lot of synergies with Alibaba and Jack Ma, but you can also say it's a clear conflict of interest. It's a double-edged sword," said one person whose firm turned down an opportunity to invest.
To be sure, it is not unusual for successful businessmen in China or elsewhere to invest personal wealth in private equity and venture capital firms. In Silicon Valley, lines between entrepreneurs and their companies can also become blurred, leading to complaints about conflicts of interest.
Yunfeng still raised $1.1 billion for its latest fund, but its challenge attracting institutional investors underscores the long-standing concerns some investors have had about Ma as he prepares to take Alibaba public next month in a New York IPO that could raise more than $15 billion.
Alibaba acknowledged the broader issue of potential conflicts in its IPO prospectus issued on May 6, saying that given Ma's other roles it could not guarantee he would "act in the best interests of our company should any conflicts of interest arise".
Alibaba declined to comment or make Ma available for an interview for this story.
TIES THAT BIND
There are plenty of links between Alibaba and Yunfeng, a firm Ma and Yu founded in 2010 and which has a total of $3 billion under management.
The two entities have invested in each other, according to filings from Chinese online gaming firm Giant Interactive Group Inc (>> Giant Interactive Group Inc (ADR)) with the U.S. Securities and Exchange Commission, and sportswear maker China Dongxiang Group Co Ltd (>> China Dongxiang (Group) Co., Ltd.) with the Hong Kong stock exchange.
Yuzhu Shi, the chairman of Giant, and Yihong Chen, the CEO of Dongxiang, are both founding partners of Yunfeng, according to Yunfeng's website.
Ma also has personal money invested in China-based Yunfeng, Alibaba's filings show.
Giant and Dongxiang were also among around a dozen investors that poured about $500 million into Alibaba in 2011 via Yunfeng, the two firm's filings show. The investment was part of a $1.6 billion tender offer to Alibaba employees that allowed some Alibaba staff to cash out their shares in the company.
Yunfeng and Alibaba have also invested in at least three deals together, starting in 2010 when they paid $24 million for a combined 16 percent stake in a Chinese online search engine company called Sogou Inc, a subsidiary of Sohu.com Inc (>> Sohu.com Inc).
Yu said the firm told institutional players that Ma did not sit on its investment committee, which decides which companies to invest in. Yunfeng also showed investors documents to prove it sourced its deals independently, Yu added.
Among the investors that bought into the fund were Singapore sovereign wealth fund GIC [GIC.UL], Yu said. Alibaba's arch-rival, Chinese Internet firm Tencent Holdings Ltd (>> Tencent Holdings Ltd), was another.
Yu declined to say which entities expressed concern about Yunfeng's independence or if any of those that did chose not to invest. GIC did not respond to calls and emails seeking comment. Tencent declined to comment.
"There is a very clear separation of duties," said Yu, who sits on the four-member committee that makes investment decisions.
According to Alibaba's SEC filings, Ma donates any profits he makes from Yunfeng to charity.
Ma and Yu agreed to form the private equity firm to invest some of their personal wealth after a boat ride in June 2009 at West Lake in the eastern Chinese city of Hangzhou, Alibaba's home base.
Also on the boat was global financier George Soros, who was visiting Alibaba's headquarters, and who discussed their idea of helping a new generation of entrepreneurs in China, Yu said. Soros Fund Management declined to comment.
This year, Yunfeng has invested around $182 million alongside Alibaba in two deals totaling close to $1.4 billion. The most recent was in April, when the two agreed to pay $1.22 billion for an 18.5 percent stake in online video website Youku Tudou Inc (>> Youku Tudou Inc (ADR)), a business akin to Google Inc's (>> Google Inc) YouTube.
(Additional reporting by Gerry Shih in San Francisco, Saaed Azhar in Singapore and Paul Carsten in Beijing; Editing by Dean Yates)
By Stephen Aldred and Matthew Miller