* 96% of proxy votes back deal, but 20.4% of shareholders against

* All-cash offer is one of Australia's biggest-ever buyouts

* UniSuper the only investor rolling scrip into unlisted vehicle

SYDNEY, Feb 3 (Reuters) - Sydney Airport shareholders on Thursday were poised to approve a A$23.6 billion ($16.85 billion) cash takeover by infrastructure investors, though many small retail investors voted against delisting Australia's only listed airport.

The Sydney Airport takeover, one of Australia's biggest-ever buyouts, was backed by 96% of proxy votes cast, but 20.4% of shareholders representing a 4% stake were against the deal. A final tally is due on Thursday.

Several retail shareholders said at a meeting that they were disappointed there was not a scrip component to the bid because they wanted to remain invested in the company over the long term. One long-term shareholder said he faced a big tax bill from accepting cash.

Chairman David Gonski said the bidders came to the board with an offer, and it did not include the opportunity for shareholders to roll their interests into the unlisted vehicle.

There was however, one exception. The consortium's bid was contingent on the company's largest investor pension fund UniSuper folding its 15% stake into the unlisted company rather than accepting the cash on offer to others.

After the deal is complete, UniSuper will join IFM Investors, QSuper, AustralianSuper and U.S.-based Global Infrastructure Partners as major shareholders.

The purchase is a long-term bet on the travel sector, which has been battered by the pandemic. Record-low interest rates have prompted pension funds and their investment managers to chase higher yields.

Australia's other major airports are unlisted and owned by pension funds and infrastructure investors, though New Zealand's Auckland International Airport remains listed.

An independent expert last month said the Sydney Airport deal was "fair and reasonable" to shareholders at a time of significant revenue uncertainty due to COVID-19 and the looming end of its monopoly when Western Sydney Airport opens in 2026.

Sydney Airport's domestic traffic fell by 74% in 2021 compared to 2019's pre-pandemic levels, while international traffic was down by 95.5%.

($1 = 1.4010 Australian dollars) (Reporting by Jamie Freed. Editing by Gerry Doyle)