TOKYO, June 18 (Reuters) - Japanese stocks closed lower on Friday, with the Topix index dragged lower by financial stocks and Toyota Motor's retreat from a record high.

The broader Topix index ended 0.9% lower at 1,946.56, while the relatively tech-heavy Nikkei share average shed 0.2% to close at 28,964.08.

"The market has been unsettled all week and it will be difficult to decide how to allocate money going forward as the direction of the U.S. rates remain unclear," said Soichiro Matsumoto, chief investment officer Japan at Credit Suisse Private Banking.

Meanwhile, Japan may hold the Olympics with spectators despite opposition from medical experts, raising concerns that the Summer Games could spark another surge in COVID-19 infections.

The president of the Tokyo 2020 organising committee wants to allow up to 10,000 spectators for Olympic stadiums, local media reported.

"The Olympics is making the Japanese market's outlook unclear, which is why the market has failed to catch up with those in the U.S. and Europe," Matsumoto said.

Eisai jumped 5.9% after the drugmaker and Bristol-Myers Squibb entered into an agreement worth up to $3.10 billion to jointly develop and market an experimental cancer drug.

Medical platform services firm M3 jumped 3.4% and medical equipment maker Terumo rose 1.3%.

Toyota Motor fell 3.9% after hitting a record high this week.

Among sectors, insurers slid 3.4% and banks dropped 2.1%.

Shipping was the biggest decliner, falling 6.7% after surging over the past three months. (Reporting by Junko Fujita, additional reporting by Kevin Buckland; editing by Uttaresh.V)