The part-nationalised bank, which had set aside 1.9 billion pounds to deal with the claims, could be asked by the Federal Housing Finance Agency (FHFA) to pay settlements of more than 5 billion pounds over the sale of about $32 billion of mortgage-backed debt in the United States, the report said.

Spokeswomen for RBS and for the FHFA declined to comment.

The bank agreed to pay $99.5 million in June to settle claims it misrepresented more than $2 billion of mortgage-backed bonds during the U.S. housing bubble between 2005 and 2007.

The federal court in Connecticut is handling a second case which addresses RBS' sale of $30.4 billion in mortgage securities to Freddie Mac and Fannie Mae.

Shares in RBS fell 1.4 percent to 388.5 pence by 1538 GMT.

"We believe that a 5 billion settlement would be towards the top end of market expectations for this particular issue, and slightly above the levels where other banks have settled in relation to the assets involved, but it's not inconceivable," said RBC Capital Markets banks analyst Leigh Goodwin.

UBS in 2013 agreed to pay $885 million to settle a lawsuit that alleged the bank misled Fannie Mae and Freddie Mac into buying $4.5 billion of problem mortgage bonds.

The Times report, citing unnamed sources familiar with the situation, said RBS's 1.9 billion pounds provision was unlikely to be enough to cover the expected claims.

"The FHFA settlement is likely to be the largest of the outstanding litigation and conduct hits that RBS still faces, but there are many others, and they are likely to drag on for many years," Goodwin added.

RBS Chief Executive Ross McEwan said in October the bank would not pay a dividend until it had strengthened its capital position and had more clarity over future misconduct charges.

In September the FHFA had reached a total of $17.9 billion of settlements, stemming from lawsuits it filed in 2011 to recoup losses on roughly $200 billion of mortgage-backed securities purchased by Freddie Mac and Fannie Mae.

(Additional reporting by Nate Raymond; Editing by Simon Jessop, Keith Weir and Michael Urquhart)

By Pamela Barbaglia and Lionel Laurent