Britain's Financial Conduct Authority (FCA), which opposed the scheme in court, said it thought Amigo could propose a fairer deal for customers.
Subprime lenders in Britain have been hit by a regulatory clampdown in recent years that has led to a wave of claims, and in some cases compensation payouts, for mis-selling loans. The effect of the COVID-19 pandemic has added to the strain.
Amigo - which has 137,000 current customers and has lent to 500,000 people since 2005 - had applied to the High Court for permission to cap payouts, saying a surge in claims threatened it with collapse.
In his judgment, Mr Justice Miles said the company's counsel had said an administration was both probable and imminent without the scheme.
But he said he agreed with the FCA that Amigo did not face an immediate liquidity crunch and urged the company to propose a fairer deal that spread losses with other stakeholders.
Amigo's shares tumbled as much as 61% in early trading.
The lender said its board was reviewing all options including an appeal after the court's decision.
"We suspect a potential sweetening of the terms must be under very careful consideration to avoid an outright collapse of the business into administration," Goodbody analyst John Cronin wrote in a note.
The FCA said other finance firms should take account of the judgment, adding it had "significant concerns" about schemes being used to unfairly avoid paying redress to customers.
Amigo specialises in guarantor loans, providing finance to customers with poor credit histories if they are guaranteed by a friend or family member.
Rival Provident Financial quit its around 140-year-old doorstep lending business this month.
Amigo's plan had drawn criticism from politicians and consumer groups.
Around 95% of votes cast by current and former Amigo customers ahead of the court hearing were in favour of the proposal.
Mr Justice Miles in his judgment said he agreed with the FCA that customers lacked the necessary information or experience to assess potential alternative options when voting on the scheme.
(Reporting by Kirstin Ridley and Iain Withers; editing by Rachel Armstrong, Mark Potter and Jason Neely)
By Kirstin Ridley and Iain Withers