The decision is a further blow to the Australian Securities and Investments Commission (ASIC), after a public inquiry into the financial sector found it had failed to stop systemic wrongdoing by the major banks.

ASIC's announcement in September that Westpac had agreed to pay the fine was seen as a badly needed victory for the regulator as it attempts to rebuild its reputation as a tough cop on the beat.

But Federal Court Judge Nye Perram declined to allow the penalty on the grounds that ASIC had failed to prove the bank's wrongdoing and had not demonstrated how it had calculated the fine.

"It is unworkable to assess the reasonableness of the penalty if it is not known what is to be penalized," Perram said in a written judgment.

"The court should not be expected to approve a record penalty without being told why. I accept the need for the court to encourage settlements in this area but the desirability of doing so does not permit the court to become a rubber stamp."

An ASIC spokesman said the regulator was reviewing the judgment and declined to comment further.

A Westpac spokesman was not immediately available for comment.

On Nov. 9, in a separate matter, another federal court judge rejected ASIC's argument that Westpac should be fined A$58 million for inappropriately handling negotiations of a key bank rate.

The judge in that case said Westpac should pay just A$3.3 million, the amount the bank had offered to pay.

Westpac shares were down 5.4 percent on Tuesday morning, in a weaker overall market, after trading ex-dividend.

(Reporting by Byron Kaye; Editing by Stephen Coates)