In a statement to the London Stock Exchange, Ashley, who owns 61 percent of Sports Direct's equity and is the firm's CEO, criticised shareholders for failing to support the board and "repeatedly hounding" former chairman Keith Hellawell.
At Wednesday's annual shareholders' meeting, the re-election of Ashley was opposed by 9.78 percent of votes cast, while Hellawell, the company's chairman for the past nine years, stepped down just ahead of the meeting.
Their re-election had been opposed by three shareholder advisory groups over Sports Direct's alleged poor corporate governance and continuing concerns about employment practices.
"The company's shareholders appear to be affected by the pressure of the media and certain other organisations, and they have failed to support Sports Direct, Keith and myself, on this journey," Ashley, who also owns English Premier League soccer club Newcastle United, said.
"It is blatantly apparent that true entrepreneurs will never be accepted in the public arena ... the shareholders have now made it extremely challenging for future engagement to take place," he added.
Ashley said the people within the Sports Direct group were his "number one priority".
Since Sports Direct floated in 2007, there has been periodic speculation that Ashley might take the firm private.
However, analysts say he wants to retain Sports Direct's listed status because he believes it gives the firm gravitas in its negotiations with key suppliers - Nike, Adidas, Puma and Under Armour.
Shares in Sports Direct, down 13 percent over the last year, were up 1 percent at 352 pence at 1218 GMT, valuing the group at about 1.9 billion pounds.
On Wednesday, Sports Direct officially ruled out a takeover bid for Debenhams after an outgoing director said the board had discussed combining the department store group with House of Fraser.
Sports Direct purchased House of Fraser out of administration last month and also owns a 29.7 percent stake in Debenhams.
(Reporting by James Davey, Editing by Paul Sandle and Mark Potter)