The 60-year-old Swede will join the Zurich-based maker of industrial drives and robots in February and take over as CEO in March, the company said on Sunday.
ABB, whose previous CEO Ulrich Spiesshofer left suddenly in April, is in the middle of overhauling its business following years of lacklustre profit and bungled acquisitions.
Chairman Peter Voser, who has taken on CEO functions on an interim basis, said on Monday he was confident he had got the right man.
"He has a great track record of empowering businesses in a decentralised way, that is what he has done at Sandvik and at his earlier businesses as well," Voser said.
"He is focused on driving performance," added the former CEO of oil major Royal Dutch Shell who will revert back to his chairman role when Rosengren takes charge.
During his four years at Sandvik, Rosengren revamped the metal-cutting tools and mining gear maker and sold off underperforming businesses.
Voser said Rosengren would follow ABB's strategy of reducing its corporate centre and giving more power to its four business units - industrial automation, electrification, motion and robotics and discrete automation.
"Over the last few months ... the company has outlined a strategy and we are following through on that," Voser said.
"But a strategy will not be cast in stone for five years."
ABB, whose shares have lost 14% over the last five years, is selling its power grids business to Hitachi and last month announced a deal to quit the solar energy inverters business.
Voser has launched a review of businesses which generate $3 billion (£2.49 billion) of revenue, or 11% of ABB's total, which could be improved or sold.
"With the appointment of Rosengren, ABB will have a CEO with a proven track record and a strong reputation in the capital markets for delivering operational improvement," said Bank of America Merrill Lynch analyst Alexander Virgo.
"In our view, his success at Sandvik in implementing a decentralised business model should make him a good match for ABB."
ABB's largest shareholder Investor AB also welcomed the appointment of Rosengren, who before Sandvik was CEO of Wartsila, which makes and services power sources for the marine and energy markets.
"From his previous leading roles in several companies, he has a proven track record in driving a decentralised culture, improving performance and delivering value," said Johan Forssell, CEO of Investor, which holds 11.2% of ABB.
Sandvik's shares have increased by 78% with Rosengren in charge, as margins improved, while Wartsila's share price increased 83% on his watch.
Activist investor Cevian Capital, ABB's second-biggest shareholder with a 5.3% stake, said ABB shares could almost double in value if Rosengren could lift margins and valuations to match rivals.
"If these gaps are closed, ABB should be worth more than 35 CHF (Swiss francs). That is Bjorn's main task," said Cevian managing partner Christer Gardell.
At 0940 GMT, ABB shares were up 4.3% at 18.29 francs
(Reporting by John Revill; Additional reporting by Simon Johnson in Stockholm; Editing by David Holmes and Mark Potter)
By John Revill