By Anna Hirtenstein

Oil prices wobbled after a deal by OPEC and its allies to extend production cuts was offset by the prospect of increased output from Libya and U.S. shale producers.

West Texas Intermediate, the benchmark for U.S. crude, edged down 0.7% after briefly climbing above $40 a barrel, its highest price since early March. Brent crude, the gauge for international oil prices, wavered between gains and losses at just over $42 a barrel.

The new deal, struck Saturday by the Organization of the Petroleum Exporting Countries and its allies, calls for 23 countries to collectively reduce output by 9.6 million barrels a day until the end of July, amending and extending a historic agreement sealed in April. But Libya on Saturday restarted production at its largest oil field, while Mexico refused to continue with output curbs. American oil producers are also turning the oil taps back on.

"There's a couple of muting factors this morning," said Bjarne Schieldrop, chief commodities analyst at Nordic bank SEB Group. Libya's production boost and U.S. companies' decision to reopen shale wells come at an inconvenient time, threatening to boost supply when demand is still depressed, he said.

Libya, an OPEC member exempt from previous quotas because of a long-running civil war, ended a five-month shutdown at its Sharara oil field, according to government officials. That will add about 300,000 barrels a day in production.

Mexico refused to continue its curbs of 100,000 barrels a day beyond June as the government seeks to revive its oil industry.

The American shale industry is also ramping up production, complicating the crude market's recovery. Companies such as Parsley Energy Inc. and WPX Energy Inc. said they are turning their spigots back on after oil prices rallied in recent weeks. But investment in new drilling in the U.S. is still depressed, which may mean that this boost is short-lived.

The latest agreement also puts forward new, stricter rules around compliance, raising the threat of the deal potentially falling apart if some countries don't comply, analysts said. OPEC has frequently struggled to enforce its deals, particularly when it comes to members with weaker economies that are dependent on income from crude exports.

"Unusually, the updated agreement calls for compensation by recent overproducers through excess compliance in the next deal phases," said Jon Rigby, an oil analyst at UBS. "Indeed, continuation of the agreement has been made explicitly contingent on this."

This means that countries such as Iraq and Nigeria, which didn't reduce their output in line with the last deal's requirements, will have to cut more until the end of July, he said.

The stakes are too high for this deal to fail, given the need to support oil prices while lockdowns are eased and demand recovers, said Ole Hansen, head of commodity strategy at Saxo Bank.

"The group has become much more reluctant to accept cheating from others, especially Iraq and Nigeria," said Mr. Hansen. There is also still a risk of a second wave of the coronavirus, which could erode demand and threaten the oil market's recovery again, he said.

Meanwhile, Persian Gulf producers said at the latest meeting that they were already overcomplying by 1.2 million barrels a day in May. The revised deal doesn't extend those cuts by countries such as Saudi Arabia and the United Arab Emirates.

"This additional cut will thus be placed back into the market in July," Mr. Schieldrop said. That is "adding to the lack of 'oomph' in the oil price today," he added.

Write to Anna Hirtenstein at anna.hirtenstein@wsj.com