Optimism outage unrelated to scheduled maintenance

The Optimism Mainnet blockchain experienced a series of outages and instabilities on Feb. 15, 2024, beginning around 6:00 am ET. The team started investigating reports of an “unsafe head stall,” meaning the sequencer stopped producing blocks.

Shortly after at 6:10 am ET, developers identified the cause of the issue and applied a fix that briefly restored operations.

However, the resolution was short-lived; by 6:26 am ET, reports of additional instability called into question the efficacy of the patch, according to the network’s status page.

The Optimism sequencer remains centralized, meaning OP Labs is responsible for identifying and troubleshooting problems.

Read more: Optimism devs tackle bad actors with fault-proof system testnet launch

After finding the suspected root cause, the network resumed around 8:00 am ET when the status page noted the incident was unrelated to the planned pause of OP Mainnet withdrawals — a scheduled maintenance window began at 9:01 am ET and ended one hour later.

Blockworks has contacted Optimism for details.

Arbitrum, the largest Ethereum rollup by total value locked (TVL), suffered a similar outage in December 2023.

Read more: Arbitrum suffers ‘partial outage’ amid traffic influx

Although the team stated that no action was required from users or node operators following the resolution, community members in Discord dispute that.

“I can also confirm that a full node restart of op-node and op-geth was required to get unstuck,” wrote one member hazim-j.eth, co-founder of Stackup, who works on account abstraction tooling.

“If the status page had advised a restart we might have been able to get back online a lot sooner,” he said.  

During the outage, node operators observed single-block reorganizations — or reorg — of the unsafe chain.

A single-block reorg, not uncommon in blockchain operations, occurs when the chain diverges into two paths due to discrepancies in block validation and then reconverges on a single, agreed-upon path.

In the case of Optimism, an “unsafe” head reorg refers to blocks produced by the sequencer, but not yet posted in batches to Ethereum mainnet, according to pseudonymous developer support engineer at OP Labs known as Soyboy Vegan.

According to Optimism’s documentation, “it is still possible for [unsafe transactions] to be excluded from the final blockchain if the Sequencer fails to publish the block to Ethereum in a timely manner.”

The planned withdrawal pause scheduled for 9:00 to 10:00 am ET was intended to “test Optimism’s incident response capabilities,” according to an OP Labs X post.

Updated on Feb. 15, 2024, at 11:05 am ET, with additional context.

---

Get the news in your inbox. Explore Blockworks newsletters:

• The Breakdown: Decoding crypto and the markets. Daily.
• 0xResearch: Alpha in your inbox. Think like an analyst.
• Empire: Crypto news and analysis to start your day.
• Forward Guidance: The intersection of crypto, macro and policy.
• The Drop: Apps, games, memes and more.
• Lightspeed: All things Solana.
• Supply Shock: Bitcoin, bitcoin, bitcoin.