Arbitrum Network Bug Points to Experimental State of Rollups

Arbitrum failed to settle on Ethereum after a pre-programmed “fail-safe” meant a critical node didn’t receive enough ETH for gas fees

article-image

CryptoFX/Shutterstock, modified by Blockworks

share

Arbitrum, the Ethereum layer-2 scaling solution, stopped sending transactions to mainnet for roughly two hours on Wednesday due to a software bug affecting the optimistic rollup’s sequencer.

The snafu effectively left more than $2.2 billion in crypto in the lurch, locked inside dapps running on the blockchain, including GMX and Uniswap.

Arbitrum, like other rollup-centric networks, is intended to relieve Ethereum mainnet of activity by providing users an off-chain environment in which to transact. The company behind Arbitrum is Offchain Labs.

Every so often, Arbitrum must batch those transactions together and forward them to Ethereum for finalization. Code associated with a network node crucial to this process — the sequencer — was responsible for the outage. 

In order for the node to send the transactions to Ethereum, one of the two wallets associated with the node needs enough ETH to pay for transaction fees. 

Arbitrum developers explained in a tweet thread that a “malfunction” in the sequencer meant batches of transactions stopped being passed to the Ethereum mainnet. This interrupted an automated function that refilled the node’s wallet with ETH for transaction fees — effectively causing transactions to pile up for hours, awaiting settlement. 

“The Sequencer spent the ~5 ETH it had, but the gas-refunder (which had ~250 ETH) didn’t refill it until it posted a successful batch,” the developers tweeted.

Loading Tweet..

Despite the incident, the group described the system as “working as intended,” as the sequencer is only provided ETH if it can successfully post batches. “If there is a malfunction in the Sequencer, then it cannot spend its way through the high ETH balance of the gas refunder.”

The network was back processing transactions on Ethereum mainnet as normal as of 10:00 am ET, Wednesday. Since its Nitro upgrade in August 2022, Arbitrum has encountered 8 outages of between 30 minutes and 2 hours, according to blockchain bridge developer Shotaro Granzier-Nakajima.

Wednesday’s incident was not an outage of that magnitude, according to A.J. Warner, chief strategy Officer at Offchain Labs.

“There is no parallel. This was not sequencer downtime. This was a bug that caused a delay in batch posting,” Warner told Blockworks. In fact, Arbitrum “often delays posting to [layer-1] in times of high [layer-1] congestion,” Warner said, noting that alternative zk rollups can also take several hours before posting data to Ethereum mainnet under normal operation.

Offchain Labs said a full postmortem on the situation would be released later today.

News analysis by Macauley Peterson

All Ethereum layer-2’s currently operating, employ a centralized sequencer — in Arbitrum’s case one run by Offchain Labs. That’s not the way it’s meant to be forever, but moving further out on the decentralization spectrum is mostly aspirational at this stage of their development.

According to Arbitrum’s documentation, “eventually, sequencing affordances could be given to a distributed committee of sequencers which come to consensus on ordering.”

Blockworks Research’s timely report “Sequencers: The Key to The Rollup Investment Thesis” includes the key takeaway that “centralized sequencers bring concerns surrounding liveness, censorship resistance, and regulatory arbitrage for the L2s.”

Wednesday’s incident represents a potential censorship resistance failure. No transaction batches posted → no layer-1 finality.

The sequencer was misbehaving, although it was not strictly “down” and continued to process transactions without posting them to Ethereum mainnet.

In theory, there’s another route to getting layer-2 transactions processed by submitting them via layer-1 into what Arbitrum calls “the delayed Inbox.” That is a primary selling point of Ethereum layer-2s and the basis of the claim that they are “inheriting,” in some sense, the security of Ethereum’s mainnet.

Following a previous incident which did involve sequencer downtime in January 2022, the Arbitrum team wrote, “the network is designed to sustain even permanent Sequencer failures by falling back to using Ethereum to process transactions.”

That wasn’t necessary in this case.

Be that as it may, there is still a considerable amount of trust in Offchain Labs required, even after the launch of the ARB token and initial efforts to decentralize governance.

Until fraud proofs can be validated permissionlessly, rather than via whitelisted nodes, and solutions to the decentralized sequencer are found, tested, and battle hardened, there will always be additional risks with rollups. It’s a tradeoff users make for higher performance and lower transaction costs.

And plenty of users are making that tradeoff! Funds at risk have climbed from a July 2022 trough of about $560 million to a recent peak in May of $2.55 billion.

The Arbitrum token (ARB), with a current market cap of $1.44 billion, fell roughly 5% on the news, but has recovered about half of that since.

Updated: June 8, 2023, at 10:48 am ET with comments from Offchain Labs received after publication.


Get the news in your inbox. Explore Blockworks newsletters:

  • Blockworks Daily: The newsletter that helps thousands of investors understand crypto and the markets, by Byron Gilliam.
  • Empire: Start your morning with the top news and analysis to inform your day in crypto.
  • Forward Guidance: Reporting and analysis on the growing intersection of crypto and macroeconomics, policy and finance.
  • 0xResearch: Alpha directly in your inbox. Market highlights, data, degen trade ideas, governance updates, token performance and more.
  • Lightspeed: Built for Solana investors, developers and community members. The latest from one of crypto’s hottest networks.
  • The Drop: For crypto collectors and traders, covering apps, games, memes and more.
Tags

Upcoming Events

Javits Center North | 445 11th Ave

Tues - Thurs, March 18 - 20, 2025

Blockworks’ Digital Asset Summit (DAS) will feature conversations between the builders, allocators, and legislators who will shape the trajectory of the digital asset ecosystem in the US and abroad.

Industry City | Brooklyn, NY

TUES - THURS, JUNE 24 - 26, 2025

Permissionless IV serves as the definitive gathering for crypto’s technical founders, developers, and builders to come together and create the future.If you’re ready to shape the future of crypto, Permissionless IV is where it happens.

Old Billingsgate

Mon - Wed, October 13 - 15, 2025

Blockworks’ Digital Asset Summit (DAS) will feature conversations between the builders, allocators, and legislators who will shape the trajectory of the digital asset ecosystem in the US and abroad.

recent research

Unlocked by Template (4).png

Research

Wormhole Settlement allows for a highly scalable liquidity venue to fill user intents into a multichain, multi-VM future. By concentrating solvers’ balance sheets on Solana, transaction costs associated with solvers rebalancing inventory across destinations are eliminated. With the ability to settle bridging, swapping, and arbitrary interactions, without the costs and frictions of fragmenting solver liquidity, Wormhole Settlement has the opportunity to settle a large share of volumes in the crosschain interoperability market with a beneficial framework for both users and solvers. 

article-image

On Supply Shock, Asymmetric founder Dan Held discussed why Bitcoin DeFi will take market share from Solana, Ethereum and other top blockchains

article-image

Pillsbury partner Brian Montgomery said that banks are mulling how to gain exposure to crypto

article-image

The company has now acquired three Solana validator operators since its September pivot into Solana

article-image

Those hoping for an executive order, a bill draft, or a major announcement from the CFTC or SEC were disappointed

article-image

Uncertainty around the US economy’s outlook is spurring a risk-off wave

article-image

The team says they’re still building despite the massive weekend selloff