Ethereum Devs Issue Final Report on Finality SNAFU

Loss of finality a week ago led to ‘the lowest participation of any epoch in Mainnet ever,’ developers say

article-image

alfernec/Shutterstock, modified by Blockworks

share

One week ago, the Ethereum network encountered a finality glitch that caused significant angst within the project’s developer community.

The problem stemmed from two consensus clients, including one of the leading clients, Prysm, spending too much computing resources on processing “mostly worthless” block attestations —  a part of the proof-of-stake consensus mechanism.

In a report issued by the team behind Prysm, the failure to finalize — while largely unnoticed by users — was worse the second time around on Friday. In one epoch, client participation fell to 30.7%, which the Prysm team says was “the lowest participation of any epoch in [Ethereum] Mainnet ever.”

The Prysm client is run on 37.6% of Ethereum validating nodes, according to clientdiversity.org. It is developed by Prysmatic Labs, which was acquired by Offchain Labs — the developers of Arbitrum — in October 2022.

An Ethereum epoch is roughly 6.4 minutes and comprises 32 “slots” — read blocks in the blockchain — of 12 seconds each. The network failed to finalize on Friday for nine epochs (nearly an hour). This was long enough to trigger the first-ever “inactivity leak” mode, which penalizes validators that have stopped contributing to consensus. 

But the damage was inconsequential: about $0.27 per validator.

“In total, we estimate that 28 ETH of penalties were applied and validators missed 55 ETH or more of potential revenue,” the Prysm team said. “This is less than 0.00015 ETH per validator.”

What went wrong

Prysm’s client began to struggle to handle incoming attestations with target checkpoints two epochs behind the then-current one. These valid attestations required clients to recalculate previous states of Ethereum’s Beacon chain and caused Prysm clients to exhaust their resources, resulting in a failure to meet validator clients’ requests promptly.

Ironically, contributing to the client’s woes was the huge success of Ethereum staking post-Shapella, when withdrawals of staked ether became possible for the first time.

A sharp increase in new validators requesting to join the Beacon chain since the start of May has pushed ETH staked net of withdrawals up by about 1.3 million, to a total 20.5 million — a value in dollar terms of more than $37 billion, on-chain data shows.

That has lengthened the queue to join the network by nearly 60,000 validators, meaning a new validator requesting to join today will not be active until some time in late June.

Beacon chain clients were stressed by all these incoming ether staking deposits, the Offchain Labs report explained.

“When the beacon state was smaller, Prysm would be able to handle these attestations and recover appropriately. However, with the large spike in deposits and the growing validator registry size, Prysm was unable to recover this time,” the team said.

Bug squashed

The Prysm client now ignores attestations for old blocks, which the report says are generally of no value to the network. Minority client Teku implemented a similar fix last weekend.

The current majority consensus client, Lighthouse, already was programmed to drop the problematic attestations and suffered no loss of liveness. The mere existence of client diversity has been largely credited with allowing the network to recover on its own without any manual intervention.

Aside from the technical stress on the software and hardware, the incident also took a toll on some members of the Ethereum community.

Loading Tweet..

Ethereum’s “community health consultant,” known as Superphiz, for one, declared he was taking some time off from Twitter, to “regain some physical fitness routines and mental clarity.”


Start your day with top crypto insights from David Canellis and Katherine Ross. Subscribe to the Empire newsletter.

Explore the growing intersection between crypto, macroeconomics, policy and finance with Ben Strack, Casey Wagner and Felix Jauvin. Subscribe to the Forward Guidance newsletter.

Get alpha directly in your inbox with the 0xResearch newsletter — market highlights, charts, degen trade ideas, governance updates, and more.

The Lightspeed newsletter is all things Solana, in your inbox, every day. Subscribe to daily Solana news from Jack Kubinec and Jeff Albus.

Tags

Upcoming Events

Javits Center North | 445 11th Ave

Tues - Thurs, March 18 - 20, 2025

Blockworks’ Digital Asset Summit (DAS) will feature conversations between the builders, allocators, and legislators who will shape the trajectory of the digital asset ecosystem in the US and abroad.

Brooklyn, NY

TUES - THURS, JUNE 24 - 26, 2025

Permissionless IV serves as the definitive gathering for crypto’s technical founders, developers, and builders to come together and create the future.If you’re ready to shape the future of crypto, Permissionless IV is where it happens.

Old Billingsgate

Mon - Wed, October 13 - 15, 2025

Blockworks’ Digital Asset Summit (DAS) will feature conversations between the builders, allocators, and legislators who will shape the trajectory of the digital asset ecosystem in the US and abroad.

recent research

Research Report Templates.jpg

Research

The L2 ecosystem has evolved into a complex landscape, with over $17B in market value and $50B in secured assets. While traditionally seen as high-beta plays on Ethereum, no L2 token achieved a beta higher than 1.0 relative to ETH in 2024. Furthermore, token dilution significantly impacted the sector, with a 1% increase in circulating supply corresponding to a 1.4% decrease in returns.

Key figures including Vitalik Buterin struggle to communicate the network’s core mission in an era of memecoin-minimized attention spans

article-image

World Liberty Financial has been busy buying crypto ahead of its launch

article-image

Trump’s token is setting the “tone” for crypto, Empire co-host Santiago Santos said

article-image

A pair of fundraises by Tapestry and Oh both came with expansive ideas about the future of AI and the role of humans in it

article-image

The Calamos Bitcoin Structured Alt Protection ETF — January is set to offer “systematic risk management” across its roughly one-year outcome period

article-image

Crypto industry elites (and a journalist or two) donned black tie at the Crypto Ball Friday night