Ethereum Devs Issue Final Report on Finality SNAFU

Loss of finality a week ago led to ‘the lowest participation of any epoch in Mainnet ever,’ developers say


alfernec/Shutterstock, modified by Blockworks


One week ago, the Ethereum network encountered a finality glitch that caused significant angst within the project’s developer community.

The problem stemmed from two consensus clients, including one of the leading clients, Prysm, spending too much computing resources on processing “mostly worthless” block attestations —  a part of the proof-of-stake consensus mechanism.

In a report issued by the team behind Prysm, the failure to finalize — while largely unnoticed by users — was worse the second time around on Friday. In one epoch, client participation fell to 30.7%, which the Prysm team says was “the lowest participation of any epoch in [Ethereum] Mainnet ever.”

The Prysm client is run on 37.6% of Ethereum validating nodes, according to It is developed by Prysmatic Labs, which was acquired by Offchain Labs — the developers of Arbitrum — in October 2022.

An Ethereum epoch is roughly 6.4 minutes and comprises 32 “slots” — read blocks in the blockchain — of 12 seconds each. The network failed to finalize on Friday for nine epochs (nearly an hour). This was long enough to trigger the first-ever “inactivity leak” mode, which penalizes validators that have stopped contributing to consensus. 

But the damage was inconsequential: about $0.27 per validator.

“In total, we estimate that 28 ETH of penalties were applied and validators missed 55 ETH or more of potential revenue,” the Prysm team said. “This is less than 0.00015 ETH per validator.”

What went wrong

Prysm’s client began to struggle to handle incoming attestations with target checkpoints two epochs behind the then-current one. These valid attestations required clients to recalculate previous states of Ethereum’s Beacon chain and caused Prysm clients to exhaust their resources, resulting in a failure to meet validator clients’ requests promptly.

Ironically, contributing to the client’s woes was the huge success of Ethereum staking post-Shapella, when withdrawals of staked ether became possible for the first time.

A sharp increase in new validators requesting to join the Beacon chain since the start of May has pushed ETH staked net of withdrawals up by about 1.3 million, to a total 20.5 million — a value in dollar terms of more than $37 billion, on-chain data shows.

That has lengthened the queue to join the network by nearly 60,000 validators, meaning a new validator requesting to join today will not be active until some time in late June.

Beacon chain clients were stressed by all these incoming ether staking deposits, the Offchain Labs report explained.

“When the beacon state was smaller, Prysm would be able to handle these attestations and recover appropriately. However, with the large spike in deposits and the growing validator registry size, Prysm was unable to recover this time,” the team said.

Bug squashed

The Prysm client now ignores attestations for old blocks, which the report says are generally of no value to the network. Minority client Teku implemented a similar fix last weekend.

The current majority consensus client, Lighthouse, already was programmed to drop the problematic attestations and suffered no loss of liveness. The mere existence of client diversity has been largely credited with allowing the network to recover on its own without any manual intervention.

Aside from the technical stress on the software and hardware, the incident also took a toll on some members of the Ethereum community.

Loading Tweet..

Ethereum’s “community health consultant,” known as Superphiz, for one, declared he was taking some time off from Twitter, to “regain some physical fitness routines and mental clarity.”

Get the day’s top crypto news and insights delivered to your email every evening. Subscribe to Blockworks’ free newsletter now.

Want alpha sent directly to your inbox? Get degen trade ideas, governance updates, token performance, can’t-miss tweets and more from Blockworks Research’s Daily Debrief.

Can’t wait? Get our news the fastest way possible. Join us on Telegram and follow us on Google News.


upcoming event

MON - WED, MARCH 18 - 20, 2024

Digital Asset Summit (DAS) is returning March 2024. This year’s event will be held in our nation’s capital, where industry leaders, policymakers, and institutional experts will come together to discuss the latest developments and challenges in the ever-evolving world of cryptocurrency. […]

upcoming event

MON - WED, SEPT. 11 - 13, 2023

2022 was a meme.Skeptics danced, believers believed.Eventually, newcomers turned away, drained of liquidity and hope.Now, the tide is shifting and it’s time to rebuild. Permissionless II is the brainchild of Blockworks and Bankless. It’s not just a conference, but a call […]

recent research

Cosmos Hub: ATOM Economic Zone


Replicated Security, the Hub’s Validation-as-a-Service offering that went live in March, is the first step in bringing value accrual to ATOM stakers.



OP holders might not be feeling very optimistic with the rollup’s token unlock schedule now in effect


Two lords are pushing for the government to ensure that the Bank of England could not implement “Britcoin” without legislative backing


This is the second fee switch proposal that failed to pass a community vote


This partnership comes as Nike released its first NFT collection, Our Force 1, this month


The weaknesses of large language models like ChatGPT are “too great to use reliably for security,” OpenZeppelin’s machine learning lead says


Giddy will now enable users to spend their yield on real-world goods and services