Ethereum Devs Issue Final Report on Finality SNAFU

Loss of finality a week ago led to ‘the lowest participation of any epoch in Mainnet ever,’ developers say

article-image

alfernec/Shutterstock, modified by Blockworks

share

One week ago, the Ethereum network encountered a finality glitch that caused significant angst within the project’s developer community.

The problem stemmed from two consensus clients, including one of the leading clients, Prysm, spending too much computing resources on processing “mostly worthless” block attestations —  a part of the proof-of-stake consensus mechanism.

In a report issued by the team behind Prysm, the failure to finalize — while largely unnoticed by users — was worse the second time around on Friday. In one epoch, client participation fell to 30.7%, which the Prysm team says was “the lowest participation of any epoch in [Ethereum] Mainnet ever.”

The Prysm client is run on 37.6% of Ethereum validating nodes, according to clientdiversity.org. It is developed by Prysmatic Labs, which was acquired by Offchain Labs — the developers of Arbitrum — in October 2022.

An Ethereum epoch is roughly 6.4 minutes and comprises 32 “slots” — read blocks in the blockchain — of 12 seconds each. The network failed to finalize on Friday for nine epochs (nearly an hour). This was long enough to trigger the first-ever “inactivity leak” mode, which penalizes validators that have stopped contributing to consensus. 

But the damage was inconsequential: about $0.27 per validator.

“In total, we estimate that 28 ETH of penalties were applied and validators missed 55 ETH or more of potential revenue,” the Prysm team said. “This is less than 0.00015 ETH per validator.”

What went wrong

Prysm’s client began to struggle to handle incoming attestations with target checkpoints two epochs behind the then-current one. These valid attestations required clients to recalculate previous states of Ethereum’s Beacon chain and caused Prysm clients to exhaust their resources, resulting in a failure to meet validator clients’ requests promptly.

Ironically, contributing to the client’s woes was the huge success of Ethereum staking post-Shapella, when withdrawals of staked ether became possible for the first time.

A sharp increase in new validators requesting to join the Beacon chain since the start of May has pushed ETH staked net of withdrawals up by about 1.3 million, to a total 20.5 million — a value in dollar terms of more than $37 billion, on-chain data shows.

That has lengthened the queue to join the network by nearly 60,000 validators, meaning a new validator requesting to join today will not be active until some time in late June.

Beacon chain clients were stressed by all these incoming ether staking deposits, the Offchain Labs report explained.

“When the beacon state was smaller, Prysm would be able to handle these attestations and recover appropriately. However, with the large spike in deposits and the growing validator registry size, Prysm was unable to recover this time,” the team said.

Bug squashed

The Prysm client now ignores attestations for old blocks, which the report says are generally of no value to the network. Minority client Teku implemented a similar fix last weekend.

The current majority consensus client, Lighthouse, already was programmed to drop the problematic attestations and suffered no loss of liveness. The mere existence of client diversity has been largely credited with allowing the network to recover on its own without any manual intervention.

Aside from the technical stress on the software and hardware, the incident also took a toll on some members of the Ethereum community.

Loading Tweet..

Ethereum’s “community health consultant,” known as Superphiz, for one, declared he was taking some time off from Twitter, to “regain some physical fitness routines and mental clarity.”


Start your day with top crypto insights from David Canellis and Katherine Ross. Subscribe to the Empire newsletter.

Explore the growing intersection between crypto, macroeconomics, policy and finance with Ben Strack, Casey Wagner and Felix Jauvin. Subscribe to the On the Margin newsletter.

The Lightspeed newsletter is all things Solana, in your inbox, every day. Subscribe to daily Solana news from Jack Kubinec and Jeff Albus.

Tags

Upcoming Events

Salt Lake City, UT

MON - TUES, OCT. 7 - 8, 2024

Blockworks and Bankless in collaboration with buidlbox are excited to announce the second installment of the Permissionless Hackathon – taking place October 7-8 in Salt Lake City, Utah. We’ve partnered with buidlbox to bring together the brightest minds in crypto for […]

Salt Lake City, UT

WED - FRI, OCTOBER 9 - 11, 2024

Permissionless is a conference for founders, application developers, and users. Come meet the next generation of people building and using crypto.

recent research

Blinks Report Image.png

Research

Blinks enable the ability to vampire attack user monetization of existing networks by inserting onchain and financialized functionalities directly within the popular social feeds and digital experiences of today.

article-image

Kalshi founder Tarek Mansour said Thursday marked the “the first trade on regulated election markets in nearly a century”

article-image

I was excited about being on the precipice of realigning societal incentives and solving many issues plaguing our modern financial world

article-image

Cypherpunk Holdings has rebranded to Sol Strategies in a pivot to a Solana-first investment approach

article-image

BitGo’s wrapped bitcoin (wBTC) has a new custodial challenger

article-image

Make no mistake: Tether makes a ton of money. But exactly how much depends a lot on the price of bitcoin.

article-image

A new report on stablecoin activity in emerging markets shows their immense popularity in Nigeria