Ethereum Devs Issue Final Report on Finality SNAFU

Loss of finality a week ago led to ‘the lowest participation of any epoch in Mainnet ever,’ developers say

article-image

alfernec/Shutterstock, modified by Blockworks

share

One week ago, the Ethereum network encountered a finality glitch that caused significant angst within the project’s developer community.

The problem stemmed from two consensus clients, including one of the leading clients, Prysm, spending too much computing resources on processing “mostly worthless” block attestations —  a part of the proof-of-stake consensus mechanism.

In a report issued by the team behind Prysm, the failure to finalize — while largely unnoticed by users — was worse the second time around on Friday. In one epoch, client participation fell to 30.7%, which the Prysm team says was “the lowest participation of any epoch in [Ethereum] Mainnet ever.”

The Prysm client is run on 37.6% of Ethereum validating nodes, according to clientdiversity.org. It is developed by Prysmatic Labs, which was acquired by Offchain Labs — the developers of Arbitrum — in October 2022.

An Ethereum epoch is roughly 6.4 minutes and comprises 32 “slots” — read blocks in the blockchain — of 12 seconds each. The network failed to finalize on Friday for nine epochs (nearly an hour). This was long enough to trigger the first-ever “inactivity leak” mode, which penalizes validators that have stopped contributing to consensus. 

But the damage was inconsequential: about $0.27 per validator.

“In total, we estimate that 28 ETH of penalties were applied and validators missed 55 ETH or more of potential revenue,” the Prysm team said. “This is less than 0.00015 ETH per validator.”

What went wrong

Prysm’s client began to struggle to handle incoming attestations with target checkpoints two epochs behind the then-current one. These valid attestations required clients to recalculate previous states of Ethereum’s Beacon chain and caused Prysm clients to exhaust their resources, resulting in a failure to meet validator clients’ requests promptly.

Ironically, contributing to the client’s woes was the huge success of Ethereum staking post-Shapella, when withdrawals of staked ether became possible for the first time.

A sharp increase in new validators requesting to join the Beacon chain since the start of May has pushed ETH staked net of withdrawals up by about 1.3 million, to a total 20.5 million — a value in dollar terms of more than $37 billion, on-chain data shows.

That has lengthened the queue to join the network by nearly 60,000 validators, meaning a new validator requesting to join today will not be active until some time in late June.

Beacon chain clients were stressed by all these incoming ether staking deposits, the Offchain Labs report explained.

“When the beacon state was smaller, Prysm would be able to handle these attestations and recover appropriately. However, with the large spike in deposits and the growing validator registry size, Prysm was unable to recover this time,” the team said.

Bug squashed

The Prysm client now ignores attestations for old blocks, which the report says are generally of no value to the network. Minority client Teku implemented a similar fix last weekend.

The current majority consensus client, Lighthouse, already was programmed to drop the problematic attestations and suffered no loss of liveness. The mere existence of client diversity has been largely credited with allowing the network to recover on its own without any manual intervention.

Aside from the technical stress on the software and hardware, the incident also took a toll on some members of the Ethereum community.

Loading Tweet..

Ethereum’s “community health consultant,” known as Superphiz, for one, declared he was taking some time off from Twitter, to “regain some physical fitness routines and mental clarity.”


Don’t miss the next big story – join our free daily newsletter.

Tags

Upcoming Events

Hilton Metropole | 225 Edgware Rd, London

Mon - Wed, March 18 - 20, 2024

Crypto’s premier institutional conference returns to London in March 2024. The DAS: London Experience: Attend expert-led panel discussions and fireside chats Hear the latest developments regarding the crypto and digital asset regulatory environment directly from policymakers and experts.

Salt Lake City, UT

WED - FRI, OCTOBER 9 - 11, 2024

Pack your bags, anon — we’re heading west! Join us in the beautiful Salt Lake City for the third installment of Permissionless. Come for the alpha, stay for the fresh air. Permissionless III promises unforgettable panels, killer networking opportunities, and mountains […]

recent research

Research report - cover graphics (1).jpg

Research

In this report, we dive into crypto private market data to gather insights on where the future of the industry is headed. Despite a notable downturn in private raises, capital continues to infuse promising projects that aim to transform payments, banking, consumer experiences, community, and more, with 2023 being the fourth-largest year for crypto venture capital.

article-image

BUZZ holds shares of Coinbase, Robinhood and MicroStrategy

article-image

Opinion: Even though I didn’t pay for my “Diamond Hands” burger with BTC, don’t let that fool you into thinking that crypto’s development is futile

article-image

The results mark “a major positive inflection point,” one analyst says, as the exchange carries net income momentum into a crypto rally

article-image

While the slate of 10 US spot bitcoin funds have tallied $4.6 billion of net inflows thus far, half of the field is lagging the leaders

article-image

Trading volumes totalled $154 billion in Q4, including $125 billion in institutional volume

article-image

DeFi on Bitcoin is all the rage right now and Stacks is positioned to benefit