Ethereum’s Shapella Upgrade Went Through Despite ‘Ridiculous Bug’

The latest All Core Developer call highlighted an issue with Ethereum’s leading consensus client, Prysm

article-image

sdecoret/Shutterstock modified by Blockworks

share

Ethereum core developers met for their first group call since the activation of the Shanghai-Capella fork on Thursday to prepare for the next phase of the network’s evolution.

Tim Beiko, who runs protocol meetings for Ethereum, briefly congratulated the assembled devs before getting down to business.

“Great work everyone on Shapella — it worked,” Beiko said.

But there was one issue of consequence: A bug in the main execution client Prysm, related to MEV-Boost, caused validators to fail to propose blocks — a crucial part of the normal functioning of the blockchain.

The Prysm team issued a post-mortem analysis on Friday, identifying the root cause as a missing field during the “unblinding” of the “builder blinded block.”

In the post-Merge Ethereum blockchain, the “builder blinded block” is a type of block that has been processed by a tool called the “builder” to hide certain information. “Unblinding” this block refers to the process of revealing the hidden information using a cryptographic technique. The missing field — BlsToExecutionChanges — was supposed to be populated during this process, but it was not, which caused the issue.

“It’s safe to say that most of the missed blocks were due to this bug,” pseudonymous Prysm developer Potuz told his colleagues on the Thursday call.

This bug was difficult to detect when Shapella was activated on testnets, like Goerli.

Luckily, a “circuit breaker” triggered early on, and relayer teams reacted quickly to temporarily disable Prysm support from most relayers, preventing the issue from having a wider impact on network operation. 

However, the incident highlighted several lessons learned, including the need for more comprehensive testing, and closer collaboration with other teams, such as Flashbots.

“We should set up a way that all of our coding is oriented towards testing the builder,” Potuz said. “This is a ridiculous bug that should not have happened at all. There’s many places where this should have been tested, and all of them failed at the same time. This is impressive.”

A retrospective analysis of the Goerli testnet discovered evidence of the bug there, but the effect was not significant enough in the test environment to attract developer attention, the Prysm team explained on the call.

Around 40% of the Ethereum network nodes run Prysm software, but that percentage has been gradually shrinking as the Ethereum community recognized the need for client diversity. Other clients, Lighthouse, Teku, Nimbus and Lodestar picked up the slack to keep the chain running.

With Shapella safely in the rear view mirror, developers can now turn their full attention to the next major upgrade, known as Cancun-Deneb, or “Dencun” for short. Cancun, Mexico, was the site of the Devcon 3 conference, in 2017, and Deneb is a star in constellation Cygnus, the swan and a vertex of the Summer Triangle.


Start your day with top crypto insights from David Canellis and Katherine Ross. Subscribe to the Empire newsletter.

Explore the growing intersection between crypto, macroeconomics, policy and finance with Ben Strack, Casey Wagner and Felix Jauvin. Subscribe to the Forward Guidance newsletter.

Get alpha directly in your inbox with the 0xResearch newsletter — market highlights, charts, degen trade ideas, governance updates, and more.

The Lightspeed newsletter is all things Solana, in your inbox, every day. Subscribe to daily Solana news from Jack Kubinec and Jeff Albus.

Tags

Upcoming Events

Javits Center North | 445 11th Ave

Tues - Thurs, March 18 - 20, 2025

Blockworks’ Digital Asset Summit (DAS) will feature conversations between the builders, allocators, and legislators who will shape the trajectory of the digital asset ecosystem in the US and abroad.

recent research

Research Report Templates (6).png

Research

In recent months, a number of highly accretive developments were implemented across the protocol to improve fee capture, expand product functionality, and ultimately drive value accrual to the RUNE token, with more upgrades on the immediate horizon. These developments include hiking the minimum swap fee parameter to increase revenue, adding a Burn System Income Lever to reduce the RUNE supply, the addition of COSM-WASM smart contracting and IBC to enable an application layer, new chain integrations, and more.

article-image

Permissionless provokes thoughts on the future of crypto as folks engage in pull-up competitions

article-image

Plus, Casey sits with House Majority Whip Tom Emmer to get his views on crypto policy progress

article-image

Blockworks Advisory helps protocols coordinate their launch and growth through white-glove, bespoke services

article-image

The app has just two tabs, one for making swaps and one for viewing your wallet balance