Ethereum’s Shapella Upgrade Went Through Despite ‘Ridiculous Bug’

The latest All Core Developer call highlighted an issue with Ethereum’s leading consensus client, Prysm

article-image

sdecoret/Shutterstock modified by Blockworks

share

Ethereum core developers met for their first group call since the activation of the Shanghai-Capella fork on Thursday to prepare for the next phase of the network’s evolution.

Tim Beiko, who runs protocol meetings for Ethereum, briefly congratulated the assembled devs before getting down to business.

“Great work everyone on Shapella — it worked,” Beiko said.

But there was one issue of consequence: A bug in the main execution client Prysm, related to MEV-Boost, caused validators to fail to propose blocks — a crucial part of the normal functioning of the blockchain.

The Prysm team issued a post-mortem analysis on Friday, identifying the root cause as a missing field during the “unblinding” of the “builder blinded block.”

In the post-Merge Ethereum blockchain, the “builder blinded block” is a type of block that has been processed by a tool called the “builder” to hide certain information. “Unblinding” this block refers to the process of revealing the hidden information using a cryptographic technique. The missing field — BlsToExecutionChanges — was supposed to be populated during this process, but it was not, which caused the issue.

“It’s safe to say that most of the missed blocks were due to this bug,” pseudonymous Prysm developer Potuz told his colleagues on the Thursday call.

This bug was difficult to detect when Shapella was activated on testnets, like Goerli.

Luckily, a “circuit breaker” triggered early on, and relayer teams reacted quickly to temporarily disable Prysm support from most relayers, preventing the issue from having a wider impact on network operation. 

However, the incident highlighted several lessons learned, including the need for more comprehensive testing, and closer collaboration with other teams, such as Flashbots.

“We should set up a way that all of our coding is oriented towards testing the builder,” Potuz said. “This is a ridiculous bug that should not have happened at all. There’s many places where this should have been tested, and all of them failed at the same time. This is impressive.”

A retrospective analysis of the Goerli testnet discovered evidence of the bug there, but the effect was not significant enough in the test environment to attract developer attention, the Prysm team explained on the call.

Around 40% of the Ethereum network nodes run Prysm software, but that percentage has been gradually shrinking as the Ethereum community recognized the need for client diversity. Other clients, Lighthouse, Teku, Nimbus and Lodestar picked up the slack to keep the chain running.

With Shapella safely in the rear view mirror, developers can now turn their full attention to the next major upgrade, known as Cancun-Deneb, or “Dencun” for short. Cancun, Mexico, was the site of the Devcon 3 conference, in 2017, and Deneb is a star in constellation Cygnus, the swan and a vertex of the Summer Triangle.


Get the news in your inbox. Explore Blockworks newsletters:

Tags

Upcoming Events

Old Billingsgate

Mon - Wed, October 13 - 15, 2025

Blockworks’ Digital Asset Summit (DAS) will feature conversations between the builders, allocators, and legislators who will shape the trajectory of the digital asset ecosystem in the US and abroad.

Industry City | Brooklyn, NY

TUES - THURS, JUNE 24 - 26, 2025

Permissionless IV serves as the definitive gathering for crypto’s technical founders, developers, and builders to come together and create the future.If you’re ready to shape the future of crypto, Permissionless IV is where it happens.

Brooklyn, NY

SUN - MON, JUN. 22 - 23, 2025

Blockworks and Cracked Labs are teaming up for the third installment of the Permissionless Hackathon, happening June 22–23, 2025 in Brooklyn, NY. This is a 36-hour IRL builder sprint where developers, designers, and creatives ship real projects solving real problems across […]

recent research

Axiom report graphic.png

Research

Axiom has rapidly become the leading Solana trading bot, capturing over 50% of Solana bot volume and hovering at 3-4% of total Solana memecoin volume just three months post-launch, driven by high user activity and larger average trade sizes.

article-image

Bloomberg Intelligence analysts pegged the odds of the SEC approving US litecoin and solana ETFs in 2025 at 90%

article-image

Digital Assets Subcommittee Chair Bryan Steil called for a “roundtable” discussion in lieu of a formal hearing

article-image

The exchange sees a way to reach gamers through the deal

article-image

The drop gives buyers Adidas outfits for their in-game characters, but the game hasn’t fully released yet

article-image

In 1999, Daniel Bernstein fought for code to be protected, just like free speech