Ethereum’s Shapella Upgrade Went Through Despite ‘Ridiculous Bug’

The latest All Core Developer call highlighted an issue with Ethereum’s leading consensus client, Prysm


sdecoret/Shutterstock modified by Blockworks


Ethereum core developers met for their first group call since the activation of the Shanghai-Capella fork on Thursday to prepare for the next phase of the network’s evolution.

Tim Beiko, who runs protocol meetings for Ethereum, briefly congratulated the assembled devs before getting down to business.

“Great work everyone on Shapella — it worked,” Beiko said.

But there was one issue of consequence: A bug in the main execution client Prysm, related to MEV-Boost, caused validators to fail to propose blocks — a crucial part of the normal functioning of the blockchain.

The Prysm team issued a post-mortem analysis on Friday, identifying the root cause as a missing field during the “unblinding” of the “builder blinded block.”

In the post-Merge Ethereum blockchain, the “builder blinded block” is a type of block that has been processed by a tool called the “builder” to hide certain information. “Unblinding” this block refers to the process of revealing the hidden information using a cryptographic technique. The missing field — BlsToExecutionChanges — was supposed to be populated during this process, but it was not, which caused the issue.

“It’s safe to say that most of the missed blocks were due to this bug,” pseudonymous Prysm developer Potuz told his colleagues on the Thursday call.

This bug was difficult to detect when Shapella was activated on testnets, like Goerli.

Luckily, a “circuit breaker” triggered early on, and relayer teams reacted quickly to temporarily disable Prysm support from most relayers, preventing the issue from having a wider impact on network operation. 

However, the incident highlighted several lessons learned, including the need for more comprehensive testing, and closer collaboration with other teams, such as Flashbots.

“We should set up a way that all of our coding is oriented towards testing the builder,” Potuz said. “This is a ridiculous bug that should not have happened at all. There’s many places where this should have been tested, and all of them failed at the same time. This is impressive.”

A retrospective analysis of the Goerli testnet discovered evidence of the bug there, but the effect was not significant enough in the test environment to attract developer attention, the Prysm team explained on the call.

Around 40% of the Ethereum network nodes run Prysm software, but that percentage has been gradually shrinking as the Ethereum community recognized the need for client diversity. Other clients, Lighthouse, Teku, Nimbus and Lodestar picked up the slack to keep the chain running.

With Shapella safely in the rear view mirror, developers can now turn their full attention to the next major upgrade, known as Cancun-Deneb, or “Dencun” for short. Cancun, Mexico, was the site of the Devcon 3 conference, in 2017, and Deneb is a star in constellation Cygnus, the swan and a vertex of the Summer Triangle.

Get the day’s top crypto news and insights delivered to your email every evening. Subscribe to Blockworks’ free newsletter now.

Want alpha sent directly to your inbox? Get degen trade ideas, governance updates, token performance, can’t-miss tweets and more from Blockworks Research’s Daily Debrief.

Can’t wait? Get our news the fastest way possible. Join us on Telegram and follow us on Google News.


upcoming event

MON - WED, MARCH 18 - 20, 2024

Digital Asset Summit (DAS) is returning March 2024. This year’s event will be held in our nation’s capital, where industry leaders, policymakers, and institutional experts will come together to discuss the latest developments and challenges in the ever-evolving world of cryptocurrency. […]

upcoming event

MON - WED, SEPT. 11 - 13, 2023

2022 was a meme.Skeptics danced, believers believed.Eventually, newcomers turned away, drained of liquidity and hope.Now, the tide is shifting and it’s time to rebuild. Permissionless II is the brainchild of Blockworks and Bankless. It’s not just a conference, but a call […]

recent research

The State of LSTFi


There are five broad use cases for LSTs that are gaining traction alongside growth in demand: leverage farming, liquidity providing, LST baskets, stablecoin collateral, and interest rate derivatives.




Web3 real estate investing platform Parcl leverages blockchain to address the current bottlenecks facing property investing


Over 130 family office professionals surveyed noted a 90% client interest in crypto, a finding at odds with Goldman Sachs earlier this month


Newly announced Republican presidential hopeful Ron DeSantis claimed that the crypto industry would be doomed if President Biden is reelected


The 7-day average trading volumes for bitcoin have dropped to their lowest level in two and a half years alongside a muted derivatives market


Alexander Vinnik’s lawyers aim to swap his freedom for detained WSJ reporter Evan Gershkovich


This latest update will introduce immutability to token metadata but ensure that its key characteristics are preserved, and it will also introduce network fees