Macauley Peterson

The Ethereum network <a href="https://blockworks.co/news/beacon-chain-temporarily-stops-finalizing">suffered a hiccup</a> Thursday; one that developers are still parsing for clues. But as a user, you would hardly have noticed.



At worst, it would have taken slightly longer for a transaction to be included in a block, and there was a somewhat lower certainty that a given action would be considered final — for about 25 minutes.



The blockchain’s developer community saw the incident as a strong test of the network’s resilience. It also served as further impetus to increase diversity of the independent software clients that make Ethereum run.



The extent of Ethereum’s client diversity is unique among major blockchains, and a cause championed by members of the community such as the pseudonymous Superphiz — known as the “Ethereum community health consultant” — who was one of the first to notice the issue.



“Barely a blip happened, but it was so uncharacteristic for the Ethereum network that it was a big deal anyway,” he <a href="https://twitter.com/superphiz/status/1657043885938421760?s=20">tweeted</a> Friday.



The same issue resurfaced again Friday, which Superphiz <a href="https://twitter.com/superphiz/status/1657076998676570119?s=20">called</a>, “not surprising,” adding, these things tend to happen in waves.



<h2 class="wp-block-heading">What happened?</h2>



The precise cause is not yet known, but the effect was a failure to finalize blocks. It’s likely to have involved a coincidence of “two or three issues,” according to Irina Timchenko, Ethereum blockchain manager at Everstake.



&#8220;During the incident, validators experienced issues with Prysm and Teku nodes. Some of these issues were resolved after a restart, while others returned to normal without requiring a restart,” Timchenko told Blockworks.



“Subsequently, validators noticed a spike in CPU usage, which appears to be a second-order effect rather than the root cause of the problem,” she added.



According to a Prysm developer known as Potuz, “several valid, but untimely attestations were broadcast in the network. This typically happens when a node that is struggling to be synced, attests with an old view of the chain.”



<figure class="wp-block-embed is-type-rich is-provider-twitter wp-block-embed-twitter"><div class="wp-block-embed__wrapper">
<blockquote class="twitter-tweet" data-width="550" data-dnt="true">This stressed consensus clients in different ways to the point that they, specially those running on weaker hardware, started struggling to follow the chain. 2/8&mdash; Potuz (@potuz1) <a href="https://twitter.com/potuz1/status/1657031086046601222?ref_src=twsrc%5Etfw">May 12, 2023</a></blockquote><script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
</div></figure>



A fix is already in the <a href="https://twitter.com/potuz1/status/1657031099501932544?s=20">works</a>, which has become more urgent since the problem has recurred.



<figure class="wp-block-embed is-type-rich is-provider-twitter wp-block-embed-twitter"><div class="wp-block-embed__wrapper">
<blockquote class="twitter-tweet" data-width="550" data-dnt="true">Mainnet is not finalizing again. Prysm is releasing a patch today. Will keep you updated!&mdash; terence.eth (@terencechain) <a href="https://twitter.com/terencechain/status/1657082775034560512?ref_src=twsrc%5Etfw">May 12, 2023</a></blockquote><script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
</div></figure>



Timchenko and others say the loss of finality could have been avoided entirely if less than a third of all validators ran the same consensus layer software client. Currently, two clients, Prysm and Lighthouse, are estimated to be operating on greater than a third of Ethereum nodes, according to clientdiversity.org.


<div class="wp-block-image">
<figure class="alignright size-medium"><img loading="lazy" decoding="async" width="748" height="792" src="https://blockworks-co.imgix.net/wp-content/uploads/2023/05/Screenshot-2023-05-12-at-3.01.00-PM.png?w=300&amp;h=300" alt="" class="wp-image-55553" srcset="https://blockworks-co.imgix.net/wp-content/uploads/2023/05/Screenshot-2023-05-12-at-3.01.00-PM.png?w=748 748w, https://blockworks-co.imgix.net/wp-content/uploads/2023/05/Screenshot-2023-05-12-at-3.01.00-PM.png?w=283 283w, https://blockworks-co.imgix.net/wp-content/uploads/2023/05/Screenshot-2023-05-12-at-3.01.00-PM.png?w=243 243w, https://blockworks-co.imgix.net/wp-content/uploads/2023/05/Screenshot-2023-05-12-at-3.01.00-PM.png?w=247 247w, https://blockworks-co.imgix.net/wp-content/uploads/2023/05/Screenshot-2023-05-12-at-3.01.00-PM.png?w=207 207w, https://blockworks-co.imgix.net/wp-content/uploads/2023/05/Screenshot-2023-05-12-at-3.01.00-PM.png?w=76 76w" sizes="(max-width: 748px) 100vw, 748px" /></figure></div>


Freddy Zwanzger, Ethereum ecosystem lead at Blockdaemon agrees.



“The fact that Ethereum does have multiple client implementations actually seemed to have prevented the blockchain from halting, and if all consensus clients would have less than 33% of the network, the finalization delay would also have been avoided,” Zwanzger told Blockworks.



Since blocks were still being produced, transactions were processed continuously, which is why end users didn’t see much, if any, effect, he added.



<h2 class="wp-block-heading">Liveness over safety</h2>



In designing blockchains with proof-of-stake consensus, developers have to make choices over how to prioritize one of two properties: safety and liveness.



Colloquially, safety is thought of as “avoiding bad things.” That means every node in the network agrees consistently on the state of the chain.



Ethereum prioritizes liveness, which is thought of as “good things happen eventually.” The blockchain is designed to be very difficult to halt.



<figure class="wp-block-embed is-type-rich is-provider-twitter wp-block-embed-twitter"><div class="wp-block-embed__wrapper">
<blockquote class="twitter-tweet" data-width="550" data-dnt="true">I developed a newfound appreciation for blockchain liveness (especially LMD-GHOST fork choice) since yesterday&#39;s incident. Ethereum didn&#39;t finalize for 3 epochs, but it kept going. Users were barely affected or noticed the finality gap.&mdash; terence.eth (@terencechain) <a href="https://twitter.com/terencechain/status/1657053102321836032?ref_src=twsrc%5Etfw">May 12, 2023</a></blockquote><script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
</div></figure>



It could have been worse if Prysm were still installed on upwards of 66% of nodes, which has been the case in the past.



When a consensus layer client causes a loss of finality — which is what happened Thursday — the network averts a fork (a “bad thing” in this case) so long it doesn’t have a supermajority client. But liveness was impacted because Prysm remains above the 33% threshold.



Ethereum’s specific consensus algorithm, called <a href="https://arxiv.org/pdf/2003.03052.pdf">Gasper</a>, saved the day, according to Gaetan Semp, a blockchain and DeFi consultant at Alyra.



“Gasper is truly an amazing consensus algorithm,” he <a href="https://twitter.com/GaetanSemp/status/1656769775207346176">tweeted</a>, noting that even without a supermajority of nodes, the blockchain continued to be validated by LMD-Ghost, a “weaker ‘short-term consensus’ as venture firm Paradigm has <a href="https://www.paradigm.xyz/2022/09/goldfish">called</a> it. LMD is short of “Latest Message Driven” and LMD-Ghost is part of Ethereum’s “<a href="https://celestia.org/glossary/fork-choice-rule">fork-choice rule</a>.”



This is a trade-off, however, and one which not all users would approve of, <a href="https://twitter.com/mattfiebach">Matt Fiebach</a> of Blockworks Research noted.



“Without the ability for nodes to reach consensus and achieve transaction finality, a user cannot be sure that their transaction will be included in the Ethereum canonical chain,” Fiebach said.



Alternative architectures from Solana and Cosmos favor a chain halt to preserve safety. “A 30-minute loss of finality is unacceptable for end users because it puts transactions at risk of being reverted,” Richard Patel, a software developer for <a href="https://jumpcrypto.com/firedancer/">Firedancer</a>, <a href="https://twitter.com/fd_ripatel/status/1656811632796872715?s=20">tweeted</a> in response to the Ethereum glitch.



Firedancer, developed by Jump Crypto, will soon become the first independent client for Solana.



Vance Spencer, co-founder at Framework Ventures described the failure to finalize as “a significant event,” but one with “a lot of nuance.”



“Finalization is especially important for entities like exchanges which need to ensure on-chain transactions aren’t reverted after they process them off-chain,” Spencer told Blockworks.



<h2 class="wp-block-heading">A resilience test</h2>



Marius van der Wijden, who develops the Go Ethereum (or Geth) execution layer client for Ethereum, <a href="https://twitter.com/vdWijden/status/1656790867506520064?s=20">called</a> the incident “a great fire drill” and explained that having client diversity was key to the ability of the network to heal itself.



“The different implementations react slightly [differently] to extreme scenarios which provides a huge boost in reliability,” van der Wijden told Blockworks. “Having different implementations means we can test [them] against each other, which can help us quickly pinpointing issues and determining the root cause of an issue,” he added.



Van der Wijden was confident that proof-of-work consensus algorithms would have performed “much worse” under the circumstances.



Industry observers, such as Fabrice Cheng, co-founder and CEO at Quadrata, <a href="https://twitter.com/fabdaRice/status/1656812550858887168?s=20">sounded</a> an optimistic note about the long-term prospects of the Ethereum network, coming out of this affair.



“Even when over 60% of all nodes went offline, the longest time you had you wait for your transaction to be included was [1 minute and] 24 seconds,” he said.



Friday’s recurrence was more severe, but developers note that block production continues unabated.



<figure class="wp-block-embed is-type-rich is-provider-twitter wp-block-embed-twitter"><div class="wp-block-embed__wrapper">
<blockquote class="twitter-tweet" data-width="550" data-dnt="true">tl;dr -Blocks still chugging along, just increase missed slots -Longer finality delay than yesterday. Inactivity leak kicking in to slowly, but continuously incentivize validators (<a href="https://t.co/nnU2BZPInL">https://t.co/nnU2BZPInL</a>) -Prysm looking to push update today (<a href="https://t.co/QM7FnVirJP">https://t.co/QM7FnVirJP</a>) -Elite…&mdash; Joe | 0xOsprey.eth (@0x_Osprey) <a href="https://twitter.com/0x_Osprey/status/1657089732470517760?ref_src=twsrc%5Etfw">May 12, 2023</a></blockquote><script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
</div></figure>



Updated May 12, 2023, at 5:11 pm ET: Additional detail and comments.



<hr class="wp-block-separator has-alpha-channel-opacity"/>



Start your day with top crypto insights from David Canellis and Katherine Ross. <a href="https://blockworks.co/newsletter/empire">Subscribe to the Empire newsletter</a>.



Explore the growing intersection between crypto, macroeconomics, policy and finance with Ben Strack, Casey Wagner and Felix Jauvin. <a href="https://blockworks.co/newsletter/onthemargin">Subscribe to the Forward Guidance newsletter</a>.



Get alpha directly in your inbox with the <a href="https://blockworks.co/newsletter/research">0xResearch newsletter</a> — market highlights, charts, degen trade ideas, governance updates, and more.



The Lightspeed newsletter is all things Solana, in your inbox, every day. <a href="https://blockworks.co/newsletter/lightspeed">Subscribe to daily Solana news</a> from Jack Kubinec and Jeff Albus.

Ethereum Finality Hiccup a ‘Fire Drill’ That ‘Could Have Been Avoided’

<div id="highlights-block_62bdd4b2a6493" class="highlights">
 <ul>
 <li>The hard fork pushed back the so-called “difficulty bomb,” which had been gradually slowing Ethereum block production, improving performance in the short term</li>
 <li>The Merge has not been scheduled, but the new timing of the difficulty bomb points to when Ethereum core developers think it will take effect</li>
 </ul>
 
</div>



Ethereum’s scheduled Grey Glacier hard fork went into effect on Thursday at 6:54 am ET — <a href="https://etherscan.io/block/15050000" target="_blank" rel="noreferrer noopener">Block 15050000</a> to be precise — and caused no problems, according to Ethereum client developer <a href="https://nethermind.io/" target="_blank" rel="noreferrer noopener">Nethermind</a> and the Ethereum Foundation’s <a href="https://twitter.com/TimBeiko/status/1542463390475423745" target="_blank" rel="noreferrer noopener">Tim Beiko</a>.&nbsp;



The sole purpose of the fork was to postpone the so-called <a href="https://ethereum.org/en/glossary/#difficulty-bomb" target="_blank" rel="noreferrer noopener">difficulty bomb</a> by about three months. That remains the most likely time frame for the hotly anticipated <a href="https://cms.blockworks.co/shadow-forks-and-testnets-the-ethereum-merge-approaches/" target="_blank" rel="noreferrer noopener">Merge with the proof-of-stake Beacon Chain</a>. A hard <a href="https://www.blockworksresearch.com/glossary/?search=fork" target="_blank" rel="noreferrer noopener">fork</a> is a software upgrade that splits a blockchain in two, with all network participants coordinating around the new chain.



Learn More: <a href="https://cms.blockworks.co/the-investors-guide-to-the-ethereum-merge/">The Investor&#8217;s Guide to the Ethereum Merge</a> 



The difficulty bomb doesn’t “go off” all at once, but it gradually forces an “exponential increase in proof-of-work difficulty setting designed to motivate the transition [from proof-of-work] to proof-of-stake,” according to ethereum.org. In essence, it deliberately increases the time it takes to make a new block of transactions, slowing down the network. As a result, miners have no incentive to continue mining the proof-of-work chain as it gradually grinds to a halt.



A prior hard fork, in December 2021, postponed the difficulty bomb until June, when the Ethereum community previously hoped to execute The Merge. So the Ethereum network was in fact slowing down; block production times had increased from the usual 12-14 seconds to 16 seconds in recent weeks.



The new network upgrade pushes back the bomb — also known as the Ice Age — by 700,000 Ethereum blocks, or around 100 days’ time, restoring the status quo.



The upgrade passed by so smoothly, it was barely mentioned during Thursday’s <a href="https://youtu.be/Fid8hTxkRHM?t=1177" target="_blank" rel="noreferrer noopener">Ethereum Core Developers Meeting</a> to discuss The Merge.



“Grey Glacier went fine, by the way,” Marius van der Wijden said during the call, noting that only a small fraction of Ethereum nodes had not upgraded in time.



Shortly before the hard fork, Ethereum passed another milestone, <a href="https://watchtheburn.com/" target="_blank" rel="noreferrer noopener">burning 2.5 million ETH</a>, thanks to <a href="https://cms.blockworks.co/?s=eip+1559" target="_blank" rel="noreferrer noopener">EIP-1559</a>, an upgrade activated in August 2021. That represents over $2.5 billion in value, which offsets some of the ETH issuance through proof-of-work. After The Merge ether issuance is expected to drop by as much as 90%.



If it doesn’t get postponed again, the new difficulty bomb will start to take effect in October, giving developers a clear target by when to deliver The Merge upgrade and execute the historic transition to a proof-of-stake regime.



<hr class="wp-block-separator has-alpha-channel-opacity"/>



Start your day with top crypto insights from David Canellis and Katherine Ross. <a href="https://blockworks.co/newsletter/empire">Subscribe to the Empire newsletter</a>.



Explore the growing intersection between crypto, macroeconomics, policy and finance with Ben Strack, Casey Wagner and Felix Jauvin. <a href="https://blockworks.co/newsletter/onthemargin">Subscribe to the Forward Guidance newsletter</a>.



Get alpha directly in your inbox with the <a href="https://blockworks.co/newsletter/research">0xResearch newsletter</a> — market highlights, charts, degen trade ideas, governance updates, and more.



The Lightspeed newsletter is all things Solana, in your inbox, every day. <a href="https://blockworks.co/newsletter/lightspeed">Subscribe to daily Solana news</a> from Jack Kubinec and Jeff Albus.

beacon Chain