Solana outage caused by a previously identified bug, devs say

The report, by Anza, detailed the technical issue behind the Tuesday stoppage

article-image

Vladimir Kazakov/Shutterstock modified by Blockworks

share

A post-mortem report on this week’s Solana network outage pointed the finger at a previously known bug.

The network was unusable for five hours Tuesday. Validators — a key infrastructure layer of Solana, responsible for running the software that adds and verifies transactions to the blockchain — later restarted the network before 10 am ET.

The post-mortem, penned by Anza, concluded that a bug resulted in an infinite loop, which caused the validators to stall on a certain block. An infinite loop is a type of error that occurs when a condition to terminate a loop is never met, causing it to run indefinitely. In the context of the Solana network, such a bug within the validators’ operations would lead to them being stuck in a continuous cycle of attempting to process the same block, thereby preventing the network from confirming more transactions. 

Specifically, Solana’s Just-in-Time (JIT) compilation cache encountered a bug, leading to an infinite loop of recompilations for some older programs, thus monopolizing network resources and halting operations. 

Read more: Solana price dips as outage requires a network restart

As a result, the compiler — meant to enhance performance by compiling code in real time — became a bottleneck. To resolve this, the Solana team deployed a fix in a new release, addressing the bug and preventing such loops. 

The issue was “consistent with a bug that had been identified during the investigation of a recent Devnet outage and for which a patch was to be imminently deployed.” ​​The problem was first flagged to the Solana security team in April 2022.

Read more: Solana’s DeFi users mostly unbothered by network outage

“This fix eliminates the ability to create the preconditions required to trigger the bug, which was a simpler resolution. A more complete fix will be included with further improvements to LoadedPrograms and allowed to stabilize with the regular release cycle,” the report explained. 

This was the sixth major outage incurred by Solana since its inception. The network’s last outage took place in February of last year. 

As Blockworks’ David Canellis noted at the time, the price of SOL — Solana’s native asset — dipped briefly on the news of the outage. But it recovered swiftly, and according to market data, the event indicated that market responses to Solana network outages have become more muted over time.


Start your day with top crypto insights from David Canellis and Katherine Ross. Subscribe to the Empire newsletter.

The Lightspeed newsletter is all things Solana, in your inbox, every day. Subscribe to daily Solana news from Jack Kubinec and Jeff Albus.

Tags

Upcoming Events

Salt Lake City, UT

MON - TUES, OCT. 7 - 8, 2023

Blockworks and Bankless in collaboration with buidlbox are excited to announce the second installment of the Permissionless Hackathon – taking place October 7-8 in Salt Lake City, Utah. We’ve partnered with buidlbox to bring together the brightest minds in crypto for […]

Salt Lake City, UT

WED - FRI, OCTOBER 9 - 11, 2024

Pack your bags, anon — we’re heading west! Join us in the beautiful Salt Lake City for the third installment of Permissionless. Come for the alpha, stay for the fresh air. Permissionless III promises unforgettable panels, killer networking opportunities, and mountains […]

recent research

Research Report Cover Vertex.jpg

Research

The proliferation of new perp DEXs has led to fragmented liquidity across various DEXs and chains. Vertex, known for its vertically-integrated DEX that includes spot, perpetual, and integrated money markets, is now tackling cross-chain liquidity fragmentation through horizontal integration with the launch of new Edge instances. Vertex's integrated offerings and cross-margined account structure amplify the benefits of new instances: native cross-chain spot trading, optimized cross-chain basis trading, consistent interest rates, reduced bridging friction, and more.

article-image

Partnering with EtherFi and Angle, the fully on-chain perp DEX features bespoke collateral

article-image

Sponsored

Gavin Wood introduced the next evolutionary step for the Polkadot network: the Join-Accumulate Machine, or JAM

article-image

The side events were the places to be at Consensus 2024, according to attendees

article-image

Also, who’s come out swinging in the spot ether ETF fee war — and who could undercut them

article-image

I know it is not in their nature, but US regulators could learn a lot by researching the digital asset frameworks that overseas regulators have already gotten right

article-image

Also, the ETF hype train can count out at least one member