Avalanche operational again after 5 hour outage

Avalanche nodes recently began the Durango upgrade, which is scheduled to activate on its mainnet next month

article-image

Furkan Cubuk/Shutterstock, modified by Blockworks

share

Avalanche’s C-Chain stopped producing blocks at around 6:30 am ET Friday, due to a client code bug, developers have learned. The network resumed normal operations roughly 5 hours later.

Most transactions on the network take place on the C-Chain, however some subnets were also affected since all validators are required to process C-Chain transactions.

In an X post, Ava Labs co-founder Kevin Sekniqi attributed the likely cause to “a gossip-related mempool management bug, which is purely a code-related bug, and not an issue with performance handling,” meaning the outage was not related to a spike in usage of the network.

The initial working theory was that it was an “edge case” related to inscriptions.

Loading Tweet..

Shortly after the outage began, Avalanche’s network status page reported that “developers across the community are currently investigating.”

Avalanche validator operators have been in the process of upgrading their node software to a new version that would bring a major upgrade called Durango online.

The upgrade is expected to activate on March 6, 2024, and as of the time of the outage, roughly 17% of the 1,838 network validators were already running the new client production code version 1.11.0, according to data provider snowpeer.io.

That release of AvalancheGo was published on Wednesday, but it’s not thought to be the cause, according to Sekniqi.

“I do not think it is related to v1.11, it just seems to have hit an edge case in mempool processing,” Sekniqi told Blockworks. “The bug hit majority of nodes, it seems.”

By 11:00 am ET, Ava Labs engineering vice president reported the bug had been identified, writing on X that it resulted from “logic added in v1.10.18 that lead to validators sending an excessive amount of gossip to each other.”

Validators are encouraged to update to a newly released version 1.11.1. Over a quarter of the network had done so by 11:45 am ET, while the percentage of validators running the buggy version dipped to 19%, allowing the network to come back online snowpeer.io showed.

Subnet validators currently must also validate the C-Chain, which is why the outage also affected subnets, although this architecture is slated to change. That can’t come soon enough according to Avalanche director Omer Demirel.

“Today’s outage is a good example of why we need ACP-13 (Subnet-Only Validators), which will improve the Avalanche network’s fault tolerance by relieving subnets from primary network validation duty,” Demirel posted to X.

Loading Tweet..

The outage bears resemblance to one that knocked Solana offline earlier this month.

Read more: Solana price dips as outage requires a network restart

In that instance, a software patch followed by a coordinated restart was required before service was restored several hours later.

According to community manager Leandro Matos on the Avalanche Discord, the outage is unusual. “The chain only had one minor outage since 2020 which lasted a few minutes,” he said.

Avalanche’s native currency AVAX as down 3.2% on the day to around $35.60 as of 11:30 am ET.

Updated Feb. 23, 2024 at 8:38, 9:24, 10:27 and 11:32 am ET with additional context and 11:46 am ET revising the headline.


Start your day with top crypto insights from David Canellis and Katherine Ross. Subscribe to the Empire newsletter.

Tags

Upcoming Events

Salt Lake City, UT

WED - FRI, OCTOBER 9 - 11, 2024

Pack your bags, anon — we’re heading west! Join us in the beautiful Salt Lake City for the third installment of Permissionless. Come for the alpha, stay for the fresh air. Permissionless III promises unforgettable panels, killer networking opportunities, and mountains […]

recent research

Avail.jpg

Research

Data publishing costs have historically been a bottleneck for rollups, and as more rollups launch, interoperability will continue to be a major challenge. Avail presents a potential solution to rollup fragmentation through its three products: Avail DA, Nexus, and Fusion, which together aim to unify the web3 experience.

article-image

Bitcoin miners need to explore unconventional energy avenues or be buried by the financial realities created by this halving

article-image

BlackRock’s iShares Bitcoin Trust continues to see daily positive net flows, though its inflow total for a single day hit a new low Wednesday

article-image

Binance is making moves, from receiving a new license in Dubai to switching its SAFU fund to USDC

article-image

Miner stocks have historically underperformed bitcoin before the halving and outperformed the asset after the event, analysts note

article-image

After a one-week trial, a jury convicted crypto trader Avraham Eisenberg

article-image

A verdict in Avraham Eisenberg’s criminal fraud trial is expected this afternoon