TON outage invokes speculation about role of centralization
Without a technical post-mortem, we’re mostly in the dark as to what caused the recent seven-hour outage
CryptoFX/Shutterstock modified by Blockworks
Today, enjoy the Empire newsletter on Blockworks.co. Tomorrow, get the news delivered directly to your inbox. Subscribe to the Empire newsletter.
The Offline Network
On October 4, 2021, Facebook went down for about seven hours.
No Facebook, no WhatsApp, no Instagram — Web2’s flagship apps were dark to the outside world.
On the ground, devs and other problem solvers were even locked out of tools they needed to work and communicate. Some employees couldn’t enter company buildings or conference rooms, as the authentication systems for scanning badges had also been knocked out.
Facebook had somehow disconnected itself from the internet.
The culprit was a configuration error in software used to route traffic to and from Facebook data centers, Border Gateway Protocol v4, also described as the “postal service of the internet.”
Tech giants like Facebook, Google and Amazon famously build their own internets. An internet within an internet, so that they can stay online without relying on anyone else.
With this in mind, Facebook, WhatsApp and Instagram were still there — technically — it’s just the internet had no way of knowing how to reach them.
Facebook, like other big tech forms, spreads its data centers all over the world, making it possible to reroute traffic to different destinations at times of high load in particular areas (and protect against natural disaster swallowing all our personal data).
So, while Facebook had worked to decentralize its physical operations from the broader internet, a single bug still managed to bring it all tumbling down.
TON’s recent outage — which also lasted seven hours — could be a similar situation. But without a technical post-mortem, we’re mostly in the dark as to what caused it.
We know that an airdrop for a new memecoin, DOGS, triggered a spike in onchain activity which knocked validators out of consensus. That meant they couldn’t agree which block should be added to the chain next.
When the airdrop was activated, transactions per second surged to 280 at most, which seems to be around the typical breaking point for the network — much lower than the 100,000-plus marketed by TON.
A prior incident in December occurred at around 300 transactions per second resulting from inscriptions spam.
In this instance, validators coordinated Wednesday morning to restart the chain, as was the case with Solana’s multiple reboots over the past few years.
It’s unclear whether centralization was a factor in the TON outage. Say, a breakdown in logic buried somewhere in the validator client, hypothetically fixed if there were more validator clients built by different teams — particularly ones that can handle higher volumes.
“The short answer is ‘tech is shitty, could be better,’ Mikko Ohtamaa, early Bitcoin developer and cofounder of Trading Strategy, told me this morning. He’s been tracking these sorts of snafus and posting about them on X.
“The root cause is poor developer culture — distribution is more important than tech in the short term. TON has had several security audits, none published. The first step would be to fix this,” he said.
Decentralization is such a vague term that it almost says nothing about the state of blockchain tech stacks. It’s all a spectrum, and something can only be decentralized in comparison to another thing.
At the same time, there are blockchains out there much more resilient to downtime than TON and even Solana (the big two being Bitcoin and Ethereum).
Decentralization for much of the Web3 space is mostly about control. Whether the network or protocol can be hijacked by any one particular entity or group — not whether it’s immune to going dark.
Maybe in Web4.
— David Canellis
Data Center
- TON fell around 6% during the outage as crypto exchanges suspended deposits and withdrawals. It’s now down 20% since the arrest of Telegram CEO Pavel Durov.
- BTC and ETH have each shed 4% in the past day. (BTC: $59,920; ETH: $2,520).
- Memecoins and AI-linked tokens lead over the past week, with POPCAT swelling 71%, followed by FET (46%) and RENDER (33%).
- $130.56 million has flowed out of Arbitrum bridges in the past week, more than three times the outflows from Ethereum.
- Global NFT sales volumes are down over 40% in the past 30 days, to $382.8 million, per CryptoSlam.
Choose your fighter
Speaking of TON, we’re finally seeing altcoins wake up a bit — though altcoin szn is still nowhere to be found.
“Altcoins are also seeing significant activity, particularly in response to specific news events. For example, the arrest of Telegram CEO Pavel Durov led to a sharp decline in the price of TON, accompanied by a large liquidation event,” Hyblock Capital wrote in a note.
“Interestingly, despite this price drop and liquidations, open interest in TON reached all-time highs, indicating that traders remain actively engaged and are perhaps seeking to capitalize on volatility. This behavior suggests that the market is not on the sidelines but rather starting to actively search for opportunities, even in a climate of uncertainty.”
Right now, it seems that the market is in a bit of a reactive moment, as different parts of crypto respond to different events.
If we zoom out then, Bitcoin — according to Hyblock — has a few liquidity zones to keep an eye on.
“Bitcoin itself experienced a notable increase in volatility, surging from 60,000 to 64,000 before entering a period of consolidation. Market participants are closely monitoring this behavior, especially as retail long positions have been on the rise. Historically, an increase in retail longs has been inversely correlated with Bitcoin’s price, often signaling a bearish outlook,” CEO Shubh Varma wrote.
BRN analysts noted that the technical picture is “mixed,” showing a “lack of clear market direction.”
A dip could see bitcoin drop back to between $54,000 and $56,000. QCP, however, isn’t terribly worried about it— despite the drop to $58,000 we saw yesterday.
“We believe that any dip in equities (and crypto) will be short-lived. With Powell and the Fed ready to kickstart a rate-cutting cycle, increased liquidity will eventually push risk assets higher. We are finally on the cusp of a rate-cutting cycle,” analysts said.
Now there’s one last bit I want to mention, related to stablecoins.
Per Dynamo DeFi’s Patrick Scott, stablecoins (excluding algorithmic stablecoins) are managing to carve out new all-time highs, even beating 2022 levels.
And, since Scott posted over the weekend, the market cap has actually jumped to $169 billion, per DeFi Llama data.
Crypto’s kind of a mixed bag right now, with a bunch of different narratives driving different segments within the larger market.
As we enter September, it’ll be interesting to see what narratives start to dominate this cycle.
— Katherine Ross
The Works
- Durov’s brother, Nikolai Durov, is also wanted by French authorities, Politico reported.
- MakerDAO rebranded to Sky in an attempt to ”appeal” to mainstream users.
- After looking for a potential sale earlier this year, Osprey Funds is selling the Osprey Bitcoin Trust’s assets to Bitwise’s bitcoin ETF.
- Crypto regulation is likely to come from the US Treasury, not necessarily the SEC, Axios reported.
- The uptick in crypto prices led to the creation of 88,000 new crypto millionaires so far this year, per Henley & Partners.
The Riff
“You all know what they are, we’ve had a lot of with fun with ’em.”
That’s NFT pitchman Donald Trump in a video advertisement that dropped this week. Yes, the 45th president of the United States of America (and would-be 47th president of the United States of America) is once again trying to sell some digital trading cards bearing his likeness.
Insisting once more that they’re just like baseball cards, Trump is back to performing one angle of what’s proven to be a rather lucrative crypto business pitch.
“These cards show me dancing and even holding some bitcoins,” he tells the camera.
Trump had gone a bit quiet on the crypto front since he dazzled the bitcoiners down in Nashville and collected a cool $25 million (estimated) from some of the industry’s deepest political pockets. But here he is, doing what he does best: selling something with his name on it.
If this riff sounds a bit weary, well, your reading skills are correct. It’s hard to shake the feeling that it’s really all about the royalties on this one.
But hey, who am I to judge? As Trump himself told the attendees in Nashville: “Have fun with your bitcoin and your crypto and everything else you’re playing with.”
— Michael McSweeney
Start your day with top crypto insights from David Canellis and Katherine Ross. Subscribe to the Empire newsletter.
Explore the growing intersection between crypto, macroeconomics, policy and finance with Ben Strack, Casey Wagner and Felix Jauvin. Subscribe to the Forward Guidance newsletter.
Get alpha directly in your inbox with the 0xResearch newsletter — market highlights, charts, degen trade ideas, governance updates, and more.
The Lightspeed newsletter is all things Solana, in your inbox, every day. Subscribe to daily Solana news from Jack Kubinec and Jeff Albus.
