Behind the scenes of Solana’s slow validator problem

This is a segment from the Lightspeed newsletter. To read full editions, subscribe.

As issues with Solana’s code have been ironed out over the years, block times — or how long it takes the network to produce a new block of transactions — have declined, even dipping under Solana’s putative level of 400ms.

But over the past month, an interesting trend has emerged: Median block times spiked, and Solana was adding new transactions to the blockchain slower than before. At fault was a new meta among Solana validators that showed it can be lucrative to produce blocks slowly. Anza, Jito and Marinade appear to be weighing fixes to the problem, Blockworks has learned.

Solana epoch lengths spiked in June. Source: Kamino

Every Solana block has a validator who serves as leader — meaning they collect transactions, create a block and broadcast it to the network. Leaders collect transaction fees from the blocks they create. More order flow means more fee opportunities, so it can be more lucrative for validators to see and process 500ms worth of transactions as opposed to 300ms, for instance.

At a basic level, some Solana validators appear to be waiting as long as possible to pack blocks with transactions in hopes of maximizing their yield, which has caused Solana’s epoch lengths to increase.

This isn’t ideal for a network focused on being as fast as the Nasdaq. Plus, fewer epochs per year means fewer opportunities for staking rewards to compound for stakers, Sol Strategies chief technology officer Max Kaplan noted.

Solana offers something called “grace ticks,” which is a late period wherein leaders can still successfully submit blocks. This prevents validators in far-flung locations from being unfairly penalized, but it also opens the door for validators to be late on purpose.

The alternative Solana client Frankendancer also recently released a revenue-maximizing scheduler, and validators running the client appear to be packing their blocks a bit more slowly than normal, Kaplan said.

Kaplan added that the Frankendancer delay is minimal compared to more egregious delayers, and that he wouldn’t portray it as a “bad thing.” Plus, block delaying isn’t a new concept on proof-of-stake blockchains. Firedancer’s upgrade may have brought this strategy to light on Solana, however. Jump did not immediately return a request for comment.

Interestingly, Firedancer software engineer Michael McGee described the phenomenon on this week’s Lightspeed podcast episode.

“One of the things we’ve seen with our current validator…[is that validators] can often make a more profitable block by delaying execution of transactions,” McGee noted.

Solana validators who are more noticeably delaying blocks tend to be running modified versions of the Agave-Jito client, Blockworks Research analyst Victor Pham noted.

For example, during epoch 802 in mid-June, Galaxy and Kiln’s median block time was higher than 570ms each. A number of unlabeled validators were also slow, and Temporal’s validators had 475ms median block times, according to Solana Compass data.

Kiln co-founder Ernest Oppetit admitted the validator — Solana’s sixth-largest by stake — had been delaying slots “for a period” but said it has stopped doing so.

“At Kiln we pride ourselves in offering the top staking APY in the market without compromising on security. We have been doing R&D on different parts of the stack, including timing games, and are in constant discussions with our customers, client teams and the foundation on this. At this point we are following the spec and not delaying blocks, but many actors are still doing so. We believe the incentive issue (fast block production leads to lower rewards) ultimately needs to be addressed at the protocol level,” Oppetit said.

“I will say that we are not the reason that people know about this,” Temporal engineering director Ben Coverston said when reached for comment about the validator’s apparent participation in the slow block trend.

“We, as a service provider, support validator configurations that prioritize maximizing staking rewards for our clients. On Solana, that may mean proposing slightly slower blocks to ensure higher reward capture. Galaxy also remains responsive to community feedback and have since increased block times to within an agreeable threshold,” a Galaxy spokesperson said.

The Solana validator community doesn’t find slowing the network down to be kosher, and slow validators are now facing public backlash.

They may soon face more material punishments. Blockworks has learned that Jito plans to blacklist slow validators from its stake pool, which is Solana’s largest.

Jito Foundation president Brian Smith said the organization is “drafting a governance proposal that empowers a committee to remove laggards from the JitoSOL delegation set. This should be available for community discussion within a few days.”

Michael Repetny, co-founder of the third-largest stake pool in Marinade, said the pool provider is “[considering] putting it to a governance proposal to discuss pros and cons of introducing [slow validators] as a hard rule/offense of the delegation strategy.”

Help is on the way at the protocol level, too. Anza’s GitHub repository shows a new proposal to shrink Solana’s grace tick period by half. Plus, Solana’s proposed consensus overhaul should fix things.

“Alpenglow will solve this by enabling skip votes,” Anza vice president of core engineering Brennan Watt said.

Watt said on a recent Lightspeed podcast episode that Anza hopes to have Alpenglow live on mainnet by Solana’s Breakpoint conference in December.

Get the news in your inbox. Explore Blockworks newsletters:

The Breakdown: Decoding crypto and the markets. Daily.
Empire: Crypto news and analysis to start your day.
Forward Guidance: The intersection of crypto, macro and policy.
0xResearch: Alpha directly in your inbox.
Lightspeed: All things Solana.
The Drop: Apps, games, memes and more.
Supply Shock: Bitcoin, bitcoin, bitcoin.