Solana developers reveal causes of seven-hour network outage on April 30

A huge number of incoming transactions and heavy traffic led to the suspension of the Solana network overnight from April 30 to May 1. This was disclosed by the developers following the investigation.

On April 30th, Solana’s Mainnet Beta cluster suffered a 7 hour outage caused by stalled consensus. Below is an investigation into the cause of the outage, and the steps core developers are taking to increase network resiliency and stability. https://t.co/pfkMTEwPda

— Solana Status (@SolanaStatus) May 3, 2022

The outage was linked to bot activity competing to mint NFTs released via the Candy Machine tool. 

The spam transactions arose from abandoning the Dutch auction in favor of selling tokens at a fixed price. This created an incentive for users and bots to send a huge number of transactions in hopes of beating competitors.

Today #Solana mainnet-beta went down partially due to botting on the Metaplex Candy Machine program. To combat this, we have merged and will soon deploy a botting penalty to the program as part of a broader effort to stabilize the network. https://t.co/QaAZT3VxXz

— Metaplex (@metaplex) May 1, 2022

Ultimately, validators ran out of memory and a crash occurred. The total number of forks that validators were supposed to evaluate exceeded their ability to process them automatically. Consequently, manual intervention was required to resolve the issue.

‘TPS reached 6 million, exceeding 100 Gbps of traffic on some nodes. No signs of DoS attacks were detected,’ the report says.

On average, the Solana network processes around 2,700 TPS, with a ceiling of over 710,000 TPS.

The incident in the night of May 1 bore the same nature as the September 14, 2021 outage.

To prevent a recurrence, developers are preparing update v1.10. It includes, among other things, improvements to memory usage algorithms, extending the window during which nodes can tolerate slow or stalled consensus.

In a subsequent version, v1.11, a mechanism for prioritising fees is expected to be implemented.

Earlier this year, the Solana network was unavailable for 48 hours. At that time, developers did not rule out that the problem could recur in the future. The incident was the sixth in the previous three months.

Subscribe to ForkLog news on Telegram: ForkLog Feed — the full news stream, ForkLog — the most important news, infographics and opinions