Solana network rebooted after validator software-update error

Solana validators rebooted the network twice due to an unknown error that caused a complete halt of economic activity on the blockchain. The glitch occurred amid updating the node software to a newer version.

The network stopped processing user transactions properly on February 25 at about 07:46 Kyiv time (08:46 MSK) and entered a “degraded” mode. The latter means that all functions except voting were disabled.

One of the validators — Stakewiz — explained that the error arose after the majority of nodes migrated to software version 1.14.16, which took place last week. He described the update as “a huge leap with a glut of code changes.”

We know that in the 6-12 hours prior to the start of today’s event the validators running 1.14.x versions cumulatively achieved a supermajority of stake (>66.6%).

We also know that there was a «fat block» propagated around the time of the start of this event. pic.twitter.com/ZAvfbzZd1M

— Laine ❤️ stakewiz.com (@laine_sa_) February 25, 2023

According to Stakewiz, shortly before the halt in economic activity on the network, a block containing about 150,000 шрэдов was generated. It caused a fault in the Turbine-based block-propagation mechanism used by the кластерами for block propagation, which led to a blockchain outage.

A few hours after the incident, validators rebooted the network for the first time. However, the measure proved ineffective, necessitating a second iteration.

Stakewiz noted that the initial incident spawned separate chains. He also stressed that the reboot of the blockchain effectively constitutes a hard fork, since it requires resynchronisation of validators and 80% agreement within a vote. 

After the second reboot in the night of February 26, developers said they had resolved the problem. As of writing, the network is generating blocks reliably, according to Solana Explorer. At the same time, periodic spikes in the average transaction-processing time have been observed in the blockchain, negatively affecting its throughput.

In June 2022, Solana experienced an outage due to a bug in handling ‘long-lived non-targeted transactions’, which led to a disruption of the consensus.