Cosmos Hub Resumes Operations After Four-Hour Outage

On June 5, around 22:00 (Kyiv/MSK), the update of Cosmos Hub to version 17 resulted in a disruption within the ecosystem, causing a four-hour halt in block production.

The network team promptly responded to the issues and began working on a solution.

The Cosmos Hub is experiencing a temporary chain halt.

The teams at @informalinc and @HyphaDoesCosmos have identified a fix.
A coordinated upgrade will be conducted ASAP.

Updates will be shared shortly. Thank you for your patience.

— Cosmos Hub ⚛️ (@cosmoshub) June 5, 2024

Later, developers reported the resolution of a security vulnerability in the Liquid Stakes Module (LSM). All protocols using LSM were advised to review the recommendations for further operation of the mechanism. 

During the v17 upgrade, the Hub validators patched a security vulnerability in the Liquid Staking Module (LSM).

If your chain utilizes the LSM, please review the security advisory posted to Github.

Link below:https://t.co/TpKBudAxmm

— Cosmos Hub ⚛️ (@cosmoshub) June 5, 2024

A few hours later, Cosmos Hub released an update with a bug fix, which, at the time of publication, had been integrated by 55% of validators. 

The teams at @informalinc & @HyphaDoesCosmos have issued a fix for the bug that was found. We’re currently at 55% of validators who have implemented the patch. We encourage any remaining validators who have not yet implemented the patch to do so.

Thanks for your patience!

— Cosmos Hub ⚛️ (@cosmoshub) June 5, 2024

Subsequently, the ecosystem team reported the full resumption of block production, promising to publish details of the incident investigation. 

The previous issue has been patched & the Cosmos Hub is producing blocks once more.

The teams at @informalinc & @HyphaDoesCosmos will be releasing a post-mortem ASAP with all the details. Thanks to the Hub validators for a swift upgrade & thank you all for your patience.

— Cosmos Hub ⚛️ (@cosmoshub) June 6, 2024

Theoretically, the bug could have allowed users to avoid penalties during the redelegation of tokenized assets.

Back in May, the Zilliqa network experienced a block production disruption. The issue was fully resolved only by May 10.

In April, Polkadot parachains halted operations for about an hour following an update to the main network. This year, disruptions were also experienced by dYdX, StarkNet, Solana, Polygon zkEVM, and Avalanche.