zkSync Era team explains cause of network outage

Block processing halt on December 25 by a Layer-2 decision zkSync Era occurred due to an error in the calculation of the state update, according to developers.

On December 25 at 05:50 UTC, zkSync Era encountered an issue triggered by one of the network’s automated safety procedures.

The issue is completely resolved, but we want to share details with the community about what happened, and what teams did to address it.

TL;DR
• There… pic.twitter.com/hO9TKlfBMQ

— zkSync Developers (∎, ∆) (@zkSyncDevs) December 25, 2023

Data are processed on both the protocol side and the nodes. In the latter software there was a small bug, the team explained. As a result of discrepancies in the calculations, safety procedures were activated — the network entered maximum-security mode.

After the bug was fixed, the sequencer resumed processing transactions, starting with those accumulated in the mempool. Developers released a software update for node operators. 

The holiday period meant the fix took longer than normal operation, as most specialists were on leave.

The team noted that the activated maximum-protection mode is now considered excessive, but they were unable to discuss lifting the procedure during the incident. The developers also plan to improve emergency-response measures for network incidents during holiday periods in the coming year.

User funds were safe during the outage, they emphasised.

In August, the Polygon unit accused the main zkSync Era developer—the company Matter Labs—in code theft for the Boojum update.