Infura outage explained as an ‘unannounced hard fork’ of Ethereum

Outages at Infura, the leading provider of infrastructure solutions for the Ethereum network, are linked to consensus errors in older versions of the Geth client. Because part of the network had not updated to the latest version, Ethereum split into two chains at block 11234873.

“At some point Ethereum developers implemented changes to the code that led to today’s split at block 11234873 (07:08 UTC). Those who did not upgrade (Blockchair, Infura, some miners and many others) were stuck for two hours on the shorter chain (~30 blocks),” wrote Blockchair developer Nikita Zhavoronkov.

According to him, “technically this was an unannounced hard fork.”

(1/2) Ok, so what happened today on #Ethereum🦄:

1. At some point Ethereum developers introduced a change in the code that led today to a chain split starting from block 11234873 (07:08 UTC)

2. Those who haven’t upgraded (@Blockchair, @infura_io, some miners, and many others)… pic.twitter.com/mbRYFU5tgn

— Nikita Zhavoronkov (@nikzh) November 11, 2020

Nikita Zhavoronkov contends that something similar happened in the Bitcoin network seven years ago when there was a database upgrade.

“The solution: update Geth and run debug.setHead(11234872)”, he added.

Even with a remedy, Zhavoronkov stresses that what happened should not be underestimated and should be considered one of the most serious issues Ethereum has faced since the The DAO incident four years ago.

In my opinion, today’s consensus failure in #Ethereum🦄 shouldn’t be underestimated and should be considered as the most serious issue Ethereum has faced since the DAO debacle 4 years ago. An investigation is in order.

— Nikita Zhavoronkov (@nikzh) November 11, 2020

A former Monero maintainer, Riccardo Spagni, confirms the rationale behind such actions.

“We have secretly fixed bugs in Monero several times, fearing they could be exploited if disclosed. For example, that is how we acted in 2017,” Spagni commented, recalling the successful fix of a bug that could have allowed unlimited XMR coins.

Geth developer Martin Holst Svende explained that older client versions contained consensus bugs.

“The change introduced was actually targeted at fixing those problems. This is a reminder that node software must be kept up to date,” he wrote.

Older geth-versions contained consensus-flaw(s) which were hit today on mainnet. The “change introduced” was actually fixing those issue(s). This is a reminder to keep your node(s) up to date! https://t.co/oZatRUgENS

— M H (((Swende))) (@mhswende) November 11, 2020

Analyst Larry Chermak of The Block, however, does not understand how such a situation became possible.

“If this is the case, things are much worse than I thought. How can it be that the largest infrastructure provider did not know about a code change that could have led to a chain split? It’s almost unbelievable.”

Earlier on Wednesday, November 11, it was announced that Infura and other services encountered outages. This caused delays in ETH and ERC-20 token quote feeds, and some exchanges halted withdrawals.

At the time of publication, developers were continuing to restore service functionality. By 15:00 UTC, the work had been completed.

Subscribe to ForkLog news on Telegram: ForkLog FEED — the full news feed, ForkLog — the most important news and polls.