Cardano Network Recovers from Outage on 50% Relay and Block-Producing Nodes

Over the weekend, Cardano stake pool operators (SPOs) and users reported that at least half of the nodes for the network briefly went offline. According to a Jan. 22 post shared on the Telegram SPO for Input Output Global, which is the engineering and research in fintech behind the Cardano blockchain, an anomaly caused 50% of Cardano nodes to disconnect and restart. The post stated, “This appears to have been triggered by a transient anomaly causing two reactions in the node, some disconnected from a peer, others threw an exception and restarted.”

Despite the momentary degradation, the Cardano network recovered without any external intervention. The post explained, “such transient issues” were considered in the node design and consensus, and “the systems behaved exactly as expected.” During the anomaly, which happened between block 8300569 and 8300570, block production reportedly kept going, but it was slowed for a few minutes, and the “impact was low, akin to the delays that occur during normal operations.”

The post also noted that “most nodes automatically recovered, depending on the SPO of choice.” At the time of writing, the root cause of the anomaly and resulting node disconnections and restarts are still under investigation. The official announcement stated, “We’re now investigating the root cause for this anomalous behavior and implementing further logging measures alongside our regular monitoring procedures.”

In conclusion, the Cardano network faced a minor disruption over the weekend, caused by an unknown anomaly that disconnected and restarted half of the nodes. The network recovered quickly and automatically, without any external intervention and the impact on the block production was minimal. The cause of this incident is still under investigation and the team is implementing further logging measures to prevent similar incidents in the future.

Tom Stokes, a co-founder of Node Shark and a Cardano stake pool operator (SPO), reported in a Jan. 22 post that a significant number of nodes were affected by the recent anomaly. He also shared a chart that illustrated the network sync falling from 100% to slightly above 40% for over 300 reporting nodes.