https://medium.com/aergo/post-mortem-aergo-stall-february-1st-2024-5cfbc56a677e
On February 1st at 13:13 KST, Aergo’s mainnet halted new block generations, which resulted in a temporary outage. The core dev team immediately responded to the issue, successfully restoring the network by applying a hotfix to address the root cause.
To be clear, no funds were at risk as a result of the network halt.
Cause
Block generation was halted due to a bug in the Fee Delegation transaction processing within the Smart Contract engine.
Recovery Timeline
February 1st, 2024:
- 13:13 — Block generation stopped at block number 150677807.
- 15:30 — Blocked External nodes to enable communication exclusively between the Trusted Nodes (whitelisted based on IPs). The initial block regeneration attempt failed.
- 17:00 — Commenced testing the hotfix development version and synchronization test among the BPs and the Trusted Nodes.
Continued on February 1st:
- 19:00 — Snapshot data prepared.
- 21:00 — Completed the hotfix development and the local test was successful.
February 2nd, 2024:
- 01:30 — Block generation resumed.
- 08:00 — Ready to snapshot Block Producers (BPs) and initiated synchronization between BPs.
- 08:50 — Snapshot prepared for the Trusted Nodes, and the hotfix was tested on the Alphanet and the testnet.
- 10:20 — Initiated synchronization test of the hotfix on the mainnet.
- 13:10 — Hotfix development version (2.5.1-dev) deployed on the mainnet with BPs and the Trusted Nodes.
- 14:20 — Initiated synchronization for the mainnet Trusted Node 1 and dApp operation resumed.
- 16:00 — Completed recovery based on Mainnet BP/Trusted Nodes.
We will continue to strive to keep the Aergo blockchain platform stable. We apologize for any inconvenience this may cause and appreciate your understanding.