Incident management: Chain halt BH 20200073

[byte-bandit]

Timeline

At 10.07.2024, 18:48:49, the entire tubmlr network halted at block height 20200073.

Observations

• The network disagrees on the hash for block height 20200073.
• Some nodes expect 7FE09400054FDF6DE2C7F83A8794720F0350F5C727066247BE60E3DEE5F61B6A and receive 59FEF09BF899CB3342B030A00F09C8D6A3380F94ECF3A9363B2D36B089A0E805
• Other nodes the opposite
• Block 20200073 contained 2 transactions with nothing out of the ordinary
• Same for 20200072
• The vote to set an ERC20 to denom mapping had landed in 20200073

Vote implications

• With the vote being the only thing suspicious around this time, initial investigation focused on the recent renaming of the module
• All test paths in PTN (including a full version update) ran successfully
• Furthermore, the denom is actually in the store and queriable (nodes from both sides will have the record in store, as well as the params, so the mismatch doesn't seem to be related to the store or the upgrade)

[verabehr]

Next steps:

1. getting validators to roll back to get the correct hash
2. try to get the tracing node set up

[taariq]

Chain restarted after 24-hours. Closing to be fixed with #1862