Heimdall 1.2.0: error on replay: Wrong Block.Header.AppHash. Expected X, got Y

[mateipopa]

Heimdall version (use heimdall version or git rev-parse --verify HEAD if installed from source):
1.2.0

Environment:

• Ubuntu 22 (e.g. from /etc/os-release):
• Install tools:
• Others:

What happened:
The node gets stuck, heimdalld returns:

ERROR: failed to create new node: error during handshake: error on replay: Wrong Block.Header.AppHash.  Expected 18F262A44C2CFD775AA5336C5C105FE5EAF503EA889F96F5A37ACC9818AA2697, got 43382BF1E1EDFBBCAEF7D9C909CE692E174BBE95055071D3D7F72DB107909013

What you expected to happen:
The node would work fine as it did before

Have you tried the latest version: yes/no
yes

How to reproduce it (as minimally and precisely as possible):
Try to sync a node from snapshots (retried loading the snapshots too).

Logs (paste a small part showing an error (< 10 lines) or link a pastebin, gist, etc. containing more of the log file):

DEBUG[2025-02-24|06:24:49.636] ABI initialized                              name="[{\"anonymous\":false,\"inputs\":[{\"indexed\":true,\"internalType\":\"address\",\"name\":\"from\",\"type\":\"address\"},{\"indexed\":true,\"internalType\":\"address\",\"name\":\"to\",\"type\":\"address\"},{\"indexed\":false,\"internalType\":\"uint256\",\"name\":\"value\",\"type\":\"uint256\"}],\"name\":\"Transfer\",\"type\":\"event\"},{\"anonymous\":false,\"inputs\":[{\"indexed\":true,\"internalType\":\"address\",\"name\":\"owner\",\"type\":\"address\"},{\"indexed\":true,\"internalType\":\"address\",\"name\":\"spender\",\"type\":\"address\"},{\"indexed\":false,\"internalType\":\"uint256\",\"name\":\"value\",\"type\":\"uint256\"}],\"name\":\"Approval\",\"type\":\"event\"},{\"constant\":true,\"inputs\":[],\"name\":\"totalSupply\",\"outputs\":[{\"internalType\":\"uint256\",\"name\":\"\",\"type\":\"uint256\"}],\"payable\":false,\"stateMutability\":\"view\",\"type\":\"function\"},{\"constant\":true,\"inputs\":[{\"internalType\":\"address\",\"name\":\"owner\",\"type\":\"address\"}],\"name\":\"balanceOf\",\"outputs\":[{\"internalType\":\"uint256\",\"name\":\"\",\"type\":\"uint256\"}],\"payable\":false,\"stateMutability\":\"view\",\"type\":\"function\"},{\"constant\":true,\"inputs\":[{\"internalType\":\"address\",\"name\":\"owner\",\"type\":\"address\"},{\"internalType\":\"address\",\"name\":\"spender\",\"type\":\"address\"}],\"name\":\"allowance\",\"outputs\":[{\"internalType\":\"uint256\",\"name\":\"\",\"type\":\"uint256\"}],\"payable\":false,\"stateMutability\":\"view\",\"type\":\"function\"},{\"constant\":false,\"inputs\":[{\"internalType\":\"address\",\"name\":\"to\",\"type\":\"address\"},{\"internalType\":\"uint256\",\"name\":\"value\",\"type\":\"uint256\"}],\"name\":\"transfer\",\"outputs\":[{\"internalType\":\"bool\",\"name\":\"\",\"type\":\"bool\"}],\"payable\":false,\"stateMutability\":\"nonpayable\",\"type\":\"function\"},{\"constant\":false,\"inputs\":[{\"internalType\":\"address\",\"name\":\"spender\",\"type\":\"address\"},{\"internalType\":\"uint256\",\"name\":\"value\",\"type\":\"uint256\"}],\"name\":\"approve\",\"outputs\":[{\"internalType\":\"bool\",\"name\":\"\",\"type\":\"bool\"}],\"payable\":false,\"stateMutability\":\"nonpayable\",\"type\":\"function\"},{\"constant\":false,\"inputs\":[{\"internalType\":\"address\",\"name\":\"from\",\"type\":\"address\"},{\"internalType\":\"address\",\"name\":\"to\",\"type\":\"address\"},{\"internalType\":\"uint256\",\"name\":\"value\",\"type\":\"uint256\"}],\"name\":\"transferFrom\",\"outputs\":[{\"internalType\":\"bool\",\"name\":\"\",\"type\":\"bool\"}],\"payable\":false,\"stateMutability\":\"nonpayable\",\"type\":\"function\"},{\"constant\":false,\"inputs\":[{\"internalType\":\"address\",\"name\":\"spender\",\"type\":\"address\"},{\"internalType\":\"uint256\",\"name\":\"addedValue\",\"type\":\"uint256\"}],\"name\":\"increaseAllowance\",\"outputs\":[{\"internalType\":\"bool\",\"name\":\"\",\"type\":\"bool\"}],\"payable\":false,\"stateMutability\":\"nonpayable\",\"type\":\"function\"},{\"constant\":false,\"inputs\":[{\"internalType\":\"address\",\"name\":\"spender\",\"type\":\"address\"},{\"internalType\":\"uint256\",\"name\":\"subtractedValue\",\"type\":\"uint256\"}],\"name\":\"decreaseAllowance\",\"outputs\":[{\"internalType\":\"bool\",\"name\":\"\",\"type\":\"bool\"}],\"payable\":false,\"stateMutability\":\"nonpayable\",\"type\":\"function\"}]"
ERROR: failed to create new node: error during handshake: error on replay: Wrong Block.Header.AppHash.  Expected 18F262A44C2CFD775AA5336C5C105FE5EAF503EA889F96F5A37ACC9818AA2697, got 43382BF1E1EDFBBCAEF7D9C909CE692E174BBE95055071D3D7F72DB107909013

Config (you can paste only the changes you've made):

node command runtime flags:

/usr/bin/heimdalld start --home=/heimdall/ --chain=mainnet --rest-server --seeds 1500161dd491b67fb1ac81868952be49e2509c9f@52.78.36.216:26656,dd4a3f1750af5765266231b9d8ac764599921736@3.36.224.80:26656,8ea4f592ad6cc38d7532aff418d1fb97052463af@34.240.245.39:26656,e772e1fb8c3492a9570a377a5eafdb1dc53cd778@54.194.245.5:26656,6726b826df45ac8e9afb4bdb2469c7771bd797f1@52.209.21.164:26656

/dump_consensus_state output for consensus bugs

Anything else we need to know:
This is happening on several of our nodes, ranging from heimdall version 1.0.5 to 1.2.0

[avalkov]

Which snapshot your using?
Are these all only sentry nodes?

Here is issue where people reported same error but got it resolved - #1238

[mateipopa]

I've been using a snapshot from Stakecraft, but I've fallen back to the one indicated in the issue you've referenced. About the seeds, I'm not sure what to answer - I've got the startup instructions from the Polygon Discord, including the seed nodes.
I'm in the process of downloading the snapshots from Publicnode, will re-initialize Heimdall once they're downloaded and will report back here.
Thanks for the quick reply and for the referenced issue, it somehow eluded me in the first place