Parallel state processor failed err="invalid merkle root ..."

[donbeave]

I'm getting this warning in running bor instance, it looks like synchronization is still in progress.

I'm not sure if is it an error that I should take care of or if I just can ignore it.

INFO [02-20|22:47:22.301] Starting peer-to-peer node               instance=bor/v2.0.0/linux-amd64/go1.23.6
...

INFO [02-20|22:47:22.345] New local node record                    seq=1,739,807,990,912 id=8ccea65000e07cf9 ip=162.55.230.100 udp=30305 tcp=30305
INFO [02-20|22:47:22.353] Started P2P networking                   self=enode://fc86e49fc965a284679060bff9cd1acea99e1c89398dacfa0f39aa4da224cd29ab8e92cb41d117432af7aae6a82fd9f3cd4f117f83da1da508b125f6b5986481@162.55.230.100:30305
INFO [02-20|22:47:22.355] IPC endpoint opened                      url=/data/bor/bor.ipc
WARN [02-20|22:47:22.355] Sanitizing invalid HTTP read header timeout provided=0s      updated=30s
INFO [02-20|22:47:22.355] HTTP server started                      endpoint=[::]:8945 auth=false prefix= cors=* vhosts=*
INFO [02-20|22:47:22.358] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-20|22:47:22.359] Whitelisting checkpoint deferred         err="chain out of sync"
INFO [02-20|22:47:31.325] Block synchronisation started
WARN [02-20|22:47:32.323] Parallel state processor failed          err="invalid merkle root (remote: 9d56272b9da021d19eb12f4305ba9528fb1e7ef2da47cb7fb6dfe06ee89452de local: c6ad6091b8bdd035b77f9d581f6dfceb47e166e9abcec6484a8ba4fca9ddbead) dberr: %!w(<nil>)"
INFO [02-20|22:47:32.536] Imported new chain segment               number=15,777,208 hash=2280dc..422fce blocks=2 txs=617 mgas=42.401 elapsed=480.797ms mgasps=88.189 age=3y8mo3w snapdiffs=4.87MiB triediffs=139.96MiB triedirty=187.63MiB
INFO [02-20|22:47:32.938] Imported new chain segment               number=15,777,210 hash=29c210..09847d blocks=2 txs=773 mgas=42.437 elapsed=401.911ms mgasps=105.589 age=3y8mo3w snapdiffs=4.93MiB triediffs=139.76MiB triedirty=187.68MiB
INFO [02-20|22:47:34.363] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-20|22:47:46.362] Whitelisting milestone deferred          err="chain out of sync"
WARN [02-20|22:47:54.562] Served eth_coinbase                      conn=194.67.206.50:43494 reqid=2 duration="15.55µs" err="the method eth_coinbase does not exist/is not available"
INFO [02-20|22:47:58.362] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-20|22:48:10.363] Whitelisting milestone deferred          err="chain out of sync"
INFO [02-20|22:48:22.362] Whitelisting milestone deferred          err="chain out of sync"
ERROR[02-20|22:48:31.914] Expired request does not exist           peer=83570f7405cf005e8cde20f99b69a935945b0af25448d95a4355e6f3a5a02b10
INFO [02-20|22:48:33.228] Fetching state updates from Heimdall     fromID=424,668 to=2021-06-16T09:19:47Z
INFO [02-20|22:48:33.228] Fetching state sync events               queryParams="from-id=424668&to-time=1623835187&limit=50"
INFO [02-20|22:48:33.242] "→ committing new state"               eventRecord=424,668
INFO [02-20|22:48:33.244] "→ committed new state"                eventRecord="id 424668, contract 0xA6FA4fB5f76172d178d61B04b0ecd319C5d1C0aa, data: 0x87a7811f4bfedea3d341ad165680ae306b01aaeacc205d227629cf157dd9f821000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000be958864291682f6fb1884455722bd483fa33c26000000000000000000000000a0b86991c6218b36c1d19d4a2e9eb0ce3606eb48000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000000000000000000000000000000038184fe5, txHash: 0xeff66a14aaf688308cce3b9c39fed45afcb4e287cb0705a62a0b0be5f5584f19, logIndex: 149, chainId: 137, time 2021-06-16T09:19:09Z, gasUsed 54139"
INFO [02-20|22:48:33.244] "→ committing new state"               eventRecord=424,669
INFO [02-20|22:48:33.244] "→ committed new state"                eventRecord="id 424669, contract 0xA6FA4fB5f76172d178d61B04b0ecd319C5d1C0aa, data: 0x87a7811f4bfedea3d341ad165680ae306b01aaeacc205d227629cf157dd9f821000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000840ea78f667b73853f4baece4a5ebe212c4039c1000000000000000000000000a0b86991c6218b36c1d19d4a2e9eb0ce3606eb48000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000000000000000000000000000004d9c77be95, txHash: 0x317ae025cb309f4bf24aab5ff5c61ac1f1188146b5ba1f5ec2aa4ee4380b310b, logIndex: 180, chainId: 137, time 2021-06-16T09:19:09Z, gasUsed 24539"
INFO [02-20|22:48:33.244] "→ committing new state"               eventRecord=424,670
INFO [02-20|22:48:33.246] "→ committed new state"                eventRecord="id 424670, contract 0xA6FA4fB5f76172d178d61B04b0ecd319C5d1C0aa, data: 0x87a7811f4bfedea3d341ad165680ae306b01aaeacc205d227629cf157dd9f821000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000a00000000000000000000000003d4517883b0212953e5136d2e261321b59158c85000000000000000000000000dac17f958d2ee523a2206206994597c13d831ec7000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000000000000000000000000000000754f989f8, txHash: 0x48128e4e5193803dbb8183841b4cf146e8be625d953e2888edbdee80f758b80f, logIndex: 275, chainId: 137, time 2021-06-16T09:19:09Z, gasUsed 57839"
INFO [02-20|22:48:33.246] "→ committing new state"               eventRecord=424,671
INFO [02-20|22:48:33.246] "→ committed new state"                eventRecord="id 424671, contract 0xA6FA4fB5f76172d178d61B04b0ecd319C5d1C0aa, data: 0x87a7811f4bfedea3d341ad165680ae306b01aaeacc205d227629cf157dd9f821000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000a000000000000000000000000005a14dbc97b5fdfc39f09a9c81acf6a3d748428f000000000000000000000000a0b86991c6218b36c1d19d4a2e9eb0ce3606eb480000000000000000000000000000000000000000000000000000000000000060000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000241cbe60, txHash: 0x3326e06584c4d00b1f5845728e74265e0258d79c65a748779007637852df91d0, logIndex: 353, chainId: 137, time 2021-06-16T09:19:09Z, gasUsed 41639"
INFO [02-20|22:48:33.246] "→ committing new state"               eventRecord=424,672
INFO [02-20|22:48:33.249] "→ committed new state"                eventRecord="id 424672, contract 0xD9c7C4ED4B66858301D0cb28Cc88bf655Fe34861, data: 0x000000000000000000000000f27474cb9bb9419fcf80ca93ee01c449102ec4d10000000000000000000000007d1afa7b718fb893db30a3abc0cfc608aacfebb00000000000000000000000000000000000000000000000039c3eeeb97aea280000000000000000000000000000000000000000000000000000000000085ad336, txHash: 0x6709ee61d5964058d8ab9945ba7e9789a59f0f9ccec4dc38d1c535c8ce95a278, logIndex: 144, chainId: 137, time 2021-06-16T09:19:39Z, gasUsed 70088"
INFO [02-20|22:48:33.249] "→ committing new state"               eventRecord=424,673
INFO [02-20|22:48:33.250] "→ committed new state"                eventRecord="id 424673, contract 0xA6FA4fB5f76172d178d61B04b0ecd319C5d1C0aa, data: 0x87a7811f4bfedea3d341ad165680ae306b01aaeacc205d227629cf157dd9f821000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000ed8454aab37befc3ee0f73ccf4c50b742edecefb0000000000000000000000002260fac5e5542a773aa44fbcfedf7c193bc2c5990000000000000000000000000000000000000000000000000000000000000060000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000004b6fd1, txHash: 0x074d3221b3781361027bedcbcbd8ce9bce424e3a325d84d97e18b0afe5b9196b, logIndex: 260, chainId: 137, time 2021-06-16T09:19:39Z, gasUsed 38239"
INFO [02-20|22:48:33.250] "→ committing new state"               eventRecord=424,674
INFO [02-20|22:48:33.250] "→ committed new state"                eventRecord="id 424674, contract 0xD9c7C4ED4B66858301D0cb28Cc88bf655Fe34861, data: 0x0000000000000000000000004d3c1c2d507fe2701febf0133641aa3f3a37266a0000000000000000000000007d1afa7b718fb893db30a3abc0cfc608aacfebb0000000000000000000000000000000000000000000000005e06fac32d3049b5e00000000000000000000000000000000000000000000000000000000085ad337, txHash: 0x3f013612a9c93741cd5eaf7d22aa4646d683f265d4810207e1b527b2c4881220, logIndex: 121, chainId: 137, time 2021-06-16T09:19:39Z, gasUsed 75186"
INFO [02-20|22:48:33.250] StateSyncData                            gas=361,669 number=15,777,216 lastStateID=424,674 "total records"=7 "fetch time"=14 "process time"=8
INFO [02-20|22:48:34.366] Whitelisting milestone deferred          err="chain out of sync"
WARN [02-20|22:48:34.807] Parallel state processor failed          err="invalid merkle root (remote: 88fd9c1c83079842bf63d1fd57769f9269e11150d15ef8f12490f81a374ca738 local: f65523b78ef58d4e487880332d0548b9cf74cee695666b2c63114ed1fbfa09ee) dberr: %!w(<nil>)"
WARN [02-20|22:48:36.307] Parallel state processor failed          err="invalid merkle root (remote: d472b373b10357ef6ff92d64d3db622910894b275117a65f8b4e7b86782603eb local: dd23ab7724419031fb55d663342820d250b3b950eb915e8f8c9d3f753bdc5d15) dberr: %!w(<nil>)"
INFO [02-20|22:48:40.022] Imported new chain segment               number=15,777,252 hash=687d64..5fab11 blocks=42 txs=11589 mgas=858.028 elapsed=8.097s    mgasps=105.963 age=3y8mo3w snapdiffs=6.50MiB triediffs=145.90MiB triedirty=188.22MiB
INFO [02-20|22:48:44.461] Fetching state updates from Heimdall     fromID=424,675 to=2021-06-16T09:24:05Z
INFO [02-20|22:48:44.461] Fetching state sync events               queryParams="from-id=424675&to-time=1623835445&limit=50"
INFO [02-20|22:48:44.531] "→ committing new state"               eventRecord=424,675
INFO [02-20|22:48:44.532] "→ committed new state"                eventRecord="id 424675, contract 0xA6FA4fB5f76172d178d61B04b0ecd319C5d1C0aa, data: 0x87a7811f4bfedea3d341ad165680ae306b01aaeacc205d227629cf157dd9f821000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000a00000000000000000000000004be70f47a39eac265a5ccb6080c01b0ba82434c7000000000000000000000000eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee00000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000002aa1efb94e0000, txHash: 0x95f304470cd207941d8a6106f0d1ba42461463aa84247905f7adc59bb09472f0, logIndex: 228, chainId: 137, time 2021-06-16T09:20:10Z, gasUsed 66270"
INFO [02-20|22:48:44.532] "→ committing new state"               eventRecord=424,676
INFO [02-20|22:48:44.533] "→ committed new state"                eventRecord="id 424676, contract 0xA6FA4fB5f76172d178d61B04b0ecd319C5d1C0aa, data: 0x87a7811f4bfedea3d341ad165680ae306b01aaeacc205d227629cf157dd9f821000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000a00000000000000000000000004ad185bb58238573cf1e739641b5d8c6cfb944a7000000000000000000000000eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee00000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000016345785d8a0000, txHash: 0xfb254a4702c751dba48b1e1a2451bece30e72c6bca7cc5632edd09daf101d0f1, logIndex: 104, chainId: 137, time 2021-06-16T09:20:10Z, gasUsed 41068"
INFO [02-20|22:48:44.533] "→ committing new state"               eventRecord=424,677
INFO [02-20|22:48:44.533] "→ committed new state"                eventRecord="id 424677, contract 0xA6FA4fB5f76172d178d61B04b0ecd319C5d1C0aa, data: 0x87a7811f4bfedea3d341ad165680ae306b01aaeacc205d227629cf157dd9f821000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000d956dd349bb158b390e6bc4abd73faebd8194708000000000000000000000000eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee0000000000000000000000000000000000000000000000000000000000000060000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000107ad8f556c6c0000, txHash: 0xc94247f0fd6efba82865994a0e34865e9948dfd46bf120955ee25d789f81e00f, logIndex: 134, chainId: 137, time 2021-06-16T09:20:10Z, gasUsed 41170"
INFO [02-20|22:48:44.533] "→ committing new state"               eventRecord=424,678
INFO [02-20|22:48:44.534] "→ committed new state"                eventRecord="id 424678, contract 0xA6FA4fB5f76172d178d61B04b0ecd319C5d1C0aa, data: 0x87a7811f4bfedea3d341ad165680ae306b01aaeacc205d227629cf157dd9f821000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000abe1a48d2c17f18451cf2c7d5f807425bce4f92b000000000000000000000000eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000000000000000000000004563918244f40000, txHash: 0x2843bc256d1ca870085958b1c900a8dc12bfd1322af3997ccb4a6593be7f70de, logIndex: 138, chainId: 137, time 2021-06-16T09:20:10Z, gasUsed 41170"

[cffls]

Thanks for reporting this issue @donbeave . You can just ignore it as long as the node is able to continue syncing. The warning message indicated the parallel execution computed a different merkle root from what the block header had. The serial execution (a concurrent process) succeeded so the node was able to continue syncing. We will be looking into this.