pd client panics with "tsoStream.recvLoop internal panic"

[overvenus]

Bug Report

/opt/tidb/db.log:[2025/02/09 12:14:02.117 +00:00] [FATAL] [stream.go:330] ["tsoStream.recvLoop internal panic"] [stacktrace="github.com/tikv/pd/client/clients/tso.(*tsoStream).recvLoop.func1\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20250107032658-5c4ab57d68de/clients/tso/stream.go:330\nruntime.gopanic\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.6.linux-amd64/src/runtime/panic.go:785\ngo.uber.org/zap/zapcore.CheckWriteAction.OnWrite\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/go.uber.org/zap@v1.27.0/zapcore/entry.go:196\ngo.uber.org/zap/zapcore.(*CheckedEntry).Write\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/go.uber.org/zap@v1.27.0/zapcore/entry.go:262\ngo.uber.org/zap.(*Logger).Panic\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/go.uber.org/zap@v1.27.0/logger.go:285\ngithub.com/pingcap/log.Panic\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/github.com/pingcap/log@v1.1.1-0.20241212030209-7e3ff8601a2a/global.go:54\ngithub.com/tikv/pd/client/clients/tso.(*tsoDispatcher).checkMonotonicity\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20250107032658-5c4ab57d68de/clients/tso/dispatcher.go:528\ngithub.com/tikv/pd/client/clients/tso.(*tsoDispatcher).processRequests.func2\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20250107032658-5c4ab57d68de/clients/tso/dispatcher.go:468\ngithub.com/tikv/pd/client/clients/tso.(*tsoStream).recvLoop\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20250107032658-5c4ab57d68de/clients/tso/stream.go:444"] [panicMessage="[tso] timestamp fallback"] [stack="github.com/tikv/pd/client/clients/tso.(*tsoStream).recvLoop.func1\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20250107032658-5c4ab57d68de/clients/tso/stream.go:330\nruntime.gopanic\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.6.linux-amd64/src/runtime/panic.go:785\ngo.uber.org/zap/zapcore.CheckWriteAction.OnWrite\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/go.uber.org/zap@v1.27.0/zapcore/entry.go:196\ngo.uber.org/zap/zapcore.(*CheckedEntry).Write\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/go.uber.org/zap@v1.27.0/zapcore/entry.go:262\ngo.uber.org/zap.(*Logger).Panic\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/go.uber.org/zap@v1.27.0/logger.go:285\ngithub.com/pingcap/log.Panic\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/github.com/pingcap/log@v1.1.1-0.20241212030209-7e3ff8601a2a/global.go:54\ngithub.com/tikv/pd/client/clients/tso.(*tsoDispatcher).checkMonotonicity\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20250107032658-5c4ab57d68de/clients/tso/dispatcher.go:528\ngithub.com/tikv/pd/client/clients/tso.(*tsoDispatcher).processRequests.func2\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20250107032658-5c4ab57d68de/clients/tso/dispatcher.go:468\ngithub.com/tikv/pd/client/clients/tso.(*tsoStream).recvLoop\n\t/home/jenkins/agent/workspace/build_tidb_multi_branch_master/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20250107032658-5c4ab57d68de/clients/tso/stream.go:444"]

Test: https://tcms.pingcap.net/dashboard/executions/case/13077637

What did you do?

Run jepsen tests.

What did you expect to see?

No panic.

What version of PD are you using (pd-server -V)?

TiDB: c292ec642b3fd69904c4d5223415ba6a2a040a51
PD: debceaf