[Bug]: Testcases not supported due to missing system EEPROM but not skipped

[hdwhdw]

Issue Description

Many tests, particularly those with log analyzer enabled, can fail on KVM because KVM does not support system EEPROM:

admin@vlab-01:~$ sudo decode-syseeprom -s
Failed to read system EEPROM info

admin@vlab-01:~$ show platform syseeprom
admin@vlab-01:~$

An example is the gnmi/test_gnoi_killlprocess.py, which will succeed in killing containers but when the container relaunch, this script will fail due to the aforemention issue: https://github.com/sonic-net/sonic-buildimage/blob/39e2131a7b76f6c3d5257b7e02c540dd33a24d5b/files/build_templates/docker_image_ctl.j2#L114

In particular, this line won't produce a valid command:

    $SONIC_DB_CLI STATE_DB HSET 'DEVICE_METADATA|localhost' chassis_serial_number $(decode-syseeprom -s)

See #15507

Results you see

test_gnoi_killprocess_then_restart[snmp-True-] fail

Results you expected to see

test succeeding without the failed log analyzer part.

Is it platform specific

VS

Relevant log output

E               Failed: Processes "['analyze_logs--<MultiAsicSonicHost vlab-01>']" failed with exit code "1"
E               Exception:
E               match: 2
E               expected_match: 0
E               expected_missing_match: 0
E               
E               Match Messages:
E               2025 Jan 16 20:24:46.622122 vlab-01 ERR sonic-db-cli: :- guard: RedisReply catches system_error: command: *9\r\n$4\r\nHSET\r\n$25\r\nDEVICE_METADATA|localhost\r\n$21\r\nchassis_serial_number\r\n$6\r\nFailed\r\n$2\r\nto\r\n$4\r\nread\r\n$6\r\nsystem\r\n$6\r\nEEPROM\r\n$4\r\ninfo\r\n, reason: ERR wrong number of arguments for 'hset' command: Input/output error
E               
E               2025 Jan 16 20:24:46.622209 vlab-01 INFO snmp.sh[715493]: RedisReply catches system_error: command: *9\r\n$4\r\nHSET\r\n$25\r\nDEVICE_METADATA|localhost\r\n$21\r\nchassis_serial_number\r\n$6\r\nFailed\r\n$2\r\nto\r\n$4\r\nread\r\n$6\r\nsystem\r\n$6\r\nEEPROM\r\n$4\r\ninfo\r\n, reason: ERR wrong number of arguments for 'hset' command: Input/output error: Input/output error
E               
E               Traceback:
E               Traceback (most recent call last):
E                 File "/data/sonic-mgmt/tests/common/helpers/parallel.py", line 35, in run
E                   Process.run(self)
E                 File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
E                   self._target(*self._args, **self._kwargs)
E                 File "/data/sonic-mgmt/tests/common/helpers/parallel.py", line 245, in wrapper
E                   target(*args, **kwargs)
E                 File "/data/sonic-mgmt/tests/common/plugins/loganalyzer/__init__.py", line 45, in analyze_logs
E                   dut_analyzer.analyze(markers[node.hostname], fail_test, store_la_logs=store_la_logs)
E                 File "/data/sonic-mgmt/tests/common/plugins/loganalyzer/loganalyzer.py", line 409, in analyze
E                   self._verify_log(analyzer_summary)
E                 File "/data/sonic-mgmt/tests/common/plugins/loganalyzer/loganalyzer.py", line 140, in _verify_log
E                   raise LogAnalyzerError(result_str)
E               tests.common.plugins.loganalyzer.loganalyzer.LogAnalyzerError: match: 2
E               expected_match: 0
E               expected_missing_match: 0
E               
E               Match Messages:
E               2025 Jan 16 20:24:46.622122 vlab-01 ERR sonic-db-cli: :- guard: RedisReply catches system_error: command: *9\r\n$4\r\nHSET\r\n$25\r\nDEVICE_METADATA|localhost\r\n$21\r\nchassis_serial_number\r\n$6\r\nFailed\r\n$2\r\nto\r\n$4\r\nread\r\n$6\r\nsystem\r\n$6\r\nEEPROM\r\n$4\r\ninfo\r\n, reason: ERR wrong number of arguments for 'hset' command: Input/output error
E               
E               2025 Jan 16 20:24:46.622209 vlab-01 INFO snmp.sh[715493]: RedisReply catches system_error: command: *9\r\n$4\r\nHSET\r\n$25\r\nDEVICE_METADATA|localhost\r\n$21\r\nchassis_serial_number\r\n$6\r\nFailed\r\n$2\r\nto\r\n$4\r\nread\r\n$6\r\nsystem\r\n$6\r\nEEPROM\r\n$4\r\ninfo\r\n, reason: ERR wrong number of arguments for 'hset' command: Input/output error: Input/output error

Output of show version

SONiC Software Version: SONiC.master.745173-3a621a083
SONiC OS Version: 12
Distribution: Debian 12.6
Kernel: 6.1.0-22-2-amd64
Build commit: 3a621a083
Build date: Mon Jan 13 13:31:40 UTC 2025
Built by: azureuser@a1d13faac000001

Platform: x86_64-kvm_x86_64-r0
HwSKU: Force10-S6000
ASIC: vs
ASIC Count: 1
Serial Number: N/A
Model Number: N/A
Hardware Revision: N/A
Uptime: 19:16:51 up 4 days,  2:41,  1 user,  load average: 0.12, 0.11, 0.08
Date: Tue 21 Jan 2025 19:16:51

Docker images:
REPOSITORY                    TAG                       IMAGE ID       SIZE
docker-macsec                 latest                    d2499e581c66   346MB
docker-teamd                  latest                    80b8b322536f   343MB
docker-teamd                  master.745173-3a621a083   80b8b322536f   343MB
docker-sflow                  latest                    3759e150cc9e   344MB
docker-sflow                  master.745173-3a621a083   3759e150cc9e   344MB
docker-orchagent              latest                    1ebb07abac1d   356MB
docker-orchagent              master.745173-3a621a083   1ebb07abac1d   356MB
docker-fpm-frr                latest                    356100e32d95   377MB
docker-fpm-frr                master.745173-3a621a083   356100e32d95   377MB
docker-nat                    latest                    c67cc407c99c   346MB
docker-nat                    master.745173-3a621a083   c67cc407c99c   346MB
docker-syncd-vs               latest                    9717d2c9ae82   2.16GB
docker-syncd-vs               master.745173-3a621a083   9717d2c9ae82   2.16GB
docker-gbsyncd-vs             latest                    2aa392c3d6a8   2.16GB
docker-gbsyncd-vs             master.745173-3a621a083   2aa392c3d6a8   2.16GB
docker-sonic-bmp              latest                    ec0f49b7b982   314MB
docker-sonic-bmp              master.745173-3a621a083   ec0f49b7b982   314MB
docker-dhcp-relay             latest                    ecc933d6a850   323MB
docker-platform-monitor       latest                    f13c6384763c   433MB
docker-platform-monitor       master.745173-3a621a083   f13c6384763c   433MB
docker-snmp                   latest                    fddbf4945a0b   353MB
docker-snmp                   master.745173-3a621a083   fddbf4945a0b   353MB
docker-eventd                 latest                    50bf77936fde   313MB
docker-eventd                 master.745173-3a621a083   50bf77936fde   313MB
docker-lldp                   latest                    15e93bf6efe9   359MB
docker-lldp                   master.745173-3a621a083   15e93bf6efe9   359MB
docker-mux                    latest                    5c9112ba5915   365MB
docker-mux                    master.745173-3a621a083   5c9112ba5915   365MB
docker-database               latest                    34ef9deb8ce1   322MB
docker-database               master.745173-3a621a083   34ef9deb8ce1   322MB
docker-sonic-gnmi             latest                    f3dd5415da65   402MB
docker-sonic-gnmi             master.745173-3a621a083   f3dd5415da65   402MB
docker-dash-engine            latest                    afcf0f939ed9   697MB
docker-dash-engine            master.745173-3a621a083   afcf0f939ed9   697MB
docker-sonic-mgmt-framework   latest                    f3e7d9d0919c   399MB
docker-sonic-mgmt-framework   master.745173-3a621a083   f3e7d9d0919c   399MB
docker-router-advertiser      latest                    68e8ff21b708   313MB
docker-router-advertiser      master.745173-3a621a083   68e8ff21b708   313MB

Attach files (if any)

No response

[yxieca]

@qiluo-msft can you help triage this issue? @wangxin should this issue be treated as a generic infrastructure issue?

[yxieca]

@judyjoseph why do we need to get chassis serial number at start up time?

[bingwang-ms]

Can this issue be addressed by PR #16303?

[yxieca]

@hdwhdw can you confirm this issue has been fixed?

[hdwhdw]

@yxieca I think this should be fixed on images instead of just skipping the test. This is a real bug on kvm.

I'll send out the PRs:
sonic-net/sonic-utilities#3750
sonic-net/sonic-buildimage#21633

[hdwhdw]

@bingwang-ms #16303 simply skip the test, but the output of the test suggests a real image issue (on KVM).

Sent out PRs for fixes:
sonic-net/sonic-buildimage#21633
sonic-net/sonic-utilities#3750