Skip to content

[inventory] record caboose SIGN value #8021

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

[inventory] record caboose SIGN value #8021

wants to merge 8 commits into from

Conversation

karencfv
Copy link
Contributor

Cabooses for RoT archives and RoT bootloader archives (but not SP archives) contain a SIGN key that's a hash of the public key that the RoT needs its software signed by. Software that's updating the RoT or RoT bootloader needs this information to choose the appropriate Hubris archive.

This commit collects the value during inventory, so Reconfigurator will have it available.

Closes: #7914

@karencfv karencfv requested a review from davepacheco April 22, 2025 06:59
davepacheco
davepacheco reviewed Apr 22, 2025
View reviewed changes
Copy link
Collaborator

@davepacheco davepacheco left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice -- that was fast! I have a question below about the database query.

The bigger thing is: do we want to have separate types (and maybe database constraints) where the sign field is not nullable? I mean: if we expect all RoT and RoT bootloader cabooses to have SIGN field but the others don't (is that right?), should we have separate types to reflect this so that when looking at an RoT/RoT bootloader caboose you don't have to check for this? What do you think @jgallagher?

nexus/db-queries/src/db/datastore/inventory.rs Outdated
Comment on lines 647 to 648
.and(dsl_sw_caboose::sign.eq(
found_caboose.caboose.sign.clone(),
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this work regardless of whether found_caboose or the given row's caboose is NULL? I thought sometimes you needed to use IS NOT NULL and/or IS NULL.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are absolutely right. I read up on the documentation and it looks like need to change this.

comparisons with NULL using = or <> will result in NULL, not true or false. Use IS NULL or IS NOT NULL for such checks.

Sadly, because of diesel, the modified code ends up being really janky. I kept getting mistmatched arms/values so I ended up with this:

let selection_inner_join_on =  dsl_baseboard_id::part_number
    .eq(baseboard_id.part_number.clone())
    .and(
        dsl_baseboard_id::serial_number.eq(
            baseboard_id.serial_number.clone(),
        ),
    )
    .and(dsl_sw_caboose::board.eq(
        found_caboose.caboose.board.clone(),
    ))
    .and(
        dsl_sw_caboose::git_commit.eq(
            found_caboose
                .caboose
                .git_commit
                .clone(),
        ),
    )
    .and(
        dsl_sw_caboose::name.eq(found_caboose
            .caboose
            .name
            .clone()),
    )
    .and(dsl_sw_caboose::version.eq(
        found_caboose.caboose.version.clone(),
    )
);

let select_by = (
    dsl_baseboard_id::id,
    dsl_sw_caboose::id,
    db_collection_id
        .into_sql::<diesel::sql_types::Uuid>(),
    found_caboose
        .time_collected
        .into_sql::<diesel::sql_types::Timestamptz>(),
    found_caboose
        .source
        .clone()
        .into_sql::<diesel::sql_types::Text>(),
    db_which.into_sql::<CabooseWhichEnum>(),
);

let columns = (
    dsl_inv_caboose::hw_baseboard_id,
    dsl_inv_caboose::sw_caboose_id,
    dsl_inv_caboose::inv_collection_id,
    dsl_inv_caboose::time_collected,
    dsl_inv_caboose::source,
    dsl_inv_caboose::which,
);

match found_caboose.caboose.sign.clone() {
    None => {
        let selection = nexus_db_schema::schema::hw_baseboard_id::table.inner_join(
            nexus_db_schema::schema::sw_caboose::table.on(
                selection_inner_join_on
                .and(dsl_sw_caboose::sign.is_null())
            ),
        )
        .select(select_by);

        let _ = diesel::insert_into(nexus_db_schema::schema::inv_caboose::table)
        .values(selection)
        .into_columns(columns)
        .execute_async(&conn)
        .await?;
    },
    Some(_) => {
        let selection = nexus_db_schema::schema::hw_baseboard_id::table.inner_join(
            nexus_db_schema::schema::sw_caboose::table.on(
                selection_inner_join_on
                .and(dsl_sw_caboose::sign.is_not_null())
            ),
        )
        .select(select_by);

        let _ = diesel::insert_into(nexus_db_schema::schema::inv_caboose::table)
        .values(selection)
        .into_columns(columns)
        .execute_async(&conn)
        .await?;
    }
};

It's awful. Also, it would get even messier if another nullable field appears. Do you have any suggestions on how to clean this up?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use https://docs.rs/diesel/latest/diesel/expression_methods/trait.PgExpressionMethods.html#method.is_not_distinct_from instead of eq? (Got there via diesel-rs/diesel#1306, and cockroach does seem to support it too.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ooohhh very nice, thanks!

@davepacheco
Copy link
Collaborator

It'd be neat to see the output of omdb db inventory collections show latest sp from a racklette running this!

karencfv
karencfv commented Apr 23, 2025
View reviewed changes
Copy link
Contributor Author

@karencfv karencfv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Success! Thanks for the review @davepacheco, I think the output is as expected?

root@oxz_switch1:~# omdb db inventory collections show latest sp
note: database URL not specified.  Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd00:1122:3344:102::3]:32221,[fd00:1122:3344:103::3]:32221,[fd00:1122:3344:104::4]:32221,[fd00:1122:3344:104::3]:32221,[fd00:1122:3344:101::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (139.0.0)
collection: 6a3d8864-e56d-4f52-bc54-db5d4e88d00c
collector:  d41629fe-37c6-4752-8698-3241f66c8399 (likely a Nexus instance)
started:    2025-04-23T03:33:40.613Z
done:       2025-04-23T03:33:52.322Z
errors:     0

Sled BRM42220036
    part number: 913-0000019
    power:    A0
    revision: 6
    MGS slot: Sled 14 (cubby 14)
    found at: 2025-04-23 03:33:41.761459 UTC from http://[fd00:1122:3344:101::2]:12225
    cabooses:
        SLOT       BOARD       NAME        VERSION GIT_COMMIT                               SIGN                                                             
        RotSlotA   oxide-rot-1 oxide-rot-1 1.0.33  441b09270d8715275d6b3c3b57d84dba539e34d0 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        RotSlotB   oxide-rot-1 oxide-rot-1 1.0.34  71e20463580a8e17b8b1eb8172cc3cdbece479a6 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        Stage0     oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        Stage0Next oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
    RoT pages:
        SLOT         DATA_BASE64                         
        Cmpa         oAAAeAAAAAAAAAAAAAAAAP8CAP3/AgD9... 
        CfpaActive   AAAAAGsAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaInactive AAAAAGsAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaScratch  AAAAAGsAAAAAAAAAAAAAAAAAAAAAAAAA... 
    RoT: active slot: slot B
    RoT: persistent boot preference: slot B
    RoT: pending persistent boot preference: -
    RoT: transient boot preference: -
    RoT: slot A SHA3-256: 2a6a4b0c6494ae51ce4216de5d03a600d0cface267a1c6f854155ac4e41036fc
    RoT: slot B SHA3-256: 16b846eb11c9307d208624bddff21b3dc0797fe7b364feaac202eb4f92996e50

Sled BRM42220062
    part number: 913-0000019
    power:    A0
    revision: 6
    MGS slot: Sled 15 (cubby 15)
    found at: 2025-04-23 03:33:44.027992 UTC from http://[fd00:1122:3344:101::2]:12225
    cabooses:
        SLOT       BOARD       NAME        VERSION GIT_COMMIT                               SIGN                                                             
        RotSlotA   oxide-rot-1 oxide-rot-1 1.0.34  71e20463580a8e17b8b1eb8172cc3cdbece479a6 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        RotSlotB   oxide-rot-1 oxide-rot-1 1.0.33  441b09270d8715275d6b3c3b57d84dba539e34d0 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        Stage0     oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        Stage0Next oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
    RoT pages:
        SLOT         DATA_BASE64                         
        Cmpa         oAAAeAAAAAAAAAAAAAAAAP8CAP3/AgD9... 
        CfpaActive   AAAAAGsAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaInactive AAAAAGsAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaScratch  AAAAAGsAAAAAAAAAAAAAAAAAAAAAAAAA... 
    RoT: active slot: slot A
    RoT: persistent boot preference: slot A
    RoT: pending persistent boot preference: -
    RoT: transient boot preference: -
    RoT: slot A SHA3-256: 9ba01fc19057d6fa0618a5918b6b11d41cf724aba41f27dd48ec06f89aea7fed
    RoT: slot B SHA3-256: 1f511bd014845f30bff9b047afbaeef6734f9fda4315a1708b0b65821882c567

Sled BRM42220030
    part number: 913-0000019
    power:    A0
    revision: 6
    MGS slot: Sled 16 (cubby 16)
    found at: 2025-04-23 03:33:40.631552 UTC from http://[fd00:1122:3344:101::2]:12225
    cabooses:
        SLOT       BOARD       NAME        VERSION GIT_COMMIT                               SIGN                                                             
        RotSlotA   oxide-rot-1 oxide-rot-1 1.0.33  441b09270d8715275d6b3c3b57d84dba539e34d0 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        RotSlotB   oxide-rot-1 oxide-rot-1 1.0.34  71e20463580a8e17b8b1eb8172cc3cdbece479a6 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        Stage0     oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        Stage0Next oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
    RoT pages:
        SLOT         DATA_BASE64                         
        Cmpa         oAAAeAAAAAAAAAAAAAAAAP8CAP3/AgD9... 
        CfpaActive   AAAAAE8AAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaInactive AAAAAE8AAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaScratch  AAAAAE8AAAAAAAAAAAAAAAAAAAAAAAAA... 
    RoT: active slot: slot B
    RoT: persistent boot preference: slot B
    RoT: pending persistent boot preference: -
    RoT: transient boot preference: -
    RoT: slot A SHA3-256: 2a6a4b0c6494ae51ce4216de5d03a600d0cface267a1c6f854155ac4e41036fc
    RoT: slot B SHA3-256: 16b846eb11c9307d208624bddff21b3dc0797fe7b364feaac202eb4f92996e50

Sled BRM44220007
    part number: 913-0000019
    power:    A0
    revision: 6
    MGS slot: Sled 17 (cubby 17)
    found at: 2025-04-23 03:33:42.893452 UTC from http://[fd00:1122:3344:101::2]:12225
    cabooses:
        SLOT       BOARD       NAME        VERSION GIT_COMMIT                               SIGN                                                             
        RotSlotA   oxide-rot-1 oxide-rot-1 1.0.33  441b09270d8715275d6b3c3b57d84dba539e34d0 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        RotSlotB   oxide-rot-1 oxide-rot-1 1.0.34  71e20463580a8e17b8b1eb8172cc3cdbece479a6 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        Stage0     oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        Stage0Next oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
    RoT pages:
        SLOT         DATA_BASE64                         
        Cmpa         oAAAeAAAAAAAAAAAAAAAAP8CAP3/AgD9... 
        CfpaActive   AAAAAGsAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaInactive AAAAAGsAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaScratch  AAAAAGsAAAAAAAAAAAAAAAAAAAAAAAAA... 
    RoT: active slot: slot B
    RoT: persistent boot preference: slot B
    RoT: pending persistent boot preference: -
    RoT: transient boot preference: -
    RoT: slot A SHA3-256: 2a6a4b0c6494ae51ce4216de5d03a600d0cface267a1c6f854155ac4e41036fc
    RoT: slot B SHA3-256: 16b846eb11c9307d208624bddff21b3dc0797fe7b364feaac202eb4f92996e50

Power BRM11230017
    part number: 913-0000003
    power:    A2
    revision: 9
    MGS slot: Power 0
    found at: 2025-04-23 03:33:45.163019 UTC from http://[fd00:1122:3344:101::2]:12225
    cabooses:
        SLOT BOARD NAME VERSION GIT_COMMIT SIGN 
    RoT pages:
        SLOT         DATA_BASE64                         
        Cmpa         oAAAiAAAAAAAAAAAAAAAAP8CAP3/AgD9... 
        CfpaActive   AAAAABIAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaInactive AAAAABEAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaScratch  AAAAABIAAAAAAAAAAAAAAAAAAAAAAAAA... 
    RoT: active slot: slot A
    RoT: persistent boot preference: slot A
    RoT: pending persistent boot preference: -
    RoT: transient boot preference: -
    RoT: slot A SHA3-256: 17cf2709c1d55dcc839898e3073d9db0e37f540201f0cdb5bdae1ae0932327ac
    RoT: slot B SHA3-256: 42ccae12461701473cb2750e3b85bcfb3afaf8e881cfda711f625b33fb9eb932

Switch BRM31230004
    part number: 913-0000006
    power:    A0
    revision: 10
    MGS slot: Switch 0
    found at: 2025-04-23 03:33:47.631639 UTC from http://[fd00:1122:3344:101::2]:12225
    cabooses:
        SLOT BOARD NAME VERSION GIT_COMMIT SIGN 
    RoT pages:
        SLOT         DATA_BASE64                         
        Cmpa         oAAAiAAAAAAAAAAAAAAAAP8CAP3/AgD9... 
        CfpaActive   AAAAAAcAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaInactive AAAAAAcAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaScratch  AAAAAAcAAAAAAAAAAAAAAAAAAAAAAAAA... 
    RoT: active slot: slot B
    RoT: persistent boot preference: slot B
    RoT: pending persistent boot preference: -
    RoT: transient boot preference: -
    RoT: slot A SHA3-256: ccd280614a9cce6919bc935e797ec8ff3a060745a2cab5a0fa7a69e66a3f336b
    RoT: slot B SHA3-256: c63635d08181e02ecd56eb3a29c42382b36ac327b68da2cce13eda255cde09fc

Switch BRM44220013
    part number: 913-0000006
    power:    A0
    revision: 4
    MGS slot: Switch 1
    found at: 2025-04-23 03:33:46.288474 UTC from http://[fd00:1122:3344:101::2]:12225
    cabooses:
        SLOT BOARD NAME VERSION GIT_COMMIT SIGN 
    RoT pages:
        SLOT         DATA_BASE64                         
        Cmpa         oAAAeAAAAAAAAAAAAAAAAP8CAP3/AgD9... 
        CfpaActive   AAAAACoAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaInactive AAAAACoAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaScratch  AAAAACoAAAAAAAAAAAAAAAAAAAAAAAAA... 
    RoT: active slot: slot A
    RoT: persistent boot preference: slot A
    RoT: pending persistent boot preference: -
    RoT: transient boot preference: -
    RoT: slot A SHA3-256: ccd280614a9cce6919bc935e797ec8ff3a060745a2cab5a0fa7a69e66a3f336b
    RoT: slot B SHA3-256: 09259c3a0255eb861b1119c3e61dd088416a41f7a6674c28364ef36e67cb70b7

nexus/db-queries/src/db/datastore/inventory.rs Outdated
Comment on lines 647 to 648
.and(dsl_sw_caboose::sign.eq(
found_caboose.caboose.sign.clone(),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are absolutely right. I read up on the documentation and it looks like need to change this.

comparisons with NULL using = or <> will result in NULL, not true or false. Use IS NULL or IS NOT NULL for such checks.

Sadly, because of diesel, the modified code ends up being really janky. I kept getting mistmatched arms/values so I ended up with this:

let selection_inner_join_on =  dsl_baseboard_id::part_number
    .eq(baseboard_id.part_number.clone())
    .and(
        dsl_baseboard_id::serial_number.eq(
            baseboard_id.serial_number.clone(),
        ),
    )
    .and(dsl_sw_caboose::board.eq(
        found_caboose.caboose.board.clone(),
    ))
    .and(
        dsl_sw_caboose::git_commit.eq(
            found_caboose
                .caboose
                .git_commit
                .clone(),
        ),
    )
    .and(
        dsl_sw_caboose::name.eq(found_caboose
            .caboose
            .name
            .clone()),
    )
    .and(dsl_sw_caboose::version.eq(
        found_caboose.caboose.version.clone(),
    )
);

let select_by = (
    dsl_baseboard_id::id,
    dsl_sw_caboose::id,
    db_collection_id
        .into_sql::<diesel::sql_types::Uuid>(),
    found_caboose
        .time_collected
        .into_sql::<diesel::sql_types::Timestamptz>(),
    found_caboose
        .source
        .clone()
        .into_sql::<diesel::sql_types::Text>(),
    db_which.into_sql::<CabooseWhichEnum>(),
);

let columns = (
    dsl_inv_caboose::hw_baseboard_id,
    dsl_inv_caboose::sw_caboose_id,
    dsl_inv_caboose::inv_collection_id,
    dsl_inv_caboose::time_collected,
    dsl_inv_caboose::source,
    dsl_inv_caboose::which,
);

match found_caboose.caboose.sign.clone() {
    None => {
        let selection = nexus_db_schema::schema::hw_baseboard_id::table.inner_join(
            nexus_db_schema::schema::sw_caboose::table.on(
                selection_inner_join_on
                .and(dsl_sw_caboose::sign.is_null())
            ),
        )
        .select(select_by);

        let _ = diesel::insert_into(nexus_db_schema::schema::inv_caboose::table)
        .values(selection)
        .into_columns(columns)
        .execute_async(&conn)
        .await?;
    },
    Some(_) => {
        let selection = nexus_db_schema::schema::hw_baseboard_id::table.inner_join(
            nexus_db_schema::schema::sw_caboose::table.on(
                selection_inner_join_on
                .and(dsl_sw_caboose::sign.is_not_null())
            ),
        )
        .select(select_by);

        let _ = diesel::insert_into(nexus_db_schema::schema::inv_caboose::table)
        .values(selection)
        .into_columns(columns)
        .execute_async(&conn)
        .await?;
    }
};

It's awful. Also, it would get even messier if another nullable field appears. Do you have any suggestions on how to clean this up?

@davepacheco
Copy link
Collaborator

@karencfv it seems like the SP caboose information is missing from that output. Did that get broken somehow?

@karencfv
Copy link
Contributor Author

@karencfv it seems like the SP caboose information is missing from that output. Did that get broken somehow?

@davepacheco, ha yes! Looks like the SPs did not have a sign value, and that's why they disappeared. I ran the latest code with @jgallagher's suggestion, and that issue was resolved. The SP caboose information shows up everywhere now.

I'm slightly concerned with Power BRM11230017, Switch BRM31230004 and Switch BRM44220013 though. They now only show SP caboose information, but nothing for RoT nor Stage. I'm a little confused as to why those would be missing that information, or is it expected from a raclette?

root@oxz_switch1:~# omdb db inventory collections show latest sp
note: database URL not specified.  Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd00:1122:3344:104::4]:32221,[fd00:1122:3344:102::3]:32221,[fd00:1122:3344:104::3]:32221,[fd00:1122:3344:101::3]:32221,[fd00:1122:3344:103::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (139.0.0)
collection: 4bf0ca98-d7c7-4fc1-9ee7-68251ee900db
collector:  656a766d-0c46-41cc-9f30-7bddf4713a63 (likely a Nexus instance)
started:    2025-04-24T02:45:03.140Z
done:       2025-04-24T02:45:14.984Z
errors:     0

Sled BRM42220036
    part number: 913-0000019
    power:    A0
    revision: 6
    MGS slot: Sled 14 (cubby 14)
    found at: 2025-04-24 02:45:04.462185 UTC from http://[fd00:1122:3344:101::2]:12225
    cabooses:
        SLOT       BOARD       NAME        VERSION GIT_COMMIT                               SIGN                                                             
        SpSlot0    gimlet-c    gimlet-c    1.0.36  a31b6acf6900c04978302b5a1dcc017e9b35e1fc n/a                                                              
        SpSlot1    gimlet-c    gimlet-c    1.0.34  441b09270d8715275d6b3c3b57d84dba539e34d0 n/a                                                              
        RotSlotA   oxide-rot-1 oxide-rot-1 1.0.33  441b09270d8715275d6b3c3b57d84dba539e34d0 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        RotSlotB   oxide-rot-1 oxide-rot-1 1.0.34  71e20463580a8e17b8b1eb8172cc3cdbece479a6 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        Stage0     oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        Stage0Next oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
    RoT pages:
        SLOT         DATA_BASE64                         
        Cmpa         oAAAeAAAAAAAAAAAAAAAAP8CAP3/AgD9... 
        CfpaActive   AAAAAGsAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaInactive AAAAAGsAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaScratch  AAAAAGsAAAAAAAAAAAAAAAAAAAAAAAAA... 
    RoT: active slot: slot B
    RoT: persistent boot preference: slot B
    RoT: pending persistent boot preference: -
    RoT: transient boot preference: -
    RoT: slot A SHA3-256: 2a6a4b0c6494ae51ce4216de5d03a600d0cface267a1c6f854155ac4e41036fc
    RoT: slot B SHA3-256: 16b846eb11c9307d208624bddff21b3dc0797fe7b364feaac202eb4f92996e50

Sled BRM42220062
    part number: 913-0000019
    power:    A0
    revision: 6
    MGS slot: Sled 15 (cubby 15)
    found at: 2025-04-24 02:45:06.719661 UTC from http://[fd00:1122:3344:101::2]:12225
    cabooses:
        SLOT       BOARD       NAME        VERSION GIT_COMMIT                               SIGN                                                             
        SpSlot0    gimlet-c    gimlet-c    1.0.36  a31b6acf6900c04978302b5a1dcc017e9b35e1fc n/a                                                              
        SpSlot1    gimlet-c    gimlet-c    1.0.34  441b09270d8715275d6b3c3b57d84dba539e34d0 n/a                                                              
        RotSlotA   oxide-rot-1 oxide-rot-1 1.0.34  71e20463580a8e17b8b1eb8172cc3cdbece479a6 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        RotSlotB   oxide-rot-1 oxide-rot-1 1.0.33  441b09270d8715275d6b3c3b57d84dba539e34d0 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        Stage0     oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        Stage0Next oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
    RoT pages:
        SLOT         DATA_BASE64                         
        Cmpa         oAAAeAAAAAAAAAAAAAAAAP8CAP3/AgD9... 
        CfpaActive   AAAAAGsAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaInactive AAAAAGsAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaScratch  AAAAAGsAAAAAAAAAAAAAAAAAAAAAAAAA... 
    RoT: active slot: slot A
    RoT: persistent boot preference: slot A
    RoT: pending persistent boot preference: -
    RoT: transient boot preference: -
    RoT: slot A SHA3-256: 9ba01fc19057d6fa0618a5918b6b11d41cf724aba41f27dd48ec06f89aea7fed
    RoT: slot B SHA3-256: 1f511bd014845f30bff9b047afbaeef6734f9fda4315a1708b0b65821882c567

Sled BRM42220030
    part number: 913-0000019
    power:    A0
    revision: 6
    MGS slot: Sled 16 (cubby 16)
    found at: 2025-04-24 02:45:03.208444 UTC from http://[fd00:1122:3344:101::2]:12225
    cabooses:
        SLOT       BOARD       NAME        VERSION GIT_COMMIT                               SIGN                                                             
        SpSlot0    gimlet-c    gimlet-c    1.0.36  a31b6acf6900c04978302b5a1dcc017e9b35e1fc n/a                                                              
        SpSlot1    gimlet-c    gimlet-c    1.0.34  441b09270d8715275d6b3c3b57d84dba539e34d0 n/a                                                              
        RotSlotA   oxide-rot-1 oxide-rot-1 1.0.33  441b09270d8715275d6b3c3b57d84dba539e34d0 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        RotSlotB   oxide-rot-1 oxide-rot-1 1.0.34  71e20463580a8e17b8b1eb8172cc3cdbece479a6 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        Stage0     oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        Stage0Next oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
    RoT pages:
        SLOT         DATA_BASE64                         
        Cmpa         oAAAeAAAAAAAAAAAAAAAAP8CAP3/AgD9... 
        CfpaActive   AAAAAE8AAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaInactive AAAAAE8AAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaScratch  AAAAAE8AAAAAAAAAAAAAAAAAAAAAAAAA... 
    RoT: active slot: slot B
    RoT: persistent boot preference: slot B
    RoT: pending persistent boot preference: -
    RoT: transient boot preference: -
    RoT: slot A SHA3-256: 2a6a4b0c6494ae51ce4216de5d03a600d0cface267a1c6f854155ac4e41036fc
    RoT: slot B SHA3-256: 16b846eb11c9307d208624bddff21b3dc0797fe7b364feaac202eb4f92996e50

Sled BRM44220007
    part number: 913-0000019
    power:    A0
    revision: 6
    MGS slot: Sled 17 (cubby 17)
    found at: 2025-04-24 02:45:05.589204 UTC from http://[fd00:1122:3344:101::2]:12225
    cabooses:
        SLOT       BOARD       NAME        VERSION GIT_COMMIT                               SIGN                                                             
        SpSlot0    gimlet-c    gimlet-c    1.0.36  a31b6acf6900c04978302b5a1dcc017e9b35e1fc n/a                                                              
        SpSlot1    gimlet-c    gimlet-c    1.0.32  fad1769684b1ca6e71398db03b685b2bd4d4dce8 n/a                                                              
        RotSlotA   oxide-rot-1 oxide-rot-1 1.0.33  441b09270d8715275d6b3c3b57d84dba539e34d0 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        RotSlotB   oxide-rot-1 oxide-rot-1 1.0.34  71e20463580a8e17b8b1eb8172cc3cdbece479a6 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        Stage0     oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        Stage0Next oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
    RoT pages:
        SLOT         DATA_BASE64                         
        Cmpa         oAAAeAAAAAAAAAAAAAAAAP8CAP3/AgD9... 
        CfpaActive   AAAAAGsAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaInactive AAAAAGsAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaScratch  AAAAAGsAAAAAAAAAAAAAAAAAAAAAAAAA... 
    RoT: active slot: slot B
    RoT: persistent boot preference: slot B
    RoT: pending persistent boot preference: -
    RoT: transient boot preference: -
    RoT: slot A SHA3-256: 2a6a4b0c6494ae51ce4216de5d03a600d0cface267a1c6f854155ac4e41036fc
    RoT: slot B SHA3-256: 16b846eb11c9307d208624bddff21b3dc0797fe7b364feaac202eb4f92996e50

Power BRM11230017
    part number: 913-0000003
    power:    A2
    revision: 9
    MGS slot: Power 0
    found at: 2025-04-24 02:45:07.953357 UTC from http://[fd00:1122:3344:101::2]:12225
    cabooses:
        SLOT    BOARD NAME  VERSION GIT_COMMIT                               SIGN 
        SpSlot0 psc-c psc-c 1.0.33  441b09270d8715275d6b3c3b57d84dba539e34d0 n/a  
        SpSlot1 psc-c psc-c 1.0.33  441b09270d8715275d6b3c3b57d84dba539e34d0 n/a  
    RoT pages:
        SLOT         DATA_BASE64                         
        Cmpa         oAAAiAAAAAAAAAAAAAAAAP8CAP3/AgD9... 
        CfpaActive   AAAAABIAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaInactive AAAAABEAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaScratch  AAAAABIAAAAAAAAAAAAAAAAAAAAAAAAA... 
    RoT: active slot: slot A
    RoT: persistent boot preference: slot A
    RoT: pending persistent boot preference: -
    RoT: transient boot preference: -
    RoT: slot A SHA3-256: 17cf2709c1d55dcc839898e3073d9db0e37f540201f0cdb5bdae1ae0932327ac
    RoT: slot B SHA3-256: 42ccae12461701473cb2750e3b85bcfb3afaf8e881cfda711f625b33fb9eb932

Switch BRM31230004
    part number: 913-0000006
    power:    A0
    revision: 10
    MGS slot: Switch 0
    found at: 2025-04-24 02:45:11.429204 UTC from http://[fd00:1122:3344:101::2]:12225
    cabooses:
        SLOT    BOARD     NAME      VERSION GIT_COMMIT                               SIGN 
        SpSlot0 sidecar-c sidecar-c 1.0.37  a31b6acf6900c04978302b5a1dcc017e9b35e1fc n/a  
        SpSlot1 sidecar-c sidecar-c 1.0.35  441b09270d8715275d6b3c3b57d84dba539e34d0 n/a  
    RoT pages:
        SLOT         DATA_BASE64                         
        Cmpa         oAAAiAAAAAAAAAAAAAAAAP8CAP3/AgD9... 
        CfpaActive   AAAAAAcAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaInactive AAAAAAcAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaScratch  AAAAAAcAAAAAAAAAAAAAAAAAAAAAAAAA... 
    RoT: active slot: slot B
    RoT: persistent boot preference: slot B
    RoT: pending persistent boot preference: -
    RoT: transient boot preference: -
    RoT: slot A SHA3-256: ccd280614a9cce6919bc935e797ec8ff3a060745a2cab5a0fa7a69e66a3f336b
    RoT: slot B SHA3-256: c63635d08181e02ecd56eb3a29c42382b36ac327b68da2cce13eda255cde09fc

Switch BRM44220013
    part number: 913-0000006
    power:    A0
    revision: 4
    MGS slot: Switch 1
    found at: 2025-04-24 02:45:10.201652 UTC from http://[fd00:1122:3344:101::2]:12225
    cabooses:
        SLOT    BOARD     NAME      VERSION GIT_COMMIT                               SIGN 
        SpSlot0 sidecar-b sidecar-b 1.0.35  441b09270d8715275d6b3c3b57d84dba539e34d0 n/a  
        SpSlot1 sidecar-b sidecar-b 1.0.35  441b09270d8715275d6b3c3b57d84dba539e34d0 n/a  
    RoT pages:
        SLOT         DATA_BASE64                         
        Cmpa         oAAAeAAAAAAAAAAAAAAAAP8CAP3/AgD9... 
        CfpaActive   AAAAACoAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaInactive AAAAACoAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaScratch  AAAAACoAAAAAAAAAAAAAAAAAAAAAAAAA... 
    RoT: active slot: slot A
    RoT: persistent boot preference: slot A
    RoT: pending persistent boot preference: -
    RoT: transient boot preference: -
    RoT: slot A SHA3-256: ccd280614a9cce6919bc935e797ec8ff3a060745a2cab5a0fa7a69e66a3f336b
    RoT: slot B SHA3-256: 09259c3a0255eb861b1119c3e61dd088416a41f7a6674c28364ef36e67cb70b7

davepacheco
davepacheco reviewed Apr 24, 2025
View reviewed changes
nexus/types/src/inventory.rs Outdated
Comment on lines 263 to 265
// It is very unlikely that the sign hash will not be present.
// As far as we know, it will only be locally built BART signed images.
// We will be handling those edge cases by setting this field as an option.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't quite right. As far as I've seen, standard SP images do not provide a value here. This is probably the most specific thing I can say:

The sign will generally be present for production RoT and RoT bootloader images. It's currently absent from SP images and could be absent from RoT images as well.

but somebody else might know better.

@karencfv
Copy link
Contributor Author

Alright, so I checked London again. Someone else had used it and left their environment running so I just ran omdb db inventory collections show latest sp there to see what it shows and sure enough, all of the information is there for Power BRM11230017, Switch BRM31230004 and Switch BRM44220013:

<...>
Power BRM11230017
    part number: 913-0000003
    power:    A2
    revision: 9
    MGS slot: Power 0
    found at: 2025-04-27 21:45:47.989994 UTC from http://[fd00:1122:3344:101::2]:12225
    cabooses:
        SLOT       BOARD       NAME        VERSION GIT_COMMIT                               
        SpSlot0    psc-c       psc-c       1.0.33  441b09270d8715275d6b3c3b57d84dba539e34d0 
        SpSlot1    psc-c       psc-c       1.0.33  441b09270d8715275d6b3c3b57d84dba539e34d0 
        RotSlotA   oxide-rot-1 oxide-rot-1 1.0.34  71e20463580a8e17b8b1eb8172cc3cdbece479a6 
        RotSlotB   oxide-rot-1 oxide-rot-1 1.0.33  441b09270d8715275d6b3c3b57d84dba539e34d0 
        Stage0     oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 
        Stage0Next oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 
    RoT pages:
        SLOT         DATA_BASE64                         
        Cmpa         oAAAiAAAAAAAAAAAAAAAAP8CAP3/AgD9... 
        CfpaActive   AAAAABIAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaInactive AAAAABEAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaScratch  AAAAABIAAAAAAAAAAAAAAAAAAAAAAAAA... 
    RoT: active slot: slot A
    RoT: persistent boot preference: slot A
    RoT: pending persistent boot preference: -
    RoT: transient boot preference: -
    RoT: slot A SHA3-256: 17cf2709c1d55dcc839898e3073d9db0e37f540201f0cdb5bdae1ae0932327ac
    RoT: slot B SHA3-256: 42ccae12461701473cb2750e3b85bcfb3afaf8e881cfda711f625b33fb9eb932

Switch BRM31230004
    part number: 913-0000006
    power:    A0
    revision: 10
    MGS slot: Switch 0
    found at: 2025-04-27 21:45:50.811270 UTC from http://[fd00:1122:3344:101::2]:12225
    cabooses:
        SLOT       BOARD       NAME        VERSION GIT_COMMIT                               
        SpSlot0    sidecar-c   sidecar-c   1.0.37  a31b6acf6900c04978302b5a1dcc017e9b35e1fc 
        SpSlot1    sidecar-c   sidecar-c   1.0.35  441b09270d8715275d6b3c3b57d84dba539e34d0 
        RotSlotA   oxide-rot-1 oxide-rot-1 1.0.34  71e20463580a8e17b8b1eb8172cc3cdbece479a6 
        RotSlotB   oxide-rot-1 oxide-rot-1 1.0.34  71e20463580a8e17b8b1eb8172cc3cdbece479a6 
        Stage0     oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 
        Stage0Next oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 
    RoT pages:
        SLOT         DATA_BASE64                         
        Cmpa         oAAAiAAAAAAAAAAAAAAAAP8CAP3/AgD9... 
        CfpaActive   AAAAAAcAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaInactive AAAAAAcAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaScratch  AAAAAAcAAAAAAAAAAAAAAAAAAAAAAAAA... 
    RoT: active slot: slot B
    RoT: persistent boot preference: slot B
    RoT: pending persistent boot preference: -
    RoT: transient boot preference: -
    RoT: slot A SHA3-256: ccd280614a9cce6919bc935e797ec8ff3a060745a2cab5a0fa7a69e66a3f336b
    RoT: slot B SHA3-256: c63635d08181e02ecd56eb3a29c42382b36ac327b68da2cce13eda255cde09fc

Switch BRM44220013
    part number: 913-0000006
    power:    A0
    revision: 4
    MGS slot: Switch 1
    found at: 2025-04-27 21:45:49.540447 UTC from http://[fd00:1122:3344:101::2]:12225
    cabooses:
        SLOT       BOARD       NAME        VERSION GIT_COMMIT                               
        SpSlot0    sidecar-b   sidecar-b   1.0.35  441b09270d8715275d6b3c3b57d84dba539e34d0 
        SpSlot1    sidecar-b   sidecar-b   1.0.35  441b09270d8715275d6b3c3b57d84dba539e34d0 
        RotSlotA   oxide-rot-1 oxide-rot-1 1.0.34  71e20463580a8e17b8b1eb8172cc3cdbece479a6 
        RotSlotB   oxide-rot-1 oxide-rot-1 1.0.33  441b09270d8715275d6b3c3b57d84dba539e34d0 
        Stage0     oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 
        Stage0Next oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 
    RoT pages:
        SLOT         DATA_BASE64                         
        Cmpa         oAAAeAAAAAAAAAAAAAAAAP8CAP3/AgD9... 
        CfpaActive   AAAAACoAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaInactive AAAAACoAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaScratch  AAAAACoAAAAAAAAAAAAAAAAAAAAAAAAA... 
    RoT: active slot: slot A
    RoT: persistent boot preference: slot A
    RoT: pending persistent boot preference: -
    RoT: transient boot preference: -
    RoT: slot A SHA3-256: ccd280614a9cce6919bc935e797ec8ff3a060745a2cab5a0fa7a69e66a3f336b
    RoT: slot B SHA3-256: 09259c3a0255eb861b1119c3e61dd088416a41f7a6674c28364ef36e67cb70b7

@karencfv
Copy link
Contributor Author

Hm, I'm a bit at a loss as to what to do next, I saw the conversation on the update channel (It was a public holiday in NZ that day). so I'll continue there when it's everyone's Monday!

@karencfv
Copy link
Contributor Author

karencfv commented Apr 29, 2025 •
edited
Loading

Alright, so I checked the nexus logs when running omdb db inventory collections show latest sp and It all looks perfectly fine to me. A bunch of API calls that all responded with a 200 status

<...>
04:15:49.643Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/16/component/sp/caboose?firmware_slot=0
04:15:49.907Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/16/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "715c76ed-5d1d-4e6e-baca-fb27150f00c6", "content-length": "138", "date": "Tue, 29 Apr 2025 04:15:49 GMT"} })
04:15:49.907Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/16/component/sp/caboose?firmware_slot=1
04:15:50.187Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/16/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aae68602-8106-4241-b411-7ba056e45bf3", "content-length": "138", "date": "Tue, 29 Apr 2025 04:15:49 GMT"} })
04:15:50.187Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/16/component/rot/caboose?firmware_slot=0
04:15:50.474Z INFO 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): ensured all Crucible datasets present in inventory have database records
    background_task = blueprint_rendezvous
    file = nexus/reconfigurator/rendezvous/src/crucible_dataset.rs:113
    num_already_exist = 38
    num_inserted = 0
    num_not_in_inventory = 0
04:15:50.474Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): activation complete
    background_task = blueprint_rendezvous
    elapsed = 1.531141764s
    iteration = 5
    status = {"blueprint_id":"dbc0988d-d570-4d7f-a483-4035474a3b49","inventory_collection_id":"dcf36bb0-6214-4eb7-b056-136dafaa678b","stats":{"crucible_dataset":{"num_already_exist":38,"num_inserted":0,"num_not_in_inventory":0},"debug_dataset":{"num_already_exist":38,"num_already_tombstoned":0,"num_inserted":0,"num_not_in_inventory":0,"num_tombstoned":0}}}
04:15:50.642Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/16/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0f90604-8e92-4702-b198-5246f577fc42", "content-length": "206", "date": "Tue, 29 Apr 2025 04:15:50 GMT"} })
04:15:50.642Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/16/component/rot/caboose?firmware_slot=1
04:15:51.040Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/16/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d9cf42c-c77c-401a-b7a0-de5e2b60f362", "content-length": "206", "date": "Tue, 29 Apr 2025 04:15:50 GMT"} })
04:15:51.040Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/16/component/stage0/caboose?firmware_slot=0
04:15:51.558Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/16/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "016d38e6-b847-445c-bf7d-aee993b1fa7c", "content-length": "205", "date": "Tue, 29 Apr 2025 04:15:51 GMT"} })
04:15:51.558Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/16/component/stage0/caboose?firmware_slot=1
04:15:52.077Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/16/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee9be848-cb05-45af-9f3c-da95046c89fd", "content-length": "205", "date": "Tue, 29 Apr 2025 04:15:51 GMT"} })
04:15:52.077Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/16/component/rot/cmpa
04:15:52.140Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/16/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "740da58f-b4d7-4933-b528-975a7ad1fa72", "content-length": "702", "date": "Tue, 29 Apr 2025 04:15:51 GMT"} })
04:15:52.140Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = Some(Body)
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/16/component/rot/cfpa
04:15:52.216Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/16/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "94305b95-32fd-4837-abca-51ef2c3c5301", "content-length": "727", "date": "Tue, 29 Apr 2025 04:15:51 GMT"} })
04:15:52.216Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = Some(Body)
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/16/component/rot/cfpa
04:15:52.271Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/16/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "967f14e3-b262-4767-b1d1-3389ef0471de", "content-length": "729", "date": "Tue, 29 Apr 2025 04:15:51 GMT"} })
04:15:52.271Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = Some(Body)
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/16/component/rot/cfpa
04:15:52.323Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/16/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7aedfad2-c5d9-4ac6-8595-b0c064780260", "content-length": "728", "date": "Tue, 29 Apr 2025 04:15:51 GMT"} })
04:15:52.323Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/14
04:15:52.336Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/14", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8b6e1fe-cf16-4b9c-8ef6-9d855b504eef", "content-length": "733", "date": "Tue, 29 Apr 2025 04:15:51 GMT"} })
04:15:52.336Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/14/component/sp/caboose?firmware_slot=0
04:15:52.338Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/14/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "378e671c-c681-4820-a953-cb2188132f63", "content-length": "138", "date": "Tue, 29 Apr 2025 04:15:51 GMT"} })
04:15:52.338Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/14/component/sp/caboose?firmware_slot=1
04:15:52.340Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/14/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "869c0a4b-25a6-4764-afa5-7c002aa569f9", "content-length": "138", "date": "Tue, 29 Apr 2025 04:15:51 GMT"} })
04:15:52.340Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/14/component/rot/caboose?firmware_slot=0
04:15:52.612Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/14/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b9433bb-903f-43c1-b551-2c019262a32f", "content-length": "206", "date": "Tue, 29 Apr 2025 04:15:52 GMT"} })
04:15:52.612Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/14/component/rot/caboose?firmware_slot=1
04:15:52.886Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/14/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ec5fa9d-053d-4bf7-9244-47c469f06817", "content-length": "206", "date": "Tue, 29 Apr 2025 04:15:52 GMT"} })
04:15:52.886Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/14/component/stage0/caboose?firmware_slot=0
04:15:53.150Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/14/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf1de89e-de46-4036-a647-103057674c01", "content-length": "205", "date": "Tue, 29 Apr 2025 04:15:52 GMT"} })
04:15:53.150Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/14/component/stage0/caboose?firmware_slot=1
04:15:53.571Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/14/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "abc143f0-4bdf-4f72-bb37-52d9ceb4481a", "content-length": "205", "date": "Tue, 29 Apr 2025 04:15:53 GMT"} })
04:15:53.571Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/14/component/rot/cmpa
04:15:53.649Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/14/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "efada1ec-38b7-4c31-87a4-9f7a35473dc8", "content-length": "702", "date": "Tue, 29 Apr 2025 04:15:53 GMT"} })
04:15:53.649Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = Some(Body)
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/14/component/rot/cfpa
04:15:53.704Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/14/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "08a3a421-d6a2-45c4-b54d-d970965d89cf", "content-length": "727", "date": "Tue, 29 Apr 2025 04:15:53 GMT"} })
04:15:53.704Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = Some(Body)
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/14/component/rot/cfpa
04:15:53.756Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/14/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b90bd11d-45ac-4259-a352-7d9cbc194c8e", "content-length": "729", "date": "Tue, 29 Apr 2025 04:15:53 GMT"} })
04:15:53.756Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = Some(Body)
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/14/component/rot/cfpa
04:15:53.805Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/14/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ec1da05-84fc-4caf-945d-a0a8759c6b38", "content-length": "728", "date": "Tue, 29 Apr 2025 04:15:53 GMT"} })
04:15:53.805Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/17
04:15:53.818Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/17", status: 200, headers: {"content-type": "application/json", "x-request-id": "f820f157-3963-4994-ad94-7bf545b6feb2", "content-length": "735", "date": "Tue, 29 Apr 2025 04:15:53 GMT"} })
04:15:53.818Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/17/component/sp/caboose?firmware_slot=0
04:15:53.821Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/17/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "570a7d0c-8922-4ec3-869b-d62bb6254862", "content-length": "138", "date": "Tue, 29 Apr 2025 04:15:53 GMT"} })
04:15:53.821Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/17/component/sp/caboose?firmware_slot=1
04:15:53.824Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/17/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "36ff4361-c3d4-4a49-b90d-1d34f317ed99", "content-length": "138", "date": "Tue, 29 Apr 2025 04:15:53 GMT"} })
04:15:53.824Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/17/component/rot/caboose?firmware_slot=0
04:15:54.083Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/17/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1cb14a74-3897-44f0-b31e-acb65ccbb864", "content-length": "206", "date": "Tue, 29 Apr 2025 04:15:53 GMT"} })
04:15:54.083Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/17/component/rot/caboose?firmware_slot=1
04:15:54.341Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/17/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4af57d8-f117-407c-a29c-a8d29c1a32e1", "content-length": "206", "date": "Tue, 29 Apr 2025 04:15:53 GMT"} })
04:15:54.341Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/17/component/stage0/caboose?firmware_slot=0
04:15:54.635Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/17/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b3d0ff8-faae-4fb1-980d-6ba26249ba3e", "content-length": "205", "date": "Tue, 29 Apr 2025 04:15:54 GMT"} })
04:15:54.635Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/17/component/stage0/caboose?firmware_slot=1
04:15:54.907Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/17/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b16f9c9d-7804-49c1-88e9-d84c861dc408", "content-length": "205", "date": "Tue, 29 Apr 2025 04:15:54 GMT"} })
04:15:54.907Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/17/component/rot/cmpa
04:15:54.923Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/17/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d31787ad-0972-4482-ad0d-664c048063a3", "content-length": "702", "date": "Tue, 29 Apr 2025 04:15:54 GMT"} })
04:15:54.923Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = Some(Body)
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/17/component/rot/cfpa
04:15:54.943Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/17/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8af73a2b-93f9-4298-bc8b-d500e0d3731a", "content-length": "727", "date": "Tue, 29 Apr 2025 04:15:54 GMT"} })
04:15:54.943Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = Some(Body)
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/17/component/rot/cfpa
04:15:54.959Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/17/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "02a65290-6fdf-474c-ad2c-d4469701c01f", "content-length": "729", "date": "Tue, 29 Apr 2025 04:15:54 GMT"} })
04:15:54.959Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = Some(Body)
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/17/component/rot/cfpa
04:15:54.975Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/17/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "84370a9e-99e8-4ae1-b1b8-e6f706336ec2", "content-length": "728", "date": "Tue, 29 Apr 2025 04:15:54 GMT"} })
04:15:54.975Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/15
04:15:54.988Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/15", status: 200, headers: {"content-type": "application/json", "x-request-id": "1667cd02-4949-4327-89c3-0216f6971f06", "content-length": "733", "date": "Tue, 29 Apr 2025 04:15:54 GMT"} })
04:15:54.988Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/15/component/sp/caboose?firmware_slot=0
04:15:54.991Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/15/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f49e619f-a83b-4583-ab0e-88f590f1b593", "content-length": "138", "date": "Tue, 29 Apr 2025 04:15:54 GMT"} })
04:15:54.991Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/15/component/sp/caboose?firmware_slot=1
04:15:54.994Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/15/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ac5e688-c71f-40a9-a29d-d5744dc72690", "content-length": "138", "date": "Tue, 29 Apr 2025 04:15:54 GMT"} })
04:15:54.994Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/15/component/rot/caboose?firmware_slot=0
04:15:55.253Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/15/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "593cf4a6-cfad-4b77-b2a5-b8fcc42702c0", "content-length": "206", "date": "Tue, 29 Apr 2025 04:15:54 GMT"} })
04:15:55.253Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/15/component/rot/caboose?firmware_slot=1
04:15:55.522Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/15/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1f3640f-237f-417d-93d6-9379aa8cab0a", "content-length": "206", "date": "Tue, 29 Apr 2025 04:15:54 GMT"} })
04:15:55.522Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/15/component/stage0/caboose?firmware_slot=0
04:15:55.784Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/15/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "da09b904-9df6-4721-974c-513ba54c59b0", "content-length": "205", "date": "Tue, 29 Apr 2025 04:15:55 GMT"} })
04:15:55.784Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/15/component/stage0/caboose?firmware_slot=1
04:15:56.072Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/15/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5b35406-5364-4e3b-9134-8f7ff15e0984", "content-length": "205", "date": "Tue, 29 Apr 2025 04:15:55 GMT"} })
04:15:56.072Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/15/component/rot/cmpa
04:15:56.099Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/15/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c447cdee-ba85-45b8-8410-8410f1e1d550", "content-length": "702", "date": "Tue, 29 Apr 2025 04:15:55 GMT"} })
04:15:56.099Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = Some(Body)
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/15/component/rot/cfpa
04:15:56.131Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/15/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4bcca67d-c8e3-4b4a-8ce6-0d59d7d51420", "content-length": "727", "date": "Tue, 29 Apr 2025 04:15:55 GMT"} })
04:15:56.131Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = Some(Body)
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/15/component/rot/cfpa
04:15:56.167Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/15/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "03cda677-284b-48a6-aa73-790673480cc5", "content-length": "729", "date": "Tue, 29 Apr 2025 04:15:55 GMT"} })
04:15:56.167Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = Some(Body)
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/sled/15/component/rot/cfpa
04:15:56.198Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/sled/15/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e2d8fe7-ea0f-465a-86bb-e2fd57844926", "content-length": "728", "date": "Tue, 29 Apr 2025 04:15:55 GMT"} })
04:15:56.198Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/power/0
04:15:56.276Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/power/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3be98a16-ed92-4e87-bc04-2d9f54bda1be", "content-length": "733", "date": "Tue, 29 Apr 2025 04:15:55 GMT"} })
04:15:56.276Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/power/0/component/sp/caboose?firmware_slot=0
04:15:56.314Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/power/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0dd4e012-5a4d-42f2-bf11-b3af4f195496", "content-length": "132", "date": "Tue, 29 Apr 2025 04:15:55 GMT"} })
04:15:56.314Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/power/0/component/sp/caboose?firmware_slot=1
04:15:56.506Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/power/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "59c302ed-4589-48cb-ad48-4718d19d12bf", "content-length": "132", "date": "Tue, 29 Apr 2025 04:15:56 GMT"} })
04:15:56.506Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/power/0/component/rot/caboose?firmware_slot=0
04:15:57.029Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/power/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab311045-9fc9-48ca-91db-58c0f7459eba", "content-length": "206", "date": "Tue, 29 Apr 2025 04:15:57 GMT"} })
04:15:57.029Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/power/0/component/rot/caboose?firmware_slot=1
04:15:57.595Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/power/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ae42c86-deaa-4c57-924e-421089ae6c71", "content-length": "206", "date": "Tue, 29 Apr 2025 04:15:57 GMT"} })
04:15:57.595Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/power/0/component/stage0/caboose?firmware_slot=0
04:15:58.113Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/power/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a06814a6-3eb6-4707-be8d-ef8f10466624", "content-length": "205", "date": "Tue, 29 Apr 2025 04:15:57 GMT"} })
04:15:58.113Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/power/0/component/stage0/caboose?firmware_slot=1
04:15:58.379Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/power/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c26ad2b-b4f5-46c1-9217-f077830f9ae0", "content-length": "205", "date": "Tue, 29 Apr 2025 04:15:58 GMT"} })
04:15:58.379Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/power/0/component/rot/cmpa
04:15:58.395Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/power/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3196c6aa-4564-4db0-bfad-711d436d1d65", "content-length": "702", "date": "Tue, 29 Apr 2025 04:15:58 GMT"} })
04:15:58.395Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = Some(Body)
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/power/0/component/rot/cfpa
04:15:58.410Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/power/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eff71605-332f-48e2-abbb-ea9c7b1821af", "content-length": "727", "date": "Tue, 29 Apr 2025 04:15:58 GMT"} })
04:15:58.410Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = Some(Body)
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/power/0/component/rot/cfpa
04:15:58.429Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/power/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3d12ef2-7860-4204-b221-31a0ba8ce568", "content-length": "729", "date": "Tue, 29 Apr 2025 04:15:58 GMT"} })
04:15:58.429Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = Some(Body)
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/power/0/component/rot/cfpa
04:15:58.450Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/power/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7938693-677f-4a9e-8c17-d8bf5e490b65", "content-length": "728", "date": "Tue, 29 Apr 2025 04:15:58 GMT"} })
04:15:58.450Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/1
04:15:58.532Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2385411-ca8c-4b40-ae12-267776357191", "content-length": "733", "date": "Tue, 29 Apr 2025 04:15:58 GMT"} })
04:15:58.532Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/1/component/sp/caboose?firmware_slot=0
04:15:58.545Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "69c3edad-6c48-4d93-aaaa-b9b96fb1a363", "content-length": "140", "date": "Tue, 29 Apr 2025 04:15:58 GMT"} })
04:15:58.545Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/1/component/sp/caboose?firmware_slot=1
04:15:58.551Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b43669c7-7226-41db-abb1-2f5c640c77fe", "content-length": "140", "date": "Tue, 29 Apr 2025 04:15:58 GMT"} })
04:15:58.551Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/1/component/rot/caboose?firmware_slot=0
04:15:58.913Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e84ba272-80fd-407f-b466-000ed05caadf", "content-length": "206", "date": "Tue, 29 Apr 2025 04:15:58 GMT"} })
04:15:58.913Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/1/component/rot/caboose?firmware_slot=1
04:15:59.177Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "84779f10-d4ba-4c3b-9172-7fafbdb6511e", "content-length": "206", "date": "Tue, 29 Apr 2025 04:15:58 GMT"} })
04:15:59.177Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/1/component/stage0/caboose?firmware_slot=0
04:15:59.445Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "59b0f6d7-f8e5-4f30-b96a-f97a90872b67", "content-length": "205", "date": "Tue, 29 Apr 2025 04:15:58 GMT"} })
04:15:59.445Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/1/component/stage0/caboose?firmware_slot=1
04:15:59.732Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "87407041-0565-4d70-a063-2c77cbeba1e0", "content-length": "205", "date": "Tue, 29 Apr 2025 04:15:59 GMT"} })
04:15:59.732Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/1/component/rot/cmpa
04:15:59.752Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "538b941b-45f9-465f-90b9-4c20339ea34a", "content-length": "702", "date": "Tue, 29 Apr 2025 04:15:59 GMT"} })
04:15:59.752Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = Some(Body)
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/1/component/rot/cfpa
04:15:59.769Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "54450d12-ce40-48b4-b2a8-93ca5cd6232e", "content-length": "727", "date": "Tue, 29 Apr 2025 04:15:59 GMT"} })
04:15:59.769Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = Some(Body)
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/1/component/rot/cfpa
04:15:59.796Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "37d2302b-3594-4054-a9e4-7c46318afe9e", "content-length": "729", "date": "Tue, 29 Apr 2025 04:15:59 GMT"} })
04:15:59.796Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = Some(Body)
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/1/component/rot/cfpa
04:15:59.829Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4dac85a-0814-4ca2-8686-9a2517a4e253", "content-length": "728", "date": "Tue, 29 Apr 2025 04:15:59 GMT"} })
04:15:59.829Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0
04:15:59.842Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2826cd86-d290-4387-9464-eebb775a7ee5", "content-length": "735", "date": "Tue, 29 Apr 2025 04:15:59 GMT"} })
04:15:59.842Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=0
04:15:59.844Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb716a63-88d5-44f9-aa00-a1769e9a1a39", "content-length": "140", "date": "Tue, 29 Apr 2025 04:15:59 GMT"} })
04:15:59.844Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=1
04:15:59.847Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "97ea66fb-4ca8-4c3a-8cb8-c017dfefe854", "content-length": "140", "date": "Tue, 29 Apr 2025 04:15:59 GMT"} })
04:15:59.847Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=0
04:16:00.105Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8a1a7dd-6b3a-406e-8fa1-5b7053059cdb", "content-length": "206", "date": "Tue, 29 Apr 2025 04:15:59 GMT"} })
04:16:00.105Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=1
04:16:00.363Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e295963c-3585-493b-8326-89afa0703257", "content-length": "206", "date": "Tue, 29 Apr 2025 04:15:59 GMT"} })
04:16:00.363Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=0
04:16:00.657Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3170b46a-0375-449d-abce-8b9ed2018f1a", "content-length": "205", "date": "Tue, 29 Apr 2025 04:16:00 GMT"} })
04:16:00.657Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client request
    background_task = inventory_collection
    body = None
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    method = GET
    uri = http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=1
04:16:00.920Z DEBG 4c9e3b5e-7634-43bd-92da-2fad4d67b3e1 (ServerContext): client response
    background_task = inventory_collection
    gateway_url = http://[fd00:1122:3344:101::2]:12225
    result = Ok(Response { url: "http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "25fc8794-de05-4d1a-a3f2-4ee1034cd50d", "content-length": "205", "date": "Tue, 29 Apr 2025 04:16:00 GMT"} })
<...>

I ran every single one of the caboose API calls manually and the results are as follows:

Sled 16

root@oxz_nexus_4c9e3b5e:~# curl -s http://[fd00:1122:3344:101::2]:12225/sp/sled/16/component/sp/caboose?firmware_slot=0 | jq
curl -s http://[fd00:1122:3344:101::2]:12225/sp/sled/16/component/sp/caboose?firmware_slot=1 | jq
curl -s http://[fd00:1122:3344:101::2]:12225/sp/sled/16/component/rot/caboose?firmware_slot=0 | jq
curl -s http://[fd00:1122:3344:101::2]:12225/sp/sled/16/component/rot/caboose?firmware_slot=1 | jq
curl -s http://[fd00:1122:3344:101::2]:12225/sp/sled/16/component/stage0/caboose?firmware_slot=0 | jq
curl -s http://[fd00:1122:3344:101::2]:12225/sp/sled/16/component/stage0/caboose?firmware_slot=1 | jq
{
  "git_commit": "a31b6acf6900c04978302b5a1dcc017e9b35e1fc",
  "board": "gimlet-c",
  "name": "gimlet-c",
  "version": "1.0.36",
  "sign": null,
  "epoch": null
}
{
  "git_commit": "a31b6acf6900c04978302b5a1dcc017e9b35e1fc",
  "board": "gimlet-c",
  "name": "gimlet-c",
  "version": "1.0.36",
  "sign": null,
  "epoch": null
}
{
  "git_commit": "71e20463580a8e17b8b1eb8172cc3cdbece479a6",
  "board": "oxide-rot-1",
  "name": "oxide-rot-1",
  "version": "1.0.34",
  "sign": "11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf",
  "epoch": null
}
{
  "git_commit": "71e20463580a8e17b8b1eb8172cc3cdbece479a6",
  "board": "oxide-rot-1",
  "name": "oxide-rot-1",
  "version": "1.0.34",
  "sign": "11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf",
  "epoch": null
}
{
  "git_commit": "95e79111096d7ec258d68a2c6840b6062efc7f54",
  "board": "oxide-rot-1",
  "name": "oxide-rot-1",
  "version": "1.3.3",
  "sign": "11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf",
  "epoch": null
}
{
  "git_commit": "95e79111096d7ec258d68a2c6840b6062efc7f54",
  "board": "oxide-rot-1",
  "name": "oxide-rot-1",
  "version": "1.3.3",
  "sign": "11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf",
  "epoch": null
}

Sled 14

root@oxz_nexus_4c9e3b5e:~# curl -s http://[fd00:1122:3344:101::2]:12225/sp/sled/14/component/sp/caboose?firmware_slot=0 | jq
curl -s http://[fd00:1122:3344:101::2]:12225/sp/sled/14/component/sp/caboose?firmware_slot=1 | jq
curl -s http://[fd00:1122:3344:101::2]:12225/sp/sled/14/component/rot/caboose?firmware_slot=0 | jq
curl -s http://[fd00:1122:3344:101::2]:12225/sp/sled/14/component/rot/caboose?firmware_slot=1 | jq
curl -s http://[fd00:1122:3344:101::2]:12225/sp/sled/14/component/stage0/caboose?firmware_slot=0 | jq
curl -s http://[fd00:1122:3344:101::2]:12225/sp/sled/14/component/stage0/caboose?firmware_slot=1 | jq
{
  "git_commit": "a31b6acf6900c04978302b5a1dcc017e9b35e1fc",
  "board": "gimlet-c",
  "name": "gimlet-c",
  "version": "1.0.36",
  "sign": null,
  "epoch": null
}
{
  "git_commit": "a31b6acf6900c04978302b5a1dcc017e9b35e1fc",
  "board": "gimlet-c",
  "name": "gimlet-c",
  "version": "1.0.36",
  "sign": null,
  "epoch": null
}
{
  "git_commit": "71e20463580a8e17b8b1eb8172cc3cdbece479a6",
  "board": "oxide-rot-1",
  "name": "oxide-rot-1",
  "version": "1.0.34",
  "sign": "11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf",
  "epoch": null
}
{
  "git_commit": "71e20463580a8e17b8b1eb8172cc3cdbece479a6",
  "board": "oxide-rot-1",
  "name": "oxide-rot-1",
  "version": "1.0.34",
  "sign": "11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf",
  "epoch": null
}
{
  "git_commit": "95e79111096d7ec258d68a2c6840b6062efc7f54",
  "board": "oxide-rot-1",
  "name": "oxide-rot-1",
  "version": "1.3.3",
  "sign": "11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf",
  "epoch": null
}
{
  "git_commit": "95e79111096d7ec258d68a2c6840b6062efc7f54",
  "board": "oxide-rot-1",
  "name": "oxide-rot-1",
  "version": "1.3.3",
  "sign": "11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf",
  "epoch": null
}

Sled 17

root@oxz_nexus_4c9e3b5e:~# curl -s http://[fd00:1122:3344:101::2]:12225/sp/sled/17/component/sp/caboose?firmware_slot=0 | jq
curl -s http://[fd00:1122:3344:101::2]:12225/sp/sled/17/component/sp/caboose?firmware_slot=1 | jq
curl -s http://[fd00:1122:3344:101::2]:12225/sp/sled/17/component/rot/caboose?firmware_slot=0 | jq
curl -s http://[fd00:1122:3344:101::2]:12225/sp/sled/17/component/rot/caboose?firmware_slot=1 | jq
curl -s http://[fd00:1122:3344:101::2]:12225/sp/sled/17/component/stage0/caboose?firmware_slot=0 | jq
curl -s http://[fd00:1122:3344:101::2]:12225/sp/sled/17/component/stage0/caboose?firmware_slot=1 | jq
{
  "git_commit": "a31b6acf6900c04978302b5a1dcc017e9b35e1fc",
  "board": "gimlet-c",
  "name": "gimlet-c",
  "version": "1.0.36",
  "sign": null,
  "epoch": null
}
{
  "git_commit": "a31b6acf6900c04978302b5a1dcc017e9b35e1fc",
  "board": "gimlet-c",
  "name": "gimlet-c",
  "version": "1.0.36",
  "sign": null,
  "epoch": null
}
{
  "git_commit": "71e20463580a8e17b8b1eb8172cc3cdbece479a6",
  "board": "oxide-rot-1",
  "name": "oxide-rot-1",
  "version": "1.0.34",
  "sign": "11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf",
  "epoch": null
}
{
  "git_commit": "71e20463580a8e17b8b1eb8172cc3cdbece479a6",
  "board": "oxide-rot-1",
  "name": "oxide-rot-1",
  "version": "1.0.34",
  "sign": "11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf",
  "epoch": null
}
{
  "git_commit": "95e79111096d7ec258d68a2c6840b6062efc7f54",
  "board": "oxide-rot-1",
  "name": "oxide-rot-1",
  "version": "1.3.3",
  "sign": "11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf",
  "epoch": null
}
{
  "git_commit": "95e79111096d7ec258d68a2c6840b6062efc7f54",
  "board": "oxide-rot-1",
  "name": "oxide-rot-1",
  "version": "1.3.3",
  "sign": "11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf",
  "epoch": null
}

Sled 15

root@oxz_nexus_4c9e3b5e:~# curl -s http://[fd00:1122:3344:101::2]:12225/sp/sled/15/component/sp/caboose?firmware_slot=0 | jq
curl -s http://[fd00:1122:3344:101::2]:12225/sp/sled/15/component/sp/caboose?firmware_slot=1 | jq
curl -s http://[fd00:1122:3344:101::2]:12225/sp/sled/15/component/rot/caboose?firmware_slot=0 | jq
curl -s http://[fd00:1122:3344:101::2]:12225/sp/sled/15/component/rot/caboose?firmware_slot=1 | jq
curl -s http://[fd00:1122:3344:101::2]:12225/sp/sled/15/component/stage0/caboose?firmware_slot=0 | jq
curl -s http://[fd00:1122:3344:101::2]:12225/sp/sled/15/component/stage0/caboose?firmware_slot=1 | jq
{
  "git_commit": "a31b6acf6900c04978302b5a1dcc017e9b35e1fc",
  "board": "gimlet-c",
  "name": "gimlet-c",
  "version": "1.0.36",
  "sign": null,
  "epoch": null
}
{
  "git_commit": "a31b6acf6900c04978302b5a1dcc017e9b35e1fc",
  "board": "gimlet-c",
  "name": "gimlet-c",
  "version": "1.0.36",
  "sign": null,
  "epoch": null
}
{
  "git_commit": "71e20463580a8e17b8b1eb8172cc3cdbece479a6",
  "board": "oxide-rot-1",
  "name": "oxide-rot-1",
  "version": "1.0.34",
  "sign": "11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf",
  "epoch": null
}
{
  "git_commit": "71e20463580a8e17b8b1eb8172cc3cdbece479a6",
  "board": "oxide-rot-1",
  "name": "oxide-rot-1",
  "version": "1.0.34",
  "sign": "11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf",
  "epoch": null
}
{
  "git_commit": "95e79111096d7ec258d68a2c6840b6062efc7f54",
  "board": "oxide-rot-1",
  "name": "oxide-rot-1",
  "version": "1.3.3",
  "sign": "11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf",
  "epoch": null
}
{
  "git_commit": "95e79111096d7ec258d68a2c6840b6062efc7f54",
  "board": "oxide-rot-1",
  "name": "oxide-rot-1",
  "version": "1.3.3",
  "sign": "11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf",
  "epoch": null
}

Power

root@oxz_nexus_4c9e3b5e:~# curl -s http://[fd00:1122:3344:101::2]:12225/sp/power/0/component/sp/caboose?firmware_slot=0 | jq
curl -s http://[fd00:1122:3344:101::2]:12225/sp/power/0/component/sp/caboose?firmware_slot=1 | jq
curl -s http://[fd00:1122:3344:101::2]:12225/sp/power/0/component/rot/caboose?firmware_slot=0 | jq
curl -s http://[fd00:1122:3344:101::2]:12225/sp/power/0/component/rot/caboose?firmware_slot=1 | jq
curl -s http://[fd00:1122:3344:101::2]:12225/sp/power/0/component/stage0/caboose?firmware_slot=0 | jq
curl -s http://[fd00:1122:3344:101::2]:12225/sp/power/0/component/stage0/caboose?firmware_slot=1 | jq
{
  "git_commit": "a31b6acf6900c04978302b5a1dcc017e9b35e1fc",
  "board": "psc-c",
  "name": "psc-c",
  "version": "1.0.35",
  "sign": null,
  "epoch": null
}
{
  "git_commit": "441b09270d8715275d6b3c3b57d84dba539e34d0",
  "board": "psc-c",
  "name": "psc-c",
  "version": "1.0.33",
  "sign": null,
  "epoch": null
}
{
  "git_commit": "71e20463580a8e17b8b1eb8172cc3cdbece479a6",
  "board": "oxide-rot-1",
  "name": "oxide-rot-1",
  "version": "1.0.34",
  "sign": "f592d8f109b81881221eed5af6438abad9b5df8c220b9129c03763e7e10b22c7",
  "epoch": null
}
{
  "git_commit": "71e20463580a8e17b8b1eb8172cc3cdbece479a6",
  "board": "oxide-rot-1",
  "name": "oxide-rot-1",
  "version": "1.0.34",
  "sign": "f592d8f109b81881221eed5af6438abad9b5df8c220b9129c03763e7e10b22c7",
  "epoch": null
}
{
  "git_commit": "95e79111096d7ec258d68a2c6840b6062efc7f54",
  "board": "oxide-rot-1",
  "name": "oxide-rot-1",
  "version": "1.3.3",
  "sign": "f592d8f109b81881221eed5af6438abad9b5df8c220b9129c03763e7e10b22c7",
  "epoch": null
}
{
  "git_commit": "95e79111096d7ec258d68a2c6840b6062efc7f54",
  "board": "oxide-rot-1",
  "name": "oxide-rot-1",
  "version": "1.3.3",
  "sign": "f592d8f109b81881221eed5af6438abad9b5df8c220b9129c03763e7e10b22c7",
  "epoch": null
}

Switch 1

root@oxz_nexus_4c9e3b5e:~# curl -s http://[fd00:1122:3344:101::2]:12225/sp/switch/1/component/sp/caboose?firmware_slot=0 | jq
curl -s http://[fd00:1122:3344:101::2]:12225/sp/switch/1/component/sp/caboose?firmware_slot=1 | jq
curl -s http://[fd00:1122:3344:101::2]:12225/sp/switch/1/component/rot/caboose?firmware_slot=0 | jq
curl -s http://[fd00:1122:3344:101::2]:12225/sp/switch/1/component/rot/caboose?firmware_slot=1 | jq
curl -s http://[fd00:1122:3344:101::2]:12225/sp/switch/1/component/stage0/caboose?firmware_slot=0 | jq
curl -s http://[fd00:1122:3344:101::2]:12225/sp/switch/1/component/stage0/caboose?firmware_slot=1 | jq
{
  "git_commit": "a31b6acf6900c04978302b5a1dcc017e9b35e1fc",
  "board": "sidecar-b",
  "name": "sidecar-b",
  "version": "1.0.37",
  "sign": null,
  "epoch": null
}
{
  "git_commit": "441b09270d8715275d6b3c3b57d84dba539e34d0",
  "board": "sidecar-b",
  "name": "sidecar-b",
  "version": "1.0.35",
  "sign": null,
  "epoch": null
}
{
  "git_commit": "71e20463580a8e17b8b1eb8172cc3cdbece479a6",
  "board": "oxide-rot-1",
  "name": "oxide-rot-1",
  "version": "1.0.34",
  "sign": "1432cc4cfe5688c51b55546fe37837c753cfbc89e8c3c6aabcf977fdf0c41e27",
  "epoch": null
}
{
  "git_commit": "71e20463580a8e17b8b1eb8172cc3cdbece479a6",
  "board": "oxide-rot-1",
  "name": "oxide-rot-1",
  "version": "1.0.34",
  "sign": "1432cc4cfe5688c51b55546fe37837c753cfbc89e8c3c6aabcf977fdf0c41e27",
  "epoch": null
}
{
  "git_commit": "95e79111096d7ec258d68a2c6840b6062efc7f54",
  "board": "oxide-rot-1",
  "name": "oxide-rot-1",
  "version": "1.3.3",
  "sign": "1432cc4cfe5688c51b55546fe37837c753cfbc89e8c3c6aabcf977fdf0c41e27",
  "epoch": null
}
{
  "git_commit": "95e79111096d7ec258d68a2c6840b6062efc7f54",
  "board": "oxide-rot-1",
  "name": "oxide-rot-1",
  "version": "1.3.3",
  "sign": "1432cc4cfe5688c51b55546fe37837c753cfbc89e8c3c6aabcf977fdf0c41e27",
  "epoch": null
}

Switch 0

root@oxz_nexus_4c9e3b5e:~# curl -s http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=0 | jq
curl -s http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/sp/caboose?firmware_slot=1 | jq
curl -s http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=0 | jq
curl -s http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/rot/caboose?firmware_slot=1 | jq
curl -s http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=0 | jq
curl -s http://[fd00:1122:3344:101::2]:12225/sp/switch/0/component/stage0/caboose?firmware_slot=1 | jq
{
  "git_commit": "a31b6acf6900c04978302b5a1dcc017e9b35e1fc",
  "board": "sidecar-c",
  "name": "sidecar-c",
  "version": "1.0.37",
  "sign": null,
  "epoch": null
}
{
  "git_commit": "a31b6acf6900c04978302b5a1dcc017e9b35e1fc",
  "board": "sidecar-c",
  "name": "sidecar-c",
  "version": "1.0.37",
  "sign": null,
  "epoch": null
}
{
  "git_commit": "71e20463580a8e17b8b1eb8172cc3cdbece479a6",
  "board": "oxide-rot-1",
  "name": "oxide-rot-1",
  "version": "1.0.34",
  "sign": "1432cc4cfe5688c51b55546fe37837c753cfbc89e8c3c6aabcf977fdf0c41e27",
  "epoch": null
}
{
  "git_commit": "71e20463580a8e17b8b1eb8172cc3cdbece479a6",
  "board": "oxide-rot-1",
  "name": "oxide-rot-1",
  "version": "1.0.34",
  "sign": "1432cc4cfe5688c51b55546fe37837c753cfbc89e8c3c6aabcf977fdf0c41e27",
  "epoch": null
}
{
  "git_commit": "95e79111096d7ec258d68a2c6840b6062efc7f54",
  "board": "oxide-rot-1",
  "name": "oxide-rot-1",
  "version": "1.3.3",
  "sign": "1432cc4cfe5688c51b55546fe37837c753cfbc89e8c3c6aabcf977fdf0c41e27",
  "epoch": null
}
{
  "git_commit": "95e79111096d7ec258d68a2c6840b6062efc7f54",
  "board": "oxide-rot-1",
  "name": "oxide-rot-1",
  "version": "1.3.3",
  "sign": "1432cc4cfe5688c51b55546fe37837c753cfbc89e8c3c6aabcf977fdf0c41e27",
  "epoch": null
}

What's really strange now is that it seems that for Power BRM11230017, Switch BRM31230004 and Switch BRM44220013, the missing values for RoT and Stage0 that have a SIGN value, while the null ones (SP) are showing up just fine.

The only pattern I'm seeing here is that all of the SIGN values that show up are 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf while the ones that don't show up have another hash. Perhaps there is something wrong with the query that it somehow only shows records that are either 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf (the first SIGN value it fetched) or null? Does that even make sense?

Thoughts @davepacheco @jgallagher ?

@davepacheco
Copy link
Collaborator

Were you able to determine if the database inventory tables look correct or are the values missing (or otherwise wrong) there?

@jgallagher
Copy link
Contributor

Were you able to determine if the database inventory tables look correct or are the values missing (or otherwise wrong) there?

I hopped on london to look, and I think the database inventory tables do not look correct. I had suggested this in chat because I was suspicious of the insertions:

I'd definitely check whether there are rows in sw_caboose that don't have corresponding pointers in inv_caboose

But starting from the sw_caboose table, there's already a problem: we don't have rows for the switch or psc RoT cabooses at all:

root@[fd00:1122:3344:103::3]:32221/omicron> select * from sw_caboose;
                   id                  |    board    |                git_commit                |    name     | version |                               sign
---------------------------------------+-------------+------------------------------------------+-------------+---------+-------------------------------------------------------------------
  0ee2d038-add2-4105-89ca-e0e86415e015 | gimlet-c    | a31b6acf6900c04978302b5a1dcc017e9b35e1fc | gimlet-c    | 1.0.36  | NULL
  0fc79a96-5c84-4dae-b81b-37a4eab29d19 | sidecar-b   | a31b6acf6900c04978302b5a1dcc017e9b35e1fc | sidecar-b   | 1.0.37  | NULL
  519bac98-984a-41f9-8e54-a241e767d39e | oxide-rot-1 | 95e79111096d7ec258d68a2c6840b6062efc7f54 | oxide-rot-1 | 1.3.3   | 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf
  5450c326-0ffe-494c-be04-506f3bc93f9c | psc-c       | 441b09270d8715275d6b3c3b57d84dba539e34d0 | psc-c       | 1.0.33  | NULL
  69d9717f-2fae-44f5-b699-0dac6f3f59a2 | oxide-rot-1 | 71e20463580a8e17b8b1eb8172cc3cdbece479a6 | oxide-rot-1 | 1.0.34  | 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf
  b0ae7f60-bf9d-4827-8b29-3d08229a1d49 | sidecar-b   | 441b09270d8715275d6b3c3b57d84dba539e34d0 | sidecar-b   | 1.0.35  | NULL
  bf24c608-8328-4321-bc6c-62c96244a592 | sidecar-c   | a31b6acf6900c04978302b5a1dcc017e9b35e1fc | sidecar-c   | 1.0.37  | NULL
  ee9e40e6-4113-4fa4-9f37-c721e4c736f0 | psc-c       | a31b6acf6900c04978302b5a1dcc017e9b35e1fc | psc-c       | 1.0.35  | NULL
(8 rows)

Taking one of the cabooses missing from switch 0:

{
  "git_commit": "71e20463580a8e17b8b1eb8172cc3cdbece479a6",
  "board": "oxide-rot-1",
  "name": "oxide-rot-1",
  "version": "1.0.34",
  "sign": "1432cc4cfe5688c51b55546fe37837c753cfbc89e8c3c6aabcf977fdf0c41e27",
  "epoch": null
}

and assuming Nexus collected it faithfully, I tried to follow along where it should have been inserted into sw_caboose. It would get converted to a SwCaboose here:

omicron/nexus/db-queries/src/db/datastore/inventory.rs

Lines 112 to 116 in 7f0ac3d

let cabooses = collection
.cabooses
.iter()
.map(|s| SwCaboose::from((**s).clone()))
.collect::<Vec<_>>();

This will assign it a random id, but otherwise just copies over all the fields.

All the SwCabooses are then inserted in a single insert:

omicron/nexus/db-queries/src/db/datastore/inventory.rs

Lines 302 to 313 in 7f0ac3d

// Insert records (and generate ids) for each distinct caboose that
// we've found. Like baseboards, these might already be present and
// rows in this table are not scoped to a particular collection
// because they only map (immutable) identifiers to UUIDs.
{
use nexus_db_schema::schema::sw_caboose::dsl;
let _ = diesel::insert_into(dsl::sw_caboose)
.values(cabooses)
.on_conflict_do_nothing()
.execute_async(&conn)
.await?;
}

This has an on_conflict_do_nothing to avoid duplicating rows, but above we said each SwCaboose gets a random id during that conversion, so how does the deduplication work? I believe that's coming from this unique index on the table, which looks at the non-id columns:

omicron/schema/crdb/dbinit.sql

Lines 3352 to 3353 in 7f0ac3d

CREATE UNIQUE INDEX IF NOT EXISTS caboose_properties
on omicron.public.sw_caboose (board, git_commit, name, version);

I think this explains why the rows are missing. This PR adds a sign column to the table, but does not change the unique index. The switch 0 RoT caboose is identical to the gimlet RoT caboose in all fields except sign, so gets rejected as a duplicate by that index. Presumably the gimlet cabooses happen to be inserted first, so the switch and PSC cabooses that only differ in sign never make it in.

@karencfv
Copy link
Contributor Author

I think this explains why the rows are missing. This PR adds a sign column to the table, but does not change the unique index. The switch 0 RoT caboose is identical to the gimlet RoT caboose in all fields except sign, so gets rejected as a duplicate by that index. Presumably the gimlet cabooses happen to be inserted first, so the switch and PSC cabooses that only differ in sign never make it in.

Aha! yeah, that makes sense. I'll make the change and test it out on a raclette. Thank you @jgallagher !

@karencfv
Copy link
Contributor Author

karencfv commented Apr 30, 2025 •
edited
Loading

Success!

root@oxz_switch1:~# omdb db inventory collections show latest sp
note: database URL not specified.  Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd00:1122:3344:102::3]:32221,[fd00:1122:3344:104::3]:32221,[fd00:1122:3344:103::3]:32221,[fd00:1122:3344:101::3]:32221,[fd00:1122:3344:104::4]:32221/omicron?sslmode=disable
note: database schema version matches expected (139.0.0)
collection: 044a9a3b-c448-475a-bd61-cb6b9898d4a0
collector:  85872a2d-481b-4f4d-a34e-8eecbffdcce0 (likely a Nexus instance)
started:    2025-04-30T01:11:54.495Z
done:       2025-04-30T01:12:06.675Z
errors:     0

Sled BRM42220036
    part number: 913-0000019
    power:    A0
    revision: 6
    MGS slot: Sled 14 (cubby 14)
    found at: 2025-04-30 01:11:57.772083 UTC from http://[fd00:1122:3344:102::2]:12225
    cabooses:
        SLOT       BOARD       NAME        VERSION GIT_COMMIT                               SIGN                                                             
        SpSlot0    gimlet-c    gimlet-c    1.0.36  a31b6acf6900c04978302b5a1dcc017e9b35e1fc n/a                                                              
        SpSlot1    gimlet-c    gimlet-c    1.0.36  a31b6acf6900c04978302b5a1dcc017e9b35e1fc n/a                                                              
        RotSlotA   oxide-rot-1 oxide-rot-1 1.0.34  71e20463580a8e17b8b1eb8172cc3cdbece479a6 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        RotSlotB   oxide-rot-1 oxide-rot-1 1.0.34  71e20463580a8e17b8b1eb8172cc3cdbece479a6 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        Stage0     oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        Stage0Next oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
    RoT pages:
        SLOT         DATA_BASE64                         
        Cmpa         oAAAeAAAAAAAAAAAAAAAAP8CAP3/AgD9... 
        CfpaActive   AAAAAGwAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaInactive AAAAAGsAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaScratch  AAAAAGwAAAAAAAAAAAAAAAAAAAAAAAAA... 
    RoT: active slot: slot A
    RoT: persistent boot preference: slot A
    RoT: pending persistent boot preference: -
    RoT: transient boot preference: -
    RoT: slot A SHA3-256: 9ba01fc19057d6fa0618a5918b6b11d41cf724aba41f27dd48ec06f89aea7fed
    RoT: slot B SHA3-256: 16b846eb11c9307d208624bddff21b3dc0797fe7b364feaac202eb4f92996e50

Sled BRM42220062
    part number: 913-0000019
    power:    A0
    revision: 6
    MGS slot: Sled 15 (cubby 15)
    found at: 2025-04-30 01:12:00.040015 UTC from http://[fd00:1122:3344:102::2]:12225
    cabooses:
        SLOT       BOARD       NAME        VERSION GIT_COMMIT                               SIGN                                                             
        SpSlot0    gimlet-c    gimlet-c    1.0.36  a31b6acf6900c04978302b5a1dcc017e9b35e1fc n/a                                                              
        SpSlot1    gimlet-c    gimlet-c    1.0.36  a31b6acf6900c04978302b5a1dcc017e9b35e1fc n/a                                                              
        RotSlotA   oxide-rot-1 oxide-rot-1 1.0.34  71e20463580a8e17b8b1eb8172cc3cdbece479a6 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        RotSlotB   oxide-rot-1 oxide-rot-1 1.0.34  71e20463580a8e17b8b1eb8172cc3cdbece479a6 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        Stage0     oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        Stage0Next oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
    RoT pages:
        SLOT         DATA_BASE64                         
        Cmpa         oAAAeAAAAAAAAAAAAAAAAP8CAP3/AgD9... 
        CfpaActive   AAAAAGwAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaInactive AAAAAGsAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaScratch  AAAAAGwAAAAAAAAAAAAAAAAAAAAAAAAA... 
    RoT: active slot: slot B
    RoT: persistent boot preference: slot B
    RoT: pending persistent boot preference: -
    RoT: transient boot preference: -
    RoT: slot A SHA3-256: 9ba01fc19057d6fa0618a5918b6b11d41cf724aba41f27dd48ec06f89aea7fed
    RoT: slot B SHA3-256: 16b846eb11c9307d208624bddff21b3dc0797fe7b364feaac202eb4f92996e50

Sled BRM42220030
    part number: 913-0000019
    power:    A0
    revision: 6
    MGS slot: Sled 16 (cubby 16)
    found at: 2025-04-30 01:11:54.516447 UTC from http://[fd00:1122:3344:102::2]:12225
    cabooses:
        SLOT       BOARD       NAME        VERSION GIT_COMMIT                               SIGN                                                             
        SpSlot0    gimlet-c    gimlet-c    1.0.36  a31b6acf6900c04978302b5a1dcc017e9b35e1fc n/a                                                              
        SpSlot1    gimlet-c    gimlet-c    1.0.36  a31b6acf6900c04978302b5a1dcc017e9b35e1fc n/a                                                              
        RotSlotA   oxide-rot-1 oxide-rot-1 1.0.34  71e20463580a8e17b8b1eb8172cc3cdbece479a6 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        RotSlotB   oxide-rot-1 oxide-rot-1 1.0.34  71e20463580a8e17b8b1eb8172cc3cdbece479a6 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        Stage0     oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        Stage0Next oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
    RoT pages:
        SLOT         DATA_BASE64                         
        Cmpa         oAAAeAAAAAAAAAAAAAAAAP8CAP3/AgD9... 
        CfpaActive   AAAAAFAAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaInactive AAAAAE8AAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaScratch  AAAAAFAAAAAAAAAAAAAAAAAAAAAAAAAA... 
    RoT: active slot: slot A
    RoT: persistent boot preference: slot A
    RoT: pending persistent boot preference: -
    RoT: transient boot preference: -
    RoT: slot A SHA3-256: 9ba01fc19057d6fa0618a5918b6b11d41cf724aba41f27dd48ec06f89aea7fed
    RoT: slot B SHA3-256: 16b846eb11c9307d208624bddff21b3dc0797fe7b364feaac202eb4f92996e50

Sled BRM44220007
    part number: 913-0000019
    power:    A0
    revision: 6
    MGS slot: Sled 17 (cubby 17)
    found at: 2025-04-30 01:11:58.911771 UTC from http://[fd00:1122:3344:102::2]:12225
    cabooses:
        SLOT       BOARD       NAME        VERSION GIT_COMMIT                               SIGN                                                             
        SpSlot0    gimlet-c    gimlet-c    1.0.36  a31b6acf6900c04978302b5a1dcc017e9b35e1fc n/a                                                              
        SpSlot1    gimlet-c    gimlet-c    1.0.36  a31b6acf6900c04978302b5a1dcc017e9b35e1fc n/a                                                              
        RotSlotA   oxide-rot-1 oxide-rot-1 1.0.34  71e20463580a8e17b8b1eb8172cc3cdbece479a6 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        RotSlotB   oxide-rot-1 oxide-rot-1 1.0.34  71e20463580a8e17b8b1eb8172cc3cdbece479a6 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        Stage0     oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
        Stage0Next oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 11594bb5548a757e918e6fe056e2ad9e084297c9555417a025d8788eacf55daf 
    RoT pages:
        SLOT         DATA_BASE64                         
        Cmpa         oAAAeAAAAAAAAAAAAAAAAP8CAP3/AgD9... 
        CfpaActive   AAAAAGwAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaInactive AAAAAGsAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaScratch  AAAAAGwAAAAAAAAAAAAAAAAAAAAAAAAA... 
    RoT: active slot: slot A
    RoT: persistent boot preference: slot A
    RoT: pending persistent boot preference: -
    RoT: transient boot preference: -
    RoT: slot A SHA3-256: 9ba01fc19057d6fa0618a5918b6b11d41cf724aba41f27dd48ec06f89aea7fed
    RoT: slot B SHA3-256: 16b846eb11c9307d208624bddff21b3dc0797fe7b364feaac202eb4f92996e50

Power BRM11230017
    part number: 913-0000003
    power:    A2
    revision: 9
    MGS slot: Power 0
    found at: 2025-04-30 01:12:01.546430 UTC from http://[fd00:1122:3344:102::2]:12225
    cabooses:
        SLOT       BOARD       NAME        VERSION GIT_COMMIT                               SIGN                                                             
        SpSlot0    psc-c       psc-c       1.0.35  a31b6acf6900c04978302b5a1dcc017e9b35e1fc n/a                                                              
        SpSlot1    psc-c       psc-c       1.0.33  441b09270d8715275d6b3c3b57d84dba539e34d0 n/a                                                              
        RotSlotA   oxide-rot-1 oxide-rot-1 1.0.34  71e20463580a8e17b8b1eb8172cc3cdbece479a6 f592d8f109b81881221eed5af6438abad9b5df8c220b9129c03763e7e10b22c7 
        RotSlotB   oxide-rot-1 oxide-rot-1 1.0.34  71e20463580a8e17b8b1eb8172cc3cdbece479a6 f592d8f109b81881221eed5af6438abad9b5df8c220b9129c03763e7e10b22c7 
        Stage0     oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 f592d8f109b81881221eed5af6438abad9b5df8c220b9129c03763e7e10b22c7 
        Stage0Next oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 f592d8f109b81881221eed5af6438abad9b5df8c220b9129c03763e7e10b22c7 
    RoT pages:
        SLOT         DATA_BASE64                         
        Cmpa         oAAAiAAAAAAAAAAAAAAAAP8CAP3/AgD9... 
        CfpaActive   AAAAABMAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaInactive AAAAABMAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaScratch  AAAAABMAAAAAAAAAAAAAAAAAAAAAAAAA... 
    RoT: active slot: slot B
    RoT: persistent boot preference: slot B
    RoT: pending persistent boot preference: -
    RoT: transient boot preference: -
    RoT: slot A SHA3-256: 17cf2709c1d55dcc839898e3073d9db0e37f540201f0cdb5bdae1ae0932327ac
    RoT: slot B SHA3-256: 6c2a3bf9ef18a025497af80ad468c73a1fa985c1c4029ed5b6a55de09ec364d8

Switch BRM31230004
    part number: 913-0000006
    power:    A0
    revision: 10
    MGS slot: Switch 0
    found at: 2025-04-30 01:12:02.678147 UTC from http://[fd00:1122:3344:102::2]:12225
    cabooses:
        SLOT       BOARD       NAME        VERSION GIT_COMMIT                               SIGN                                                             
        SpSlot0    sidecar-c   sidecar-c   1.0.37  a31b6acf6900c04978302b5a1dcc017e9b35e1fc n/a                                                              
        SpSlot1    sidecar-c   sidecar-c   1.0.37  a31b6acf6900c04978302b5a1dcc017e9b35e1fc n/a                                                              
        RotSlotA   oxide-rot-1 oxide-rot-1 1.0.34  71e20463580a8e17b8b1eb8172cc3cdbece479a6 1432cc4cfe5688c51b55546fe37837c753cfbc89e8c3c6aabcf977fdf0c41e27 
        RotSlotB   oxide-rot-1 oxide-rot-1 1.0.34  71e20463580a8e17b8b1eb8172cc3cdbece479a6 1432cc4cfe5688c51b55546fe37837c753cfbc89e8c3c6aabcf977fdf0c41e27 
        Stage0     oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 1432cc4cfe5688c51b55546fe37837c753cfbc89e8c3c6aabcf977fdf0c41e27 
        Stage0Next oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 1432cc4cfe5688c51b55546fe37837c753cfbc89e8c3c6aabcf977fdf0c41e27 
    RoT pages:
        SLOT         DATA_BASE64                         
        Cmpa         oAAAiAAAAAAAAAAAAAAAAP8CAP3/AgD9... 
        CfpaActive   AAAAAAgAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaInactive AAAAAAcAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaScratch  AAAAAAgAAAAAAAAAAAAAAAAAAAAAAAAA... 
    RoT: active slot: slot A
    RoT: persistent boot preference: slot A
    RoT: pending persistent boot preference: -
    RoT: transient boot preference: -
    RoT: slot A SHA3-256: ccd280614a9cce6919bc935e797ec8ff3a060745a2cab5a0fa7a69e66a3f336b
    RoT: slot B SHA3-256: c63635d08181e02ecd56eb3a29c42382b36ac327b68da2cce13eda255cde09fc

Switch BRM44220013
    part number: 913-0000006
    power:    A0
    revision: 4
    MGS slot: Switch 1
    found at: 2025-04-30 01:12:04.072109 UTC from http://[fd00:1122:3344:101::2]:12225
    cabooses:
        SLOT       BOARD       NAME        VERSION GIT_COMMIT                               SIGN                                                             
        SpSlot0    sidecar-b   sidecar-b   1.0.37  a31b6acf6900c04978302b5a1dcc017e9b35e1fc n/a                                                              
        SpSlot1    sidecar-b   sidecar-b   1.0.35  441b09270d8715275d6b3c3b57d84dba539e34d0 n/a                                                              
        RotSlotA   oxide-rot-1 oxide-rot-1 1.0.34  71e20463580a8e17b8b1eb8172cc3cdbece479a6 1432cc4cfe5688c51b55546fe37837c753cfbc89e8c3c6aabcf977fdf0c41e27 
        RotSlotB   oxide-rot-1 oxide-rot-1 1.0.34  71e20463580a8e17b8b1eb8172cc3cdbece479a6 1432cc4cfe5688c51b55546fe37837c753cfbc89e8c3c6aabcf977fdf0c41e27 
        Stage0     oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 1432cc4cfe5688c51b55546fe37837c753cfbc89e8c3c6aabcf977fdf0c41e27 
        Stage0Next oxide-rot-1 oxide-rot-1 1.3.3   95e79111096d7ec258d68a2c6840b6062efc7f54 1432cc4cfe5688c51b55546fe37837c753cfbc89e8c3c6aabcf977fdf0c41e27 
    RoT pages:
        SLOT         DATA_BASE64                         
        Cmpa         oAAAeAAAAAAAAAAAAAAAAP8CAP3/AgD9... 
        CfpaActive   AAAAACsAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaInactive AAAAACoAAAAAAAAAAAAAAAAAAAAAAAAA... 
        CfpaScratch  AAAAACsAAAAAAAAAAAAAAAAAAAAAAAAA... 
    RoT: active slot: slot B
    RoT: persistent boot preference: slot B
    RoT: pending persistent boot preference: -
    RoT: transient boot preference: -
    RoT: slot A SHA3-256: ccd280614a9cce6919bc935e797ec8ff3a060745a2cab5a0fa7a69e66a3f336b
    RoT: slot B SHA3-256: c63635d08181e02ecd56eb3a29c42382b36ac327b68da2cce13eda255cde09fc

Thanks @davepacheco @jgallagher! I'll rebase with main, and this should be ready to go then

@karencfv
Copy link
Contributor Author

It looks like a bunch of the tests are failing now :( My changes between d408a44 (the last time the tests passed) and the last commit are so small. I don't see how they could have broken anything? Additionally, the latest commit works as expected on London.

@davepacheco is there a chance #8055 has anything to do with this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@davepacheco davepacheco davepacheco left review comments

@jgallagher jgallagher jgallagher left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

record caboose SIGN value in inventory
3 participants
@karencfv @davepacheco @jgallagher