DO NOT MERGE: Smoke-test of a costly “always compute old-style IDs” approach

[mtrmac]

This is a variant of #24287 which reverts (some of) the system test changes to again expect old-style IDs, and includes containers/storage#2155 / containers/image#2613 .

⚠️ WARNING: all of this is a quick sketch, absolutely untested.

This exists purely to give us options. With this, zstd:chunked loses some of the performance wins, and it should revert to old-style IDs. We then have the option to reintroduce the ID change and performance improvement (perhaps even based on measurements) later.

And, separately, this enforces that layers should match against the config’s DiffIDs, resolving the signing ambiguity.

FYI @giuseppe @edsantiago @mheon @baude

Does this PR introduce a user-facing change?

None

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mtrmac

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [mtrmac]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[packit-as-a-service]

Ephemeral COPR build failed. @containers/packit-build please check.

[packit-as-a-service]

Ephemeral COPR build failed. @containers/packit-build please check.

[mtrmac]

Note to self: If/after containers/storage#2156 merges, this PR will need to re-enable enable_partial_images.

Edit: done.

[mtrmac]

The test images with all images converted to zstd:chunked are too old. For now I’m leaving that variant in https://github.com/mtrmac/libpod/tree/mitr-zstd-costly-old-ids-integration-chunked-test-images , and reverting to test with the current images here.

⚠️ That significantly decreases the scope of testing; many of the unexpected behavior changes were happening only with most images converted to zstd:chunked. With this, we are mostly back to the test coverage of #21903 , which is known to not demonstrate many of the surprises.

[edsantiago]

The test images with all images converted to zstd:chunked are too old

Would you care to follow (and test and review) my new simplified automation_images doc? You will need to import https://github.com/containers/automation_images/pull/388/files but apart from that it really should be fairly simple.

[mtrmac]

Thanks, I’ll definitely try that.

[mtrmac]

The test images with all images converted to zstd:chunked are too old.

Note to self: containers/automation_images#395 .

[mtrmac]

Private notes of manual testing process/outcomes, some problems remain

chunked pulls with DiffIDs work
###
# bin/podman pull docker://quay.io/libpod/alpine
# bin/podman --log-level=debug push --tls-verify=false --force-compression --compression-format zstd:chunked quay.io/libpod/alpine   docker://localhost:5000/chunked-normal

# skopeo copy --preserve-digests --all --src-tls-verify=false docker://localhost:5000/chunked-normal dir:t-chunked-normal
FIXME: originalDigest, originalLen
# bin/podman system reset
# bin/podman --log-level=debug pull --tls-verify=false docker://localhost:5000/chunked-normal
verify log contains "Created zstd:chunked differ for blob"
verify image ID matches config digest

chunked pulls with incorrect DiffIDs fail
###
# jq . < t-chunked-normal/$originalDigest > tmp
# cp -a t-chunked-normal t-chunked-mismatch
modify diffID in tmp
# configDigest=$(sha256sum tmp | cut -d ' ' -f 1)
# configLen=$(wc --bytes < tmp)
# mv tmp t-chunked-mismatch/$configDigest
# sed "s/$originalDigest/$configDigest/ ; s/:$originalLen}/:$configLen}/" < t-chunked-normal/manifest.json > t-chunked-mismatch/manifest.json 
# skopeo copy --preserve-digests --all --dest-tls-verify=false dir:t-chunked-mismatch docker://localhost:5000/chunked-mismatch
# bin/podman --log-level=debug pull --tls-verify=false docker://localhost:5000/chunked-mismatch
(reuse) verify fails with "trying to reuse"…"does not match config's DiffID"
# bin/podman system reset
# bin/podman --log-level=debug pull --tls-verify=false docker://localhost:5000/chunked-mismatch
(fresh pull) verify fails with "uncompressed digest of layer" … "config claims"

chunked pulls with missing DiffIDs fail
###
# jq . < t-chunked-normal/$originalDigest > tmp
# cp -a t-chunked-normal t-chunked-missing
remove diffID in tmp
# configDigest=$(sha256sum tmp | cut -d ' ' -f 1)
# configLen=$(wc --bytes < tmp)
# mv tmp t-chunked-missing/$configDigest
# sed "s/$originalDigest/$configDigest/ ; s/:$originalLen}/:$configLen}/" < t-chunked-normal/manifest.json > t-chunked-missing/manifest.json 
# skopeo copy --preserve-digests --all --dest-tls-verify=false dir:t-chunked-missing docker://localhost:5000/chunked-missing
# bin/podman --log-level=debug pull --tls-verify=false docker://localhost:5000/chunked-missing
(fresh pull) verify fails with "image config has only 0 DiffID values, but a layer with index 0 exists"
# bin/podman --log-level=debug pull --tls-verify=false docker://localhost:5000/chunked-normal
# bin/podman --log-level=debug pull --tls-verify=false docker://localhost:5000/chunked-missing
(reuse) verify fails with "image config has only 0 DiffID values, but a layer with index 0 exists"

chunked pulls with empty DiffIDs fail
###
# jq . < t-chunked-normal/$originalDigest > tmp
# cp -a t-chunked-normal t-chunked-empty
set diffID in tmp to ""
# configDigest=$(sha256sum tmp | cut -d ' ' -f 1)
# configLen=$(wc --bytes < tmp)
# mv tmp t-chunked-empty/$configDigest
# sed "s/$originalDigest/$configDigest/ ; s/:$originalLen}/:$configLen}/" < t-chunked-normal/manifest.json > t-chunked-empty/manifest.json 
# skopeo copy --preserve-digests --all --dest-tls-verify=false dir:t-chunked-empty docker://localhost:5000/chunked-empty
# bin/podman --log-level=debug pull --tls-verify=false docker://localhost:5000/chunked-empty
FIXME FIXME FIXME (reuse) verify succeeds with "Copying blob 45f92e705e65 skipped: already exists (found by TOC)"
# bin/podman system reset
# bin/podman --log-level=debug pull --tls-verify=false docker://localhost:5000/chunked-empty
(fresh pull) verify succeeds with "Failed to retrieve partial blob: DiffID value for layer 0 is unknown or explicitly empty" and "Detected compression format zstd" (falling back to full pull)

non-chunked preparation
###
# for i in t-chunked-*; do j=t-nonchunked-${i#t-chunked-}; cp -a "$i" "$j" && sed 's/io.github.containers.zstd-chunked/irrelevant-annotation/g' < "$j"/manifest.json > tmp &&  mv tmp "$j"/manifest.json  && skopeo copy --preserve-digests --all --dest-tls-verify=false dir:$j docker://localhost:5000/nonchunked-${j#t-nonchunked-}; done

non-chunked pulls with DiffIDs work
###
# bin/podman system reset
# bin/podman --log-level=debug pull --tls-verify=false docker://localhost:5000/nonchunked-normal
(fresh pull) verify succeeds with "Failed to retrieve partial blob: no TOC found and convert_images is not configured"
# bin/podman system reset
# bin/podman --log-level=debug pull --tls-verify=false docker://localhost:5000/chunked-normal
# bin/podman --log-level=debug pull --tls-verify=false docker://localhost:5000/nonchunked-normal
(reuse did not trigger, no compressed digest match) verify succeeds with "Failed to retrieve partial blob: no TOC found and convert_images is not configured" and "Detected compression format zstd"

non-chunked pulls with incorrect DiffIDs fail
###
# bin/podman system reset
# bin/podman --log-level=debug pull --tls-verify=false docker://localhost:5000/nonchunked-mismatch
(fresh pull) verify fails with "writing blob"… "does not match config's DiffID"
# bin/podman --log-level=debug pull --tls-verify=false docker://localhost:5000/chunked-normal
# bin/podman --log-level=debug pull --tls-verify=false docker://localhost:5000/nonchunked-mismatch
(reuse) verify fail with "trying to reuse blob"…"does not match config's DiffID"


non-chunked pulls with missing DiffIDs work
###
# bin/podman system reset
# bin/podman --log-level=debug pull --tls-verify=false docker://localhost:5000/nonchunked-missing
FIXME FIXME FIXME (fresh pull) fails with "image config has only 0 DiffID values, but a layer with index 0 exists"
# bin/podman --log-level=debug pull --tls-verify=false docker://localhost:5000/chunked-normal
# bin/podman --log-level=debug pull --tls-verify=false docker://localhost:5000/nonchunked-missing
FIXME FIXME FIXME (reuse) fails with "image config has only 0 DiffID values, but a layer with index 0 exists"

non-chunked pulls with empty DiffIDs work
###
# bin/podman system reset
# bin/podman --log-level=debug pull --tls-verify=false docker://localhost:5000/nonchunked-empty
(fresh pull) verify succeeds with "Failed to retrieve partial blob: no TOC found and convert_images is not configured"
# bin/podman system reset
# bin/podman --log-level=debug pull --tls-verify=false docker://localhost:5000/chunked-normal
(reuse did not trigger, no compressed digest match) verify succeeds with "Failed to retrieve partial blob: no TOC found and convert_images is not configured" and "Detected compression format zstd"


estargz pulls fall back
###
FIXME: this is a random image on the internet
# bin/podman --log-level=debug pull docker://ghcr.io/stargz-containers/node:13.13.0-esgz
Verify suceeds with "Failed to retrieve partial blob: estargz layers don't support partial pulls with guaranteed consistency with non-partial pulls"


edit storage.conf
###
# sed 's/pull_options = .*/pull_options = {enable_partial_images = "true", use_hard_links = "false", ostree_repos="", "insecure_allow_unpredictable_image_contents" = "true"}/' < /usr/share/containers/storage.conf > /etc/containers/storage.conf

with option, estargz pulls work
###
FIXME: this is a random image on the internet
# bin/podman system reset
# bin/podman --log-level=debug pull docker://ghcr.io/stargz-containers/node:13.13.0-esgz
Verify suceeds with "Created eStargz differ for blob"

with option, chunked pulls with incorrect DiffIDs work
###
# bin/podman system reset
# bin/podman --log-level=debug pull --tls-verify=false docker://localhost:5000/chunked-mismatch
(fresh pull) Verify succeeds with "Created zstd:chunked differ for blob"
Verify image ID does not match config digest
# bin/podman --log-level=debug pull --tls-verify=false docker://localhost:5000/chunked-normal
# bin/podman --log-level=debug pull --tls-verify=false docker://localhost:5000/chunked-mismatch
(reuse still suceeds, TOC match without diffID known)
# bin/podman --log-level=debug pull --tls-verify=false docker://localhost:5000/nonchunked-normal
# bin/podman --log-level=debug pull --tls-verify=false docker://localhost:5000/chunked-mismatch
(reuse) verify fails on "does not match config's DiffID", now known costlessly


with option, chunked pulls with missing DiffIDs work
###
# bin/podman system reset
# bin/podman --log-level=debug pull --tls-verify=false docker://localhost:5000/chunked-missing
FIXME FIXME FIXME (fresh pull) fails with "image config has only 0 DiffID values, but a layer with index 0 exists"
# bin/podman --log-level=debug pull --tls-verify=false docker://localhost:5000/chunked-normal
# bin/podman --log-level=debug pull --tls-verify=false docker://localhost:5000/chunked-missing
FIXME FIXME FIXME (reuse) succeeds, but re-check after bug fixed
# bin/podman --log-level=debug pull --tls-verify=false docker://localhost:5000/nonchunked-normal
# bin/podman --log-level=debug pull --tls-verify=false docker://localhost:5000/chunked-missing
FIXME FIXME FIXME (fresh pull) fails with "image config has only 0 DiffID values, but a layer with index 0 exists"



with option, chunked pulls with empty DiffIDs work
###
# bin/podman system reset
# bin/podman --log-level=debug pull --tls-verify=false docker://localhost:5000/chunked-empty
(fresh pull) Verify succeeds with "Failed to retrieve partial blob: DiffID value for layer 0 is unknown or explicitly empty"
# bin/podman --log-level=debug pull --tls-verify=false docker://localhost:5000/chunked-normal
# bin/podman --log-level=debug pull --tls-verify=false docker://localhost:5000/chunked-empty
(reuse, FIXME FIXME FIXME re-check later) Verify suceeds with "Skipping blob"…"already present"
# bin/podman --log-level=debug pull --tls-verify=false docker://localhost:5000/nonchunked-normal
# bin/podman --log-level=debug pull --tls-verify=false docker://localhost:5000/chunked-empty
(reuse, FIXME FIXME FIXME re-check later) Verify suceeds with "Skipping blob"…"already present"


---

- [ ] schema1 pulls work
- [v] non-chunked pulls with DiffIDs work
- [v] non-chunked pulls with incorrect DiffIDs fail
- [X] non-chunked pulls with missing DiffIDs work
- [v] non-chunked pulls with empty DiffIDs work
- [v] chunked pulls with DiffIDs work
- [v] chunked pulls with incorrect DiffIDs fail
- [v] chunked pulls with missing DiffIDs fail
- [X] chunked pulls with empty DiffIDs fail
- [X] estargz pulls fall back
- [v] with option, estargz pulls work
- [v] with option, chunked pulls with incorrect DiffIDs work
- [X] with option, chunked pulls with missing DiffIDs work
- [v] with option, chunked pulls with empty DiffIDs work

[mtrmac]

This is now on top of #25007, and differs from it by:

• using a CI image where all images are zstd:chunked
• enabling partial pulls for the whole Podman test suite
• updating the tests to work with the unexpectedly-Zstd images

[mtrmac]

We now (e.g. with #25007 ) always run at least some tests with Zstd; converting every image is a possibility but losing gzip coverage would not be obviously better.