Remove conformance tests that use detached materials

[steiza]

Summary

For #122. All the clients (including cosign!) are moving away from detached materials in favor of bundles.

This will help unblock things like sigstore/sigstore-go#277.

Release Note

• Removed sign and verify from conformance test protocol; all tests use sign-bundle and verify-bundle.

Documentation

N/Aish; already updated docs/cli_protocol.md as part of this pull request.

[woodruffw]

Good riddance!

[loosebazooka]

Could it make sense to organize these into their own test file and let people explicitly include it? Could it help prevent a regression in cosign while we are transitioning it?

Although "good riddance" is a pretty strong argument

[woodruffw]

Could it make sense to organize these into their own test file and let people explicitly include it? Could it help prevent a regression in cosign while we are transitioning it?

I'm OK with this, if we want to keep them around: IMO having the disabled by default and then allowing integrators to do with-nonstandard-detached-material-tests: true would probably be sufficient 🙂

OTOH I think this is all 100% unstandardized, so having them in the conformance suite at all isn't fantastic, especially if it's dragging down efforts to get other things added. So if cosign et al. want these for regression testing, it might make sense to copy/vendor them directly into the test suite there.

[steiza]

Could it help prevent a regression in cosign while we are transitioning it?

Cosign's current conformance test integration uses sigstore-go for bundles, and for non-bundles it first assembles the detached materials into a bundle so it can use sigstore-go: https://github.com/sigstore/cosign/blob/e7667bd9d71d99aaa9ad3489241f5abe92fe0c9c/cmd/conformance/main.go#L114-L120

It's possible that after sigstore/cosign#3844 lands (which is currently a draft), we could update cosign's conformance test to exercise the detached materials case. On the other hand, the conformance detached materials support is blocking other work, like sigstore/sigstore-go#277.

[loosebazooka]

Alright, sgtm, let's go!

[haydentherapper]

Have we verified that all tests here are covered by the bundle tests?

[steiza]

Have we verified that all tests here are covered by the bundle tests?

There is a surprising amount of subtlety here.

I added a test back in that checks that the ctlog public key is correct. The other test on mismatched artifacts which should be covered by test_verify_rejects_mismatched_hashedrekord. The empty tests are checking for mismatched signatures (or certificates), which should be covered by test_verify_rejects_invalid_signature.

I don't think we're substantially losing coverage with this change, although the way some cases are tested are changing slightly.

[woodruffw]

Thanks a ton @steiza!