Don't hardcode usr/share/rpm

[cgwalters]

We don't need this on modern Fedora anymore for example. If we detect the sysimage path, then just use it.

[jmarrero]

lgtm

[cgwalters]

Ah right fun fun, we still set _dbpath /usr/share/rpm in rpm-ostree as canonical...

changes needed on rpm-ostree

[cgwalters]

OK right so the problem here was actually that rpm-ostree only writes the macro for the rpmdb in postprocessing.

[cgwalters]

OK this one is actually tested more

[HuijingHei]

LGTM

[solacelost]

This change recently made it into 0.2.26, and I believe it's causing an issue in bootc composes on Fedora 41 due to an interaction with rpm-ostree in some way.

When composing an image with rpm-ostree that includes bootupd==0.2.26 and running, in a postprocess step, bootupctl backend generate-update-metadata, the build errors. I've got some very long logs of the failure here so you can look through everything, but the lines I believe to be relevant follow:

Executing `postprocess` inline script '1'
+ /usr/bin/bootupctl backend generate-update-metadata
total 276
drwxr-xr-x. 2 root root    123 Feb  9 22:39 .
drwxr-xr-x. 4 root root     32 Feb  9 22:37 ..
-rw-r--r--. 1 root root      0 Feb  9 22:39 .rpm.lock
-rw-r--r--. 1 root root      0 Feb  9 22:39 .rpmdbdirsymlink_created
-rw-r--r--. 1 root root 249856 Feb  9 22:39 rpmdb.sqlite
-rw-r--r--. 1 root root  32768 Feb  9 22:39 rpmdb.sqlite-shm
-rw-r--r--. 1 root root      0 Feb  9 22:39 rpmdb.sqlite-wal
[src/ostreeutil.rs:61:9] "found" = "found"
[src/ostreeutil.rs:61:9] &arg = "--dbpath=/usr/lib/sysimage/rpm"
error: generating metadata failed: Failed to invoke rpm -qf
error: Installing packages: bwrap(/usr/bin/rpmostree-postprocess-inline-1): Child process killed by signal 1
error: compose tree failed: ExitStatus(unix_wait_status(256))

My build is quite a bit larger than necessary as a reproducer, so I'd say that using the Fedora bootc image base would be simplest:
https://gitlab.com/fedora/bootc/base-images/-/blob/7b4e013126bfe5851b2f882b1d9af0a764c6e7dc/Containerfile

Just modify rawhide to be 41 in these tags, as rawhide is having other issues related to rpm-ostree and the impacting change in 41 is bootupd. The image will still not finish properly, even without this issue, as there is yet another issue affecting the use of bind mounts in buildah on RUN --mount lines, but the layer with the compose is failing outright at the moment in the way listed above.

Personally, I'm working around this right now with this commit:
https://git.jharmison.com/james/bootc-image/commit/13ee7ce2bf5c10b81def34c7de23171b59a82333

[cgwalters]

Hm interestingly I can reproduce this if I add an explicit

rpm -q --dbpath /usr/lib/sysimage/rpm -f /usr/sbin/grub2-install || true
/usr/bin/bootupctl backend generate-update-metadata

Because doing that seems to create an empty rpm database in /usr/lib/sysimage/rpm. Could you have something else in your base image that's running rpm in a postprocess script?

To the best of my knowledge the main fedora-bootc base image builds aren't affected by this.

[solacelost]

I just ran this:

$ git clone https://gitlab.com/fedora/bootc/base-images.git fedora-bootc-base-images
Cloning into 'fedora-bootc-base-images'...
remote: Enumerating objects: 3166, done.
remote: Counting objects: 100% (565/565), done.
remote: Compressing objects: 100% (255/255), done.
remote: Total 3166 (delta 308), reused 548 (delta 298), pack-reused 2601 (from 1)
Receiving objects: 100% (3166/3166), 607.25 KiB | 10.65 MiB/s, done.
Resolving deltas: 100% (2151/2151), done.

$ cd fedora-bootc-base-images/

$ sed -i 's/:rawhide/:41/g' Containerfile

$ buildah bud --security-opt=label=disable --cap-add=all --build-arg MANIFEST=fedora-tier-0.yaml --device /dev/fuse .
[1/3] STEP 1/1: FROM quay.io/fedora/fedora:41 AS repos
--> 970f60778bc2
[2/3] STEP 1/10: FROM quay.io/fedora/fedora:41 AS builder
[2/3] STEP 2/10: RUN dnf -y install rpm-ostree selinux-policy-targeted
Updating and loading repositories:
 Fedora 41 openh264 (From Cisco) - x86_ 100% |  14.3 KiB/s |   4.8 KiB |  00m00s
 [...snip...]
 Executing `postprocess` inline script '4'
+ /usr/bin/bootupctl backend generate-update-metadata
total 18656
drwxr-xr-x.  2 root root       91 Feb 11 18:52 .
drwxr-xr-x. 70 root root     4096 Feb 11 18:53 ..
-rw-r--r--.  1 root root        0 Feb 11 18:52 .rpm.lock
-rw-r--r--.  1 root root 19001344 Feb 11 18:52 rpmdb.sqlite
-rw-r--r--.  1 root root    98304 Feb 11 18:52 rpmdb.sqlite-shm
-rw-r--r--.  1 root root        0 Feb 11 18:52 rpmdb.sqlite-wal
[src/ostreeutil.rs:61:9] "found" = "found"
[src/ostreeutil.rs:61:9] &arg = "--dbpath=/usr/share/rpm"
Generated update layout for BIOS: grub2-tools-1:2.12-15.fc41.x86_64
total 18592
drwxr-xr-x.  2 root root       91 Feb 11 18:52 .
drwxr-xr-x. 70 root root     4096 Feb 11 18:53 ..
-rw-r--r--.  1 root root        0 Feb 11 18:52 .rpm.lock
-rw-r--r--.  1 root root 19001344 Feb 11 18:52 rpmdb.sqlite
-rw-r--r--.  1 root root    32768 Feb 11 18:53 rpmdb.sqlite-shm
-rw-r--r--.  1 root root        0 Feb 11 18:52 rpmdb.sqlite-wal
[src/ostreeutil.rs:61:9] "found" = "found"
[src/ostreeutil.rs:61:9] &arg = "--dbpath=/usr/share/rpm"
Generated update layout for EFI: grub2-efi-x64-1:2.12-15.fc41.x86_64,shim-x64-15.8-3.x86_64
Executing `postprocess` inline script '5'
[...snip...]

So indeed something here is working differently than my build. I don't have any postprocess scripts that would be doing that, but it may be that a post from an RPM is interfering in some way?

I don't think I'm able to manipulate /usr/lib/sysimage/rpm at this stage in the postprocess script, am I?

[cgwalters]

I would add a
ls -al /usr/lib/sysimage/rpm right before the invocation of /usr/bin/bootupctl backend generate-update-metadata - if indeed that appears to have an "empty" rpmdb (which you could tell as it will be small) then:

but it may be that a post from an RPM is interfering in some way?

Yes, quite possibly. It would likely take some debugging to figure out which one. Note that invoking rpm from inside an rpm %post (as opposed to our postprocess scripts) has always been undefined behavior.

I don't think I'm able to manipulate /usr/lib/sysimage/rpm at this stage in the postprocess script, am I?

Sure you can. If that directory is an empty rpmdb, you could likely work around this with just an rm -rf /usr/lib/sysimage/rpm/rpmdb* or so. rpm-ostree's postprocessing should sync it up with hardlinks.

I'll reiterate here that I hope you can drop your fork of the base image and check out what we're doing in https://gitlab.com/fedora/bootc/tracker/-/issues/32 because it's exactly these kind of things that it's designed more to avoid.

[solacelost]

+ ls -alF /usr/lib/sysimage/rpm
total 276
drwxr-xr-x. 2 root root    123 Feb 11 19:19 ./
drwxr-xr-x. 4 root root     32 Feb 11 19:17 ../
-rw-r--r--. 1 root root      0 Feb 11 19:19 .rpm.lock
-rw-r--r--. 1 root root      0 Feb 11 19:19 .rpmdbdirsymlink_created
-rw-r--r--. 1 root root 249856 Feb 11 19:19 rpmdb.sqlite
-rw-r--r--. 1 root root  32768 Feb 11 19:19 rpmdb.sqlite-shm
-rw-r--r--. 1 root root      0 Feb 11 19:19 rpmdb.sqlite-wal
+ ls -alF /usr/share/rpm
total 89696
drwxr-xr-x.   2 root root       91 Feb 11 19:20 ./
drwxr-xr-x. 213 root root     8192 Feb 11 19:21 ../
-rw-r--r--.   1 root root        0 Feb 11 19:20 .rpm.lock
-rw-r--r--.   1 root root 91738112 Feb 11 19:20 rpmdb.sqlite
-rw-r--r--.   1 root root    98304 Feb 11 19:20 rpmdb.sqlite-shm
-rw-r--r--.   1 root root        0 Feb 11 19:20 rpmdb.sqlite-wal

It's not quite small enough for "empty" (which seems to be around 212992 bytes). I can't imagine it's actually useful for anything, though, so I'm just going to remove it and see what happens!

Regarding bootc #32, the current recommended approach for rechunking is currently broken until buildah #5975 gets ironed out anyways, and I know there's a lot of work to do and a lack of bandwidth to get something like self-image-builder all the way ironed out. I'll keep watching it, for sure! I'd help but I think you're all much further in the weeds than I'm able to follow.