With a btrfs root, bind mounts put all data in the overlay instead of in the volume

[saolof]

Issue Description

Here's an example when mounting a 10 GB xfs lvm volume in ~/experiements/xfstest when I run ubuntu with a btrfs root folder.

Steps to reproduce the issue

Steps to reproduce the issue

1. Use btrfs as root (tested on ubuntu, could potentially try fedora as wel)
2. Create an xfs filesytem and mount it wherever.
3. Run a container with a bind mount into xfs:
   podman run --rm -v $HOME/experiments/xfstest:/var/lib/postgresql -e POSTGRES_PASSWORD=unsafe postgres

Describe the results you received

The postgres data folder ends up being mounted in btrfs instead of in the folder I created, and all data gets wiped when the container goes away

postgres@4d5e9cb22008:/$ df -hT
Filesystem              Type      Size  Used Avail Use% Mounted on
tmpfs                   tmpfs     6.3G   96K  6.3G   1% /etc/hosts
shm                     tmpfs      63M  1.1M   62M   2% /dev/shm
/dev/mapper/vg0-xfstest xfs        10G  229M  9.8G   3% /var/lib/postgresql
/dev/md1                btrfs     256G  3.1G  250G   2% /var/lib/postgresql/data
overlay                 overlay   256G  3.1G  250G   2% /
tmpfs                   tmpfs      64M     0   64M   0% /dev
udev                    devtmpfs   32G     0   32G   0% /dev/tty

Describe the results you expected

I would have expected the files to be persisted in the xfs volume, i.e. like this:

(modified by hand to show what I expected!)
Filesystem              Type      Size  Used Avail Use% Mounted on
tmpfs                   tmpfs     6.3G   96K  6.3G   1% /etc/hosts
shm                     tmpfs      63M  1.1M   62M   2% /dev/shm
/dev/mapper/vg0-xfstest xfs        10G  229M  9.8G   3% /var/lib/postgresql
overlay                 overlay   256G  3.1G  250G   2% /
tmpfs                   tmpfs      64M     0   64M   0% /dev
udev                    devtmpfs   32G     0   32G   0% /dev/tty

podman info output

(I have also tested this with podman 5.3.1 using nix)


host:
  arch: amd64
  buildahVersion: 1.33.7
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.10+ds1-1build2_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: unknown'
  cpuUtilization:
    idlePercent: 99.59
    systemPercent: 0.33
    userPercent: 0.07
  cpus: 32
  databaseBackend: sqlite
  distribution:
    codename: noble
    distribution: ubuntu
    version: "24.04"
  eventLogger: journald
  freeLocks: 2044
  hostname: Ubuntu-2404-noble-amd64-base
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.8.0-51-generic
  linkmode: dynamic
  logDriver: journald
  memFree: 62595571712
  memTotal: 67201568768
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns_1.4.0-5_amd64
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.4.0
    package: netavark_1.4.0-4_amd64
    path: /usr/lib/podman/netavark
    version: netavark 1.4.0
  ociRuntime:
    name: crun
    package: crun_1.14.1-1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.1
      commit: de537a7965bfbe9992e2cfae0baeb56a08128171
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt_0.0~git20240220.1e6f92b-1_amd64
    version: |
      pasta unknown version
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.1-1build2_amd64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 34359734272
  swapTotal: 34359734272
  uptime: 1h 9m 40.00s (Approximately 0.04 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/olof/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/olof/.local/share/containers/storage
  graphRootAllocated: 274742640640
  graphRootUsed: 3273494528
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/olof/.local/share/containers/storage/volumes
version:
  APIVersion: 4.9.3
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.22.2
  Os: linux
  OsArch: linux/amd64
  Version: 4.9.3

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

This was on a bare metal machine on hetzner, with standard installscript configurations to set up an ubuntu btrfs-as-root configuration with a spare lvm block storage partition.

Additional information

This is on a freshly commissioned machine so I will investigate this further with different distros

[saolof]

Nevermind, this was due to the postgres docker image creating an anonymous volume by default. Changing it to mount /var/lib/postgresql/data fixed it