Rootless Buildah does not work on Fedora Cloud images, but does work on Fedora Workstation (Fedora 41)

[DCKcode]

I'm running into a weird issue that I hope someone can shed some light on - rootless Buildah behavior appears to be different between a Fedora Workstation and Fedora Cloud image.

I'm trying to run a simple toy buildah example on a rootless system like so:

$ buildah unshare
# ctr=$(buildah from scratch)
# mnt=$(buildah mount $ctr)
# dnf --use-host-config --installroot=$mnt install -y cowsay

This works as expected on Fedora Workstation 41 and installs a root directory with the cowsay binary. However, when I run this on a Fedora Cloud image, the last command runs into the following error:

Updating and loading repositories:
 Fedora 41 - aarch64 - Updates                                                                             100% |  12.3 MiB/s |  11.4 MiB |  00m01s
 Fedora 41 openh264 (From Cisco) - aarch64                                                                 100% |   6.1 KiB/s |   4.8 KiB |  00m01s
Failed to download metadata (metalink: "https://mirrors.fedoraproject.org/metalink?repo=fedora-41&arch=aarch64") for repository "fedora"
 cannot open file: (13) - Permission denied [/var/lib/dnf/repos/fedora-3353451a6fd4c0e3/countme]

There appears to be a permissions error with the cloud image I don't encounter on the Workstation image. What could be the difference? I've confirmed I get the issue occurs regardless of whether the cloud image is on x86_64 or on aarch64 and regardless of whether SELinux is enabled or not. The buildah version in all cases is from the Fedora repositories - buildah version 1.39.3. All systems are fresh installations with out of the box configuration.

[nalind]

If it's not related to SELinux, this may just be a normal permissions issue. On the system where you're seeing the error, what are the permissions on that file (ls -l, both inside and outside of the buildah unshare environment)? What permissions does your user have (id, both inside and outside of the buildah unshare environment)?

A man -K on "countme" turns up the dnf5.conf(5) man page, which suggests that running dnf with "--setopt countme=false" would disable use of this file.

[DCKcode]

That's a good sanity check to run, but it seems they're identical on both systems. ls -lRaZ /var/lib/dnf/repos is owned by root and nobody in the container, and the structure is identical.

It looks like disabling that option doesn't actually prevent dnf from reading that file so it doesn't work around the issue either sadly.

I wonder if there's way to probe directly what the the error is here. I tried straceing the dnf command, but it just tells me that it indeed fails to openat the countme file with ENOACCESS as the error message already indicates.

[DCKcode]

I found a workaround for this issue. The /var/lib/dnf directory that causes the permission error is configurable by the persistsdir option.

So the following works:

$ buildah unshare
# ctr=$(buildah from scratch)
# mnt=$(buildah mount $ctr)
# persist=$(mktemp -d) # any writable directory would do
# dnf --use-host-config --setopt=persistdir=$persist --installroot=$mnt install -y cowsay

Of course this is a workaround and doesn't really explain the difference between Fedora flavors, but this might be useful to someone.

[DCKcode]

Since the project documentation mentions dnf --installroot should just work, I filed a documentation issue for the moment 🙂