buildah is not relabeling mounted paths and instead relies on the OCI runtime to do this

[dcermak]

It appears as if buildah is not relabeling mounted directories itself and instead relies on the OCI runtime to perform this action, which is not really spec compliant.

To reproduce this issue, use the following Dockerfile:

from alpine
run --mount=type=bind,source=vendor,target=file,Z ls -al file

and run buildah bud --runtime /usr/bin/crun -f containerfile . which succeeds, whereas it fails with runc:

❯ buildah bud --runtime /usr/bin/runc -f containerfile .
STEP 1/2: FROM alpine
STEP 2/2: run --mount=type=bind,source=vendor,target=file,Z ls -al file
error running container: from /usr/bin/runc creating container for [/bin/sh -c ls -al file]: time="2025-03-19T11:45:09+01:00" level=error msg="runc create failed: invalid mount &{Source:/var/tmp/buildah954402691/mnt/buildah-bind-target-1 Destination:/file Device:bind Flags:20481 ClearedFlags:0 PropagationFlags:[262144] Data:Z Relabel: RecAttr:<nil> Extensions:0 IDMapping:<nil>}: bind mounts cannot have any filesystem-specific options applied"
: exit status 1
ERRO[0000] did not get container create message from subprocess: EOF
Error: building at STEP "RUN --mount=type=bind,source=vendor,target=file,Z ls -al file": while running runtime: exit status 1

The error here is coming from runc directly: https://github.com/opencontainers/runc/blob/8b98e829f0934fc043f57d73b5c2ee091252a1d2/libcontainer/configs/validate/validator.go#L307

runc throws this warning because it doesn't perform any special handling of the Z flag, whereas crun apparently does (however, it probably shouldn't).

The issue in buildah specifically is, that it doesn't appear to perform the same mount flag "sanitation" that podman does, which drops the z or Z flags and relabels the bind mounted directories: https://github.com/danishprakash/podman/blob/29a90c3f8a1375b585dd3a870f7569ed2599426f/libpod/container_internal_common.go#L282

cc @danishprakash @cyphar

[cyphar]

We made runc return an error in 1.2 when you specify a superblock flag for bind-mounts. Previously this was silently ignored, but now you get the error you see here. opencontainers/runc#3990

I dunno if crun has special handling of Z or if they copied our (very suspect) behaviour for mount options in this case and actually buildah is simply mistakenly passing the flags. There were some other changes in 1.2 related to mount options, outlined in the 1.2.0-rc.1 release notes. Imho the old mount option behaviour was completely broken for users that actually cared about mount options but nobody played around with it enough to notice imho (and many things were silently ignored in a way that users could be using different configs to the ones they expect).

[ricardobranco777]

Tracking in https://bugzilla.suse.com/show_bug.cgi?id=1242445

[TheRealBecks]

I also use OpenSuse Tumbleweed on two systems. The old but updated system was able to run the container, but the new system with the same Podman version was not able to start the server.

Build without z:

STEP 9/14: RUN --mount=type=cache,destination=/root/.cache/uv     --mount=type=bind,source=uv.lock,destination=uv.lock     --mount=type=bind,source=pyproject.toml,destination=pyproject.toml     uv sync --locked --no-install-project
WARN[0000] pkg/bind: error unmounting "/var/tmp/buildah3452673804/mnt/buildah-bind-target-12": retrieving list of mounts: read /proc/thread-self/mountinfo: bad file descriptor 
ERRO[0000] did not get container create message from subprocess: EOF 
ERRO[0000] retrieving list of mounts: read /proc/thread-self/mountinfo: bad file descriptor 
error running container: from /usr/bin/runc creating container for [/bin/sh -c uv sync --locked --no-install-project]: time="2025-05-06T09:00:36+02:00" level=error msg="runc create failed: invalid mount &{Source:/var/tmp/buildah3452673804/mnt/buildah-bind-target-12 Destination:/root/.cache/uv Device:bind Flags:20480 ClearedFlags:1 PropagationFlags:[1048576] Data:z Relabel: RecAttr:<nil> Extensions:0 IDMapping:<nil>}: bind mounts cannot have any filesystem-specific options applied"
: exit status 1
Error: building at STEP "RUN --mount=type=cache,destination=/root/.cache/uv --mount=type=bind,source=uv.lock,destination=uv.lock --mount=type=bind,source=pyproject.toml,destination=pyproject.toml uv sync --locked --no-install-project": while running runtime: exit status 1

Build with z:

STEP 9/14: RUN --mount=type=cache,destination=/root/.cache/uv,z     --mount=type=bind,source=uv.lock,destination=uv.lock,z     --mount=type=bind,source=pyproject.toml,destination=pyproject.toml,z     uv sync --locked --no-install-project
error running container: from /usr/bin/runc creating container for [/bin/sh -c uv sync --locked --no-install-project]: time="2025-05-06T09:01:31+02:00" level=error msg="runc create failed: invalid mount &{Source:/var/tmp/buildah3313102110/mnt/buildah-bind-target-10 Destination:/uv-ansible-example/pyproject.toml Device:bind Flags:20481 ClearedFlags:0 PropagationFlags:[262144] Data:z Relabel: RecAttr:<nil> Extensions:0 IDMapping:<nil>}: bind mounts cannot have any filesystem-specific options applied"
: exit status 1
ERRO[0000] did not get container create message from subprocess: EOF 
Error: building at STEP "RUN --mount=type=cache,destination=/root/.cache/uv,z --mount=type=bind,source=uv.lock,destination=uv.lock,z --mount=type=bind,source=pyproject.toml,destination=pyproject.toml,z uv sync --locked --no-install-project": while running runtime: exit status 1

Temporary workaround:

Instead of...

podman build --build-arg-file=".env" --tag uv-ansible-example "$SOURCE_DIR"

use:

podman build --build-arg-file=".env" --security-opt label=disable --tag uv-ansible-example "$SOURCE_DIR"

So add the --security-opt label=disable parameter to get it working.

[TheRealBecks]

I found out that in my case it's all about the cache binds that aren't working anymore:

Not working:

RUN --mount=type=cache,destination=/root/.cache/uv \
    --mount=type=bind,source=uv.lock,destination=uv.lock \
    --mount=type=bind,source=pyproject.toml,destination=pyproject.toml \
    uv sync --locked --no-install-project

Working:

RUN --mount=type=bind,source=uv.lock,destination=uv.lock \
    --mount=type=bind,source=pyproject.toml,destination=pyproject.toml \
    uv sync --locked --no-install-project

My temporary workaround with --build-arg-file=".env" also only worked once, but never again.

@ricardobranco777 Your SUSE bug report can't be accessed publicly, is it possible to change the visibility?

[ricardobranco777]

@ricardobranco777 Your SUSE bug report can't be accessed publicly, is it possible to change the visibility?

It's just a tracker bug that right now only contains a link to this issue.