Skip to content

buildah is not relabeling mounted paths and instead relies on the OCI runtime to do this #6071

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
dcermak opened this issue Mar 19, 2025 · 5 comments · May be fixed by #6132
Open

buildah is not relabeling mounted paths and instead relies on the OCI runtime to do this #6071

dcermak opened this issue Mar 19, 2025 · 5 comments · May be fixed by #6132
Assignees
@nalind
Labels
jira Issues which will be sync'd to a card at https://issues.redhat.com/projects/RUN

Comments

@dcermak
Copy link
Contributor

dcermak commented Mar 19, 2025 •
edited
Loading

It appears as if buildah is not relabeling mounted directories itself and instead relies on the OCI runtime to perform this action, which is not really spec compliant.

To reproduce this issue, use the following Dockerfile:

from alpine
run --mount=type=bind,source=vendor,target=file,Z ls -al file

and run buildah bud --runtime /usr/bin/crun -f containerfile . which succeeds, whereas it fails with runc:

❯ buildah bud --runtime /usr/bin/runc -f containerfile .
STEP 1/2: FROM alpine
STEP 2/2: run --mount=type=bind,source=vendor,target=file,Z ls -al file
error running container: from /usr/bin/runc creating container for [/bin/sh -c ls -al file]: time="2025-03-19T11:45:09+01:00" level=error msg="runc create failed: invalid mount &{Source:/var/tmp/buildah954402691/mnt/buildah-bind-target-1 Destination:/file Device:bind Flags:20481 ClearedFlags:0 PropagationFlags:[262144] Data:Z Relabel: RecAttr:<nil> Extensions:0 IDMapping:<nil>}: bind mounts cannot have any filesystem-specific options applied"
: exit status 1
ERRO[0000] did not get container create message from subprocess: EOF
Error: building at STEP "RUN --mount=type=bind,source=vendor,target=file,Z ls -al file": while running runtime: exit status 1

The error here is coming from runc directly: https://github.com/opencontainers/runc/blob/8b98e829f0934fc043f57d73b5c2ee091252a1d2/libcontainer/configs/validate/validator.go#L307

runc throws this warning because it doesn't perform any special handling of the Z flag, whereas crun apparently does (however, it probably shouldn't).

The issue in buildah specifically is, that it doesn't appear to perform the same mount flag "sanitation" that podman does, which drops the z or Z flags and relabels the bind mounted directories: https://github.com/danishprakash/podman/blob/29a90c3f8a1375b585dd3a870f7569ed2599426f/libpod/container_internal_common.go#L282

cc @danishprakash @cyphar
@cyphar
Copy link

cyphar commented Mar 19, 2025

We made runc return an error in 1.2 when you specify a superblock flag for bind-mounts. Previously this was silently ignored, but now you get the error you see here. opencontainers/runc#3990

I dunno if crun has special handling of Z or if they copied our (very suspect) behaviour for mount options in this case and actually buildah is simply mistakenly passing the flags. There were some other changes in 1.2 related to mount options, outlined in the 1.2.0-rc.1 release notes. Imho the old mount option behaviour was completely broken for users that actually cared about mount options but nobody played around with it enough to notice imho (and many things were silently ignored in a way that users could be using different configs to the ones they expect).

@ricardobranco777 ricardobranco777 mentioned this issue Mar 20, 2025
Merged
@nalind nalind added the jira Issues which will be sync'd to a card at https://issues.redhat.com/projects/RUN label Apr 16, 2025
@nalind nalind linked a pull request Apr 16, 2025 that will close this issue
Open
@nalind nalind self-assigned this Apr 16, 2025
@ricardobranco777
Copy link

Tracking in https://bugzilla.suse.com/show_bug.cgi?id=1242445

@TheRealBecks
Copy link

I also use OpenSuse Tumbleweed on two systems. The old but updated system was able to run the container, but the new system with the same Podman version was not able to start the server.

Build without z:

STEP 9/14: RUN --mount=type=cache,destination=/root/.cache/uv     --mount=type=bind,source=uv.lock,destination=uv.lock     --mount=type=bind,source=pyproject.toml,destination=pyproject.toml     uv sync --locked --no-install-project
WARN[0000] pkg/bind: error unmounting "/var/tmp/buildah3452673804/mnt/buildah-bind-target-12": retrieving list of mounts: read /proc/thread-self/mountinfo: bad file descriptor 
ERRO[0000] did not get container create message from subprocess: EOF 
ERRO[0000] retrieving list of mounts: read /proc/thread-self/mountinfo: bad file descriptor 
error running container: from /usr/bin/runc creating container for [/bin/sh -c uv sync --locked --no-install-project]: time="2025-05-06T09:00:36+02:00" level=error msg="runc create failed: invalid mount &{Source:/var/tmp/buildah3452673804/mnt/buildah-bind-target-12 Destination:/root/.cache/uv Device:bind Flags:20480 ClearedFlags:1 PropagationFlags:[1048576] Data:z Relabel: RecAttr:<nil> Extensions:0 IDMapping:<nil>}: bind mounts cannot have any filesystem-specific options applied"
: exit status 1
Error: building at STEP "RUN --mount=type=cache,destination=/root/.cache/uv --mount=type=bind,source=uv.lock,destination=uv.lock --mount=type=bind,source=pyproject.toml,destination=pyproject.toml uv sync --locked --no-install-project": while running runtime: exit status 1

Build with z:

STEP 9/14: RUN --mount=type=cache,destination=/root/.cache/uv,z     --mount=type=bind,source=uv.lock,destination=uv.lock,z     --mount=type=bind,source=pyproject.toml,destination=pyproject.toml,z     uv sync --locked --no-install-project
error running container: from /usr/bin/runc creating container for [/bin/sh -c uv sync --locked --no-install-project]: time="2025-05-06T09:01:31+02:00" level=error msg="runc create failed: invalid mount &{Source:/var/tmp/buildah3313102110/mnt/buildah-bind-target-10 Destination:/uv-ansible-example/pyproject.toml Device:bind Flags:20481 ClearedFlags:0 PropagationFlags:[262144] Data:z Relabel: RecAttr:<nil> Extensions:0 IDMapping:<nil>}: bind mounts cannot have any filesystem-specific options applied"
: exit status 1
ERRO[0000] did not get container create message from subprocess: EOF 
Error: building at STEP "RUN --mount=type=cache,destination=/root/.cache/uv,z --mount=type=bind,source=uv.lock,destination=uv.lock,z --mount=type=bind,source=pyproject.toml,destination=pyproject.toml,z uv sync --locked --no-install-project": while running runtime: exit status 1

Temporary workaround:

Instead of...

podman build --build-arg-file=".env" --tag uv-ansible-example "$SOURCE_DIR"

use:

podman build --build-arg-file=".env" --security-opt label=disable --tag uv-ansible-example "$SOURCE_DIR"

So add the --security-opt label=disable parameter to get it working.

@TheRealBecks
Copy link

I found out that in my case it's all about the cache binds that aren't working anymore:

Not working:

RUN --mount=type=cache,destination=/root/.cache/uv \
    --mount=type=bind,source=uv.lock,destination=uv.lock \
    --mount=type=bind,source=pyproject.toml,destination=pyproject.toml \
    uv sync --locked --no-install-project

Working:


RUN --mount=type=bind,source=uv.lock,destination=uv.lock \
    --mount=type=bind,source=pyproject.toml,destination=pyproject.toml \
    uv sync --locked --no-install-project

My temporary workaround with --build-arg-file=".env" also only worked once, but never again.

@ricardobranco777 Your SUSE bug report can't be accessed publicly, is it possible to change the visibility?

@ricardobranco777
Copy link

@ricardobranco777 Your SUSE bug report can't be accessed publicly, is it possible to change the visibility?

It's just a tracker bug that right now only contains a link to this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nalind nalind

Labels
jira Issues which will be sync'd to a card at https://issues.redhat.com/projects/RUN
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

run: handle relabeling bind mounts ourselves nalind/buildah
5 participants
@nalind @TheRealBecks @cyphar @ricardobranco777 @dcermak