Permission error deleting from tmp folder

[GolanTrevize10]

Hi, I am running podman on a Openshift 4.14 container, rootful and unprivileged, with overlayfs. When I try to build a docker image I get warnings like this afet every RUN step

17:33:37  time="2025-03-27T16:33:37Z" level=warning msg="pkg/bind: error detaching \"/var/tmp/buildah2331429047/mnt/buildah-bind-target-1\": permission denied"
17:33:37  time="2025-03-27T16:33:37Z" level=warning msg="pkg/bind: error removing \"/var/tmp/buildah2331429047/mnt/buildah-bind-target-1\": device or resource busy"
17:33:37  time="2025-03-27T16:33:37Z" level=warning msg="pkg/bind: error removing \"/var/tmp/buildah2331429047/mnt\": directory not empty"

This is the output of podman info

17:28:45  + podman info
17:28:47  host:
17:28:47    arch: amd64
17:28:47    buildahVersion: 1.37.6
17:28:47    cgroupControllers:
17:28:47    - cpuset
17:28:47    - cpu
17:28:47    - io
17:28:47    - memory
17:28:47    - hugetlb
17:28:47    - pids
17:28:47    - rdma
17:28:47    - misc
17:28:47    cgroupManager: cgroupfs
17:28:47    cgroupVersion: v2
17:28:47    conmon:
17:28:47      package: conmon-2.1.12-1.el9.x86_64
17:28:47      path: /usr/bin/conmon
17:28:47      version: 'conmon version 2.1.12, commit: c0564282e9befb7804c3642230f8e94f1b2ba9f8'
17:28:47    cpuUtilization:
17:28:47      idlePercent: 99.42
17:28:47      systemPercent: 0.22
17:28:47      userPercent: 0.37
17:28:47    cpus: 40
17:28:47    databaseBackend: sqlite
17:28:47    distribution:
17:28:47      distribution: rhel
17:28:47      version: "9.5"
17:28:47    eventLogger: file
17:28:47    freeLocks: 2048
17:28:47    hostname: jenkins-prg-xt09q
17:28:47    idMappings:
17:28:47      gidmap: null
17:28:47      uidmap: null
17:28:47    kernel: 5.14.0-284.86.1.el9_2.x86_64
17:28:47    linkmode: dynamic
17:28:47    logDriver: k8s-file
17:28:47    memFree: 86292946944
17:28:47    memTotal: 105574219776
17:28:47    networkBackend: netavark
17:28:47    networkBackendInfo:
17:28:47      backend: netavark
17:28:47      dns:
17:28:47        package: aardvark-dns-1.12.2-1.el9_5.x86_64
17:28:47        path: /usr/libexec/podman/aardvark-dns
17:28:47        version: aardvark-dns 1.12.2
17:28:47      package: netavark-1.12.2-1.el9.x86_64
17:28:47      path: /usr/libexec/podman/netavark
17:28:47      version: netavark 1.12.2
17:28:47    ociRuntime:
17:28:47      name: crun
17:28:47      package: crun-1.16.1-1.el9.x86_64
17:28:47      path: /usr/bin/crun
17:28:47      version: |-
17:28:47        crun version 1.16.1
17:28:47        commit: afa829ca0122bd5e1d67f1f38e6cc348027e3c32
17:28:47        rundir: /run/crun
17:28:47        spec: 1.0.0
17:28:47        +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
17:28:47    os: linux
17:28:47    pasta:
17:28:47      executable: /usr/bin/pasta
17:28:47      package: passt-0^20240806.gee36266-6.el9_5.x86_64
17:28:47      version: |
17:28:47        pasta 0^20240806.gee36266-6.el9_5.x86_64
17:28:47        Copyright Red Hat
17:28:47        GNU General Public License, version 2 or later
17:28:47          <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
17:28:47        This is free software: you are free to change and redistribute it.
17:28:47        There is NO WARRANTY, to the extent permitted by law.
17:28:47    remoteSocket:
17:28:47      exists: false
17:28:47      path: /run/podman/podman.sock
17:28:47    rootlessNetworkCmd: pasta
17:28:47    security:
17:28:47      apparmorEnabled: false
17:28:47      capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
17:28:47      rootless: false
17:28:47      seccompEnabled: true
17:28:47      seccompProfilePath: /usr/share/containers/seccomp.json
17:28:47      selinuxEnabled: false
17:28:47    serviceIsRemote: false
17:28:47    slirp4netns:
17:28:47      executable: /usr/bin/slirp4netns
17:28:47      package: slirp4netns-1.3.1-1.el9.x86_64
17:28:47      version: |-
17:28:47        slirp4netns version 1.3.1
17:28:47        commit: e5e368c4f5db6ae75c2fce786e31eef9da6bf236
17:28:47        libslirp: 4.4.0
17:28:47        SLIRP_CONFIG_VERSION_MAX: 3
17:28:47        libseccomp: 2.5.2
17:28:47    swapFree: 0
17:28:47    swapTotal: 0
17:28:47    uptime: 3900h 18m 18.00s (Approximately 162.50 days)
17:28:47    variant: ""
17:28:47  plugins:
17:28:47    authorization: null
17:28:47    log:
17:28:47    - k8s-file
17:28:47    - none
17:28:47    - passthrough
17:28:47    - journald
17:28:47    network:
17:28:47    - bridge
17:28:47    - macvlan
17:28:47    - ipvlan
17:28:47    volume:
17:28:47    - local
17:28:47  registries:
17:28:47    search:
17:28:47    - registry.access.redhat.com
17:28:47    - registry.redhat.io
17:28:47    - docker.io
17:28:47  store:
17:28:47    configFile: /etc/containers/storage.conf
17:28:47    containerStore:
17:28:47      number: 0
17:28:47      paused: 0
17:28:47      running: 0
17:28:47      stopped: 0
17:28:47    graphDriverName: overlay
17:28:47    graphOptions:
17:28:47      overlay.mountopt: nodev,metacopy=on
17:28:47    graphRoot: /var/lib/containers/storage
17:28:47    graphRootAllocated: 267887030272
17:28:47    graphRootUsed: 27154505728
17:28:47    graphStatus:
17:28:47      Backing Filesystem: xfs
17:28:47      Native Overlay Diff: "false"
17:28:47      Supports d_type: "true"
17:28:47      Supports shifting: "false"
17:28:47      Supports volatile: "true"
17:28:47      Using metacopy: "true"
17:28:47    imageCopyTmpDir: /var/tmp
17:28:47    imageStore:
17:28:47      number: 0
17:28:47    runRoot: /run/containers/storage
17:28:47    transientStore: false
17:28:47    volumePath: /var/lib/containers/storage/volumes
17:28:47  version:
17:28:47    APIVersion: 5.2.2
17:28:47    Built: 1737721907
17:28:47    BuiltTime: Fri Jan 24 13:31:47 2025
17:28:47    GitCommit: ""
17:28:47    GoVersion: go1.22.9 (Red Hat 1.22.9-2.el9_5)
17:28:47    Os: linux
17:28:47    OsArch: linux/amd64
17:28:47    Version: 5.2.2

[ninja-quokka]

Hi @GolanTrevize10

We have seen an issue like this here: #5669 (comment)

Could you try using vfs rather than overlayfs?

Checking the code where the error is coming from I don't see any extra logging sadly:

buildah/bind/mount.go

Lines 64 to 111 in 898fbb2

	unmount := []string{}
	unmountAll = func() (err error) {
	for _, mountpoint := range unmount {
	// Unmount it and anything under it.
	if err2 := UnmountMountpoints(mountpoint, nil); err2 != nil {
	logrus.Warnf("pkg/bind: error unmounting %q: %v", mountpoint, err2)
	if err == nil {
	err = err2
	}
	}
	if err2 := unix.Unmount(mountpoint, unix.MNT_DETACH); err2 != nil {
	if errno, ok := err2.(syscall.Errno); !ok || errno != syscall.EINVAL {
	logrus.Warnf("pkg/bind: error detaching %q: %v", mountpoint, err2)
	if err == nil {
	err = err2
	}
	}
	}
	// Remove just the mountpoint.
	retry := 10
	remove := unix.Unlink
	err2 := remove(mountpoint)
	for err2 != nil && retry > 0 {
	if errno, ok := err2.(syscall.Errno); ok {
	switch errno {
	default:
	retry = 0
	continue
	case syscall.EISDIR:
	remove = unix.Rmdir
	err2 = remove(mountpoint)
	case syscall.EBUSY:
	if err3 := unix.Unmount(mountpoint, unix.MNT_DETACH); err3 == nil {
	err2 = remove(mountpoint)
	}
	}
	retry--
	}
	}
	if err2 != nil {
	logrus.Warnf("pkg/bind: error removing %q: %v", mountpoint, err2)
	if err == nil {
	err = err2
	}
	}
	}
	return err
	}

As there is a permission denied error are you able to check if there are any errors logged in the systems journal.

There is also a device or resource busy error, are there any issues with the systems storage? Maybe low disk space on /tmp?

Are you able to reproduce this issue?

Are you able to provide a simple reproducer?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.