Skip to content

podman takes around 20 seconds to after the executions of containers completed or interrupted by CTRL + C #22279

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
EIC-JayGanatra opened this issue Apr 5, 2024 · 5 comments
Closed

podman takes around 20 seconds to after the executions of containers completed or interrupted by CTRL + C #22279

EIC-JayGanatra opened this issue Apr 5, 2024 · 5 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@EIC-JayGanatra
Copy link

Issue Description

I encountered a situation that required an upgrade of Podman from version 4.1, which comes with Kirkstone by default, to version 4.4.4 in order to utilize the Quadlet feature. To test this, I created a Python-based "Hello World" image. Initially, when I tried to load and run the image, the container executed successfully without any delays.

However, after upgrading Podman to 4.4.4 to use Quadlet, I encountered an issue where, upon completion of the container's execution, it took an additional 20 seconds, and I received the following error:

Exceeded conmon timeout waiting for container e6ac3bb749659a82023fbe35db7b972be8af7d2cc755a32095df04b2beb58e5f to exit

To resolve this, I upgraded not only the Conmon package but also the runc packages and libseccomp. Despite these updates, the issue persisted.

Steps to reproduce the issue

Steps to reproduce the issue

  1. Load the image with podman load -i "tar file name"
  2. Run the image with podman run -ti "image name"

Describe the results you received

root:# podman run -ti --log-level debug hello_world_py
INFO[0000] podman filtering at log level debug
DEBU[0000] Called run.PersistentPreRunE(podman run -ti --log-level debug hello_world_py)
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /mnt/internal_storage/podman/lib/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /mnt/internal_storage/podman/lib/containers/storage
DEBU[0000] Using run root /mnt/internal_storage/podman/run/containers/storage
DEBU[0000] Using static dir /mnt/internal_storage/podman/lib/containers/storage/libpod
DEBU[0000] Using tmp dir /run/libpod
DEBU[0000] Using volume path /mnt/internal_storage/podman/lib/containers/storage/volumes
DEBU[0000] Using transient store: false
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is not being used
DEBU[0000] Cached value indicated that native-diff is usable
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false
DEBU[0000] Initializing event backend journald
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument
DEBU[0000] Configured OCI runtime crun initialization failed: no valid executable found for OCI runtime crun: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/runc"
INFO[0000] Setting parallel job count to 13
DEBU[0000] Successfully loaded 1 networks
DEBU[0000] Pulling image hello_world_py (policy: missing)
DEBU[0000] Looking up image "hello_world_py" in local containers storage
DEBU[0000] Normalized platform linux/arm64 to {arm64 linux [] }
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Trying "localhost/hello_world_py:latest" ...
DEBU[0000] Trying "docker.io/library/hello_world_py:latest" ...
DEBU[0000] Trying "registry.fedoraproject.org/hello_world_py:latest" ...
DEBU[0000] Trying "quay.io/hello_world_py:latest" ...
DEBU[0000] Trying "registry.access.redhat.com/hello_world_py:latest" ...
DEBU[0000] Trying "registry.centos.org/hello_world_py:latest" ...
DEBU[0000] Trying "docker.io/library/hello_world_py:latest" ...
DEBU[0000] Trying "hello_world_py" ...
DEBU[0000] Trying "docker.io/custom/hello_world_py:latest" ...
DEBU[0000] parsed reference into "[overlay@/mnt/internal_storage/podman/lib/containers/storage+/mnt/internal_storage/podman/run/containers/storage]@be4fc274aebce36f0c519b2f942e918793796d52de4b8f81071017ffbd661255"
DEBU[0000] Found image "hello_world_py" as "docker.io/custom/hello_world_py:latest" in local containers storage
DEBU[0000] Found image "hello_world_py" as "docker.io/custom/hello_world_py:latest" in local containers storage ([overlay@/mnt/internal_storage/podman/lib/containers/storage+/mnt/internal_storage/podman/run/containers/storage]@be4fc274aebce36f0c519b2f942e918793796d52de4b8f81071017ffbd661255)
DEBU[0000] exporting opaque data as blob "sha256:be4fc274aebce36f0c519b2f942e918793796d52de4b8f81071017ffbd661255"
DEBU[0000] Looking up image "docker.io/custom/hello_world_py:latest" in local containers storage
DEBU[0000] Normalized platform linux/arm64 to {arm64 linux [] }
DEBU[0000] Trying "docker.io/custom/hello_world_py:latest" ...
DEBU[0000] parsed reference into "[overlay@/mnt/internal_storage/podman/lib/containers/storage+/mnt/internal_storage/podman/run/containers/storage]@be4fc274aebce36f0c519b2f942e918793796d52de4b8f81071017ffbd661255"
DEBU[0000] Found image "docker.io/custom/hello_world_py:latest" as "docker.io/custom/hello_world_py:latest" in local containers storage
DEBU[0000] Found image "docker.io/custom/hello_world_py:latest" as "docker.io/custom/hello_world_py:latest" in local containers storage ([overlay@/mnt/internal_storage/podman/lib/containers/storage+/mnt/internal_storage/podman/run/containers/storage]@be4fc274aebce36f0c519b2f942e918793796d52de4b8f81071017ffbd661255)
DEBU[0000] exporting opaque data as blob "sha256:be4fc274aebce36f0c519b2f942e918793796d52de4b8f81071017ffbd661255"
DEBU[0000] Looking up image "hello_world_py" in local containers storage
DEBU[0000] Normalized platform linux/arm64 to {arm64 linux [] }
DEBU[0000] Trying "localhost/hello_world_py:latest" ...
DEBU[0000] Trying "docker.io/library/hello_world_py:latest" ...
DEBU[0000] Trying "registry.fedoraproject.org/hello_world_py:latest" ...
DEBU[0000] Trying "quay.io/hello_world_py:latest" ...
DEBU[0000] Trying "registry.access.redhat.com/hello_world_py:latest" ...
DEBU[0000] Trying "registry.centos.org/hello_world_py:latest" ...
DEBU[0000] Trying "docker.io/library/hello_world_py:latest" ...
DEBU[0000] Trying "hello_world_py" ...
DEBU[0000] Trying "docker.io/custom/hello_world_py:latest" ...
DEBU[0000] parsed reference into "[overlay@/mnt/internal_storage/podman/lib/containers/storage+/mnt/internal_storage/podman/run/containers/storage]@be4fc274aebce36f0c519b2f942e918793796d52de4b8f81071017ffbd661255"
DEBU[0000] Found image "hello_world_py" as "docker.io/custom/hello_world_py:latest" in local containers storage
DEBU[0000] Found image "hello_world_py" as "docker.io/custom/hello_world_py:latest" in local containers storage ([overlay@/mnt/internal_storage/podman/lib/containers/storage+/mnt/internal_storage/podman/run/containers/storage]@be4fc274aebce36f0c519b2f942e918793796d52de4b8f81071017ffbd661255)
DEBU[0000] exporting opaque data as blob "sha256:be4fc274aebce36f0c519b2f942e918793796d52de4b8f81071017ffbd661255"
DEBU[0000] Inspecting image be4fc274aebce36f0c519b2f942e918793796d52de4b8f81071017ffbd661255
DEBU[0000] exporting opaque data as blob "sha256:be4fc274aebce36f0c519b2f942e918793796d52de4b8f81071017ffbd661255"
DEBU[0000] Inspecting image be4fc274aebce36f0c519b2f942e918793796d52de4b8f81071017ffbd661255
DEBU[0000] Inspecting image be4fc274aebce36f0c519b2f942e918793796d52de4b8f81071017ffbd661255
DEBU[0000] Inspecting image be4fc274aebce36f0c519b2f942e918793796d52de4b8f81071017ffbd661255
DEBU[0000] Inspecting image be4fc274aebce36f0c519b2f942e918793796d52de4b8f81071017ffbd661255
DEBU[0000] using systemd mode: false
DEBU[0000] No hostname set; container's hostname will default to runtime default
DEBU[0000] Loading default seccomp profile
DEBU[0000] Allocated lock 1 for container c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8
DEBU[0000] parsed reference into "[overlay@/mnt/internal_storage/podman/lib/containers/storage+/mnt/internal_storage/podman/run/containers/storage]@be4fc274aebce36f0c519b2f942e918793796d52de4b8f81071017ffbd661255"
DEBU[0000] exporting opaque data as blob "sha256:be4fc274aebce36f0c519b2f942e918793796d52de4b8f81071017ffbd661255"
DEBU[0000] Cached value indicated that idmapped mounts for overlay are not supported
DEBU[0000] Check for idmapped mounts support
DEBU[0000] Created container "c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8"
DEBU[0000] Container "c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8" has work directory "/mnt/internal_storage/podman/lib/containers/storage/overlay-containers/c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8/userdata"
DEBU[0000] Container "c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8" has run directory "/mnt/internal_storage/podman/run/containers/storage/overlay-containers/c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8/userdata"
DEBU[0000] Handling terminal attach
INFO[0000] Received shutdown.Stop(), terminating! PID=1629
DEBU[0000] Enabling signal proxying
DEBU[0000] overlay: mount_data=lowerdir=/mnt/internal_storage/podman/lib/containers/storage/overlay/l/Y5NARJP77ENBSCPOQ5A6CFVBZK:/mnt/internal_storage/podman/lib/containers/storage/overlay/l/FZ2WWB4RC7VOOBDHZ2NFUQB3MS:/mnt/internal_storage/podman/lib/containers/storage/overlay/l/YCB5QE52WKM3PKPRTJAL5C7JYO:/mnt/internal_storage/podman/lib/containers/storage/overlay/l/MNTZI3SXWFG44CFMFLL5JLJNO6,upperdir=/mnt/internal_storage/podman/lib/containers/storage/overlay/5348e734723000ba186307d7b17d1df0ca7b0086a9884f1c2fd4e0848cae18ec/diff,workdir=/mnt/internal_storage/podman/lib/containers/storage/overlay/5348e734723000ba186307d7b17d1df0ca7b0086a9884f1c2fd4e0848cae18ec/work,
DEBU[0000] Mounted container "c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8" at "/mnt/internal_storage/podman/lib/containers/storage/overlay/5348e734723000ba186307d7b17d1df0ca7b0086a9884f1c2fd4e0848cae18ec/merged"
DEBU[0000] Created root filesystem for container c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8 at /mnt/internal_storage/podman/lib/containers/storage/overlay/5348e734723000ba186307d7b17d1df0ca7b0086a9884f1c2fd4e0848cae18ec/merged
DEBU[0000] Made network namespace at /run/netns/netns-86e266ba-b56c-3e7e-bf14-663a1285af04 for container c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8
DEBU[0000] cni result for container c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8 network podman: &{0.4.0 [{Name:cni-podman0 Mac:3e:d8:fc:ae:33:99 Sandbox:} {Name:veth84aaebd9 Mac:86:4b:a6:4b:16:f9 Sandbox:} {Name:eth0 Mac:b6:f7:34:42:10:7c Sandbox:/run/netns/netns-86e266ba-b56c-3e7e-bf14-663a1285af04}] [{Version:4 Interface:0x40003f3278 Address:{IP:10.88.0.7 Mask:ffff0000} Gateway:10.88.0.1}] [{Dst:{IP:0.0.0.0 Mask:00000000} GW:}] {[] [] []}}
DEBU[0000] Adding nameserver(s) from network status of '[]'
INFO[0000] No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode subscription
DEBU[0000] Setting Cgroups for container c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8 to machine.slice:libpod:c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d
DEBU[0000] Workdir "/" resolved to host path "/mnt/internal_storage/podman/lib/containers/storage/overlay/5348e734723000ba186307d7b17d1df0ca7b0086a9884f1c2fd4e0848cae18ec/merged"
DEBU[0000] Created OCI spec for container c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8 at /mnt/internal_storage/podman/lib/containers/storage/overlay-containers/c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8/userdata/config.json
DEBU[0000] /usr/bin/conmon messages will be logged to syslog
DEBU[0000] running conmon: /usr/bin/conmon args="[--api-version 1 -c c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8 -u c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8 -r /usr/bin/runc -b /mnt/internal_storage/podman/lib/containers/storage/overlay-containers/c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8/userdata -p /mnt/internal_storage/podman/run/containers/storage/overlay-containers/c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8/userdata/pidfile -n adoring_jones --exit-dir /run/libpod/exits --full-attach -s -l journald --log-level debug --syslog -t --conmon-pidfile /mnt/internal_storage/podman/run/containers/storage/overlay-containers/c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /mnt/internal_storage/podman/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /mnt/internal_storage/podman/run/containers/storage --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --network-config-dir --exit-command-arg --exit-command-arg --network-backend --exit-command-arg cni --exit-command-arg --volumepath --exit-command-arg /mnt/internal_storage/podman/lib/containers/storage/volumes --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8]"
INFO[0000] Running conmon under slice machine.slice and unitName libpod-conmon-c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8.scope
DEBU[0000] Received: 1714
INFO[0000] Got Conmon PID as 1703
DEBU[0000] Created container c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8 in OCI runtime
DEBU[0000] Attaching to container c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8
DEBU[0000] Received a resize event: {Width:142 Height:36}
DEBU[0000] Starting container c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8 with command [/bin/sh -c /usr/bin/python3 /usr/local/bin/hello_world.py]
DEBU[0000] Started container c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8
DEBU[0000] Notify sent successfully
hello world
DEBU[0020] Exceeded conmon timeout waiting for container c74ad897ce050f0a00611b7469612b43053a8581b271fd78510fa88438d5a0b8 to exit
DEBU[0020] Called run.PersistentPostRunE(podman run -ti --log-level debug hello_world_py)
DEBU[0020] Shutting down engines

Describe the results you expected

it should not take extra time after the execution of the container is completed.

podman info output

podman info --debug
host:
  arch: arm64
  buildahVersion: 1.29.0
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - blkio
  - memory
  - devices
  - freezer
  - net_cls
  - perf_event
  - net_prio
  - hugetlb
  - pids
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: 9a3af8f438d89ffc176f3c76c164088e64bb6b19'
  cpuUtilization:
    idlePercent: 98.95
    systemPercent: 0.78
    userPercent: 0.27
  cpus: 4
  distribution:
    distribution: custom-systemd-wayland-virtualization
    version: 1.0.0
  eventLogger: journald
  hostname: custom-imx8mp-bliss2g
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.15.52+gc56eb7b65d5a
  linkmode: dynamic
  logDriver: journald
  memFree: 897892352
  memTotal: 2074460160
  networkBackend: cni
  ociRuntime:
    name: runc
    package: Unknown
    path: /usr/bin/runc
    version: |-
      runc version 1.1.7+dev
      commit: v1.1.7-2-gb6109acd-dirty
      spec: 1.0.2-dev
      go: go1.20.7
      libseccomp: 2.5.4
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: ""
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.2.0-beta.0+dev
      commit: unknown
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 0
  swapTotal: 0
  uptime: 0h 34m 33.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 0
    stopped: 2
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /mnt/internal_storage/podman/lib/containers/storage
  graphRootAllocated: 28011606016
  graphRootUsed: 327458816
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /mnt/internal_storage/podman/temp
  imageStore:
    number: 1
  runRoot: /mnt/internal_storage/podman/run/containers/storage
  transientStore: false
  volumePath: /mnt/internal_storage/podman/lib/containers/storage/volumes
version:
  APIVersion: 4.4.4
  Built: 1679938614
  BuiltTime: Mon Mar 27 17:36:54 2023
  GitCommit: c8223435f49a860c8ac4281a85db39232745a7cb
  GoVersion: go1.20.7
  Os: linux
  OsArch: linux/arm64
  Version: 4.4.4

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

No

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting
@EIC-JayGanatra EIC-JayGanatra added the kind/bug Categorizes issue or PR as related to a bug. label Apr 5, 2024
@Luap99
Copy link
Member

Luap99 commented Apr 5, 2024

Please test with the latest podman and conmon

@EIC-JayGanatra
Copy link
Author

Hello @Luap99,

Thank you for your suggestion. I've previously attempted to integrate the meta-virtualization layer of the Mickledore Yocto version alongside other BSP layers from Kirkstone, encountering the same issue. Interestingly, the integration functions smoothly with the default Kirkstone branch of meta-virtualization.

I'm considering trying the latest version of Podman, acknowledging that this would necessitate updating the dependencies in Yocto as well. I'm open to giving this a shot.

Could you advise on which packages I should focus on when planning to upgrade Podman? Your guidance would be greatly appreciated.

@Luap99
Copy link
Member

Luap99 commented Apr 8, 2024

podman and conmon,
the 20 seconds delay is because conmon does not write the exit file so you need to figure out why

@EIC-JayGanatra
Copy link
Author

"Okay, so when I checked the path for runRoot: /mnt/internal_storage/podman/run/containers/storage, I noticed it is 51 characters long. Is there a limitation on that? I mean, it is not giving any specific error, and the files are also being created there, but when I tested it with a path of 50 characters or fewer, it worked like a charm."

@Luap99
Copy link
Member

Luap99 commented Apr 10, 2024

see #22272

@Luap99 Luap99 closed this as not planned Won't fix, can't repro, duplicate, stale Apr 10, 2024
@stale-locking-app stale-locking-app bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Jul 10, 2024
@stale-locking-app stale-locking-app bot locked as resolved and limited conversation to collaborators Jul 10, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@EIC-JayGanatra @Luap99