kube play --replace doesn't always replace pod

[asnoeyink]

Issue Description

When using podman kube play, sometimes --replace doesn't replace a previously created pod of the same name. It seems to only happen if the yaml is broken in a specific way, and then pods are played again. Also, it only happens when the yaml contains multiple deployments, and the first deployment is the one that is broken.

Steps to reproduce the issue

To reproduce the issue, use the yaml below:

deployments.yaml

apiVersion: v1
kind: Deployment
metadata:
  creationTimestamp: "2023-08-22T20:00:50Z"
  labels:
    app: podman-replace-demo
  name: busybox-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: busybox-1
  template:
    metadata:
      labels:
        app: busybox-1
    spec:
      containers:
      - command:
        - /bin/sh
        image: docker.io/library/busybox:latest
        name: busybox1
        stdin: true
        tty: true
---
apiVersion: v1
kind: Deployment
metadata:
  creationTimestamp: "2023-08-22T20:00:51Z"
  labels:
    app: podman-replace-demo
  name: busybox-deployment2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: busybox-2
  template:
    metadata:
      labels:
        app: busybox-2
    spec:
      containers:
      - command:
        - /bin/sh
        image: docker.io/library/busybox:latest
        name: busybox2
        stdin: true
        tty: true

1. Play the yaml using cat deployments.yaml | podman kube play - --replace
2. Break the first deployment by changing the image from docker.io/library/busybox:latest to dockerio/library/busybox:latest (removing the dot)
3. Play the yaml again with cat deployments.yaml | podman kube play - --replace
4. There will be an error (expected)
5. Fix the first deployment's image back to docker.io/library/busybox:latest
6. Play the yaml one more time with cat deployments.yaml | podman kube play - --replace

Describe the results you received

You should see the error: Error: encountered while bringing up pod busybox-deployment-pod: adding pod to state: name "busybox-deployment-pod" is in use: pod already exists

Describe the results you expected

I expected the pod to be re-created because it shares the same name as the new pod.

podman info output

host:
  arch: amd64
  buildahVersion: 1.31.2
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc37.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 74.28
    systemPercent: 4.85
    userPercent: 20.87
  cpus: 16
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: workstation
    version: "37"
  eventLogger: journald
  freeLocks: 1976
  hostname: xxxxx.xxxxx.local
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.4.11-100.fc37.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 21857017856
  memTotal: 67093999616
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.7.0-1.fc37.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.7.0
    package: netavark-1.7.0-1.fc37.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: crun
    package: crun-1.8.6-1.fc37.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.6
      commit: 73f759f4a39769f60990e7d225f561b4f4f06bcf
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20230625.g32660ce-1.fc37.x86_64
    version: |
      pasta 0^20230625.g32660ce-1.fc37.x86_64
      Copyright Red Hat
      GNU Affero GPL version 3 or later <https://www.gnu.org/licenses/agpl-3.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-8.fc37.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 20h 17m 37.00s (Approximately 0.83 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/xxxxx/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/xxxxx/.local/share/containers/storage
  graphRootAllocated: 510389125120
  graphRootUsed: 101884309504
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 40
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/xxxxx/.local/share/containers/storage/volumes
version:
  APIVersion: 4.6.1
  Built: 1691705275
  BuiltTime: Thu Aug 10 18:07:55 2023
  GitCommit: ""
  GoVersion: go1.19.10
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.1

### Podman in a container

No

### Privileged Or Rootless

Rootless

### Upstream Latest Release

Yes

### Additional environment details

Additional environment details

### Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[vrothberg]

Thanks for reaching out and for providing the reproducer, @Abs25.

There is definitely a problem in --replace with partial removals. podman pod rm -f busybox-deployment-pod yields the same error.