podman import failed with error 'layer 0 <...> does not match config's DiffID'

[babykart]

Issue Description

Importing a tar.xz archive as a container fail with the following error :

Error: writing blob: layer 0 (blob "sha256:57eca825bd922bd6aa59c30ac95be594144816b3951caefa2cf0be2d86db12d2"/""/"sha256:ebccf21abea5bba2d53df5158901c325b33a5cb1a17c44ddd6cc97c6036a9dcc") does not match config's DiffID "sha256:57eca825bd922bd6aa59c30ac95be594144816b3951caefa2cf0be2d86db12d2"

I tested with an archive that the size is ~45MB without problem.

Steps to reproduce the issue

Steps to reproduce the issue

1. wget https://mirror.init7.net/gentoo//releases/amd64/autobuilds/current-stage3-amd64-musl-hardened/stage3-amd64-musl-hardened-20250309T170330Z.tar.xz
2. podman import stage3-amd64-musl-hardened-20250309T170330Z.tar.xz gentoo-musl:latest

Describe the results you received

Error: writing blob: layer 0 (blob "sha256:57eca825bd922bd6aa59c30ac95be594144816b3951caefa2cf0be2d86db12d2"/""/"sha256:ebccf21abea5bba2d53df5158901c325b33a5cb1a17c44ddd6cc97c6036a9dcc") does not match config's DiffID "sha256:57eca825bd922bd6aa59c30ac95be594144816b3951caefa2cf0be2d86db12d2"

Describe the results you expected

Import task successfull (with podman-5.3.2):

Getting image source signatures
Copying blob 57eca825bd92 done   |
Copying config 0b14cb6fdc done   |
Writing manifest to image destination
sha256:0b14cb6fdc54455ab9964482dff111d02a696a3846df2d4db5bdf1dae6ce0582

podman info output

host:
  arch: amd64
  buildahVersion: 1.39.2
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: app-containers/conmon-2.1.13
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: v2.1.13'
  cpuUtilization:
    idlePercent: 99.89
    systemPercent: 0.06
    userPercent: 0.05
  cpus: 16
  databaseBackend: sqlite
  distribution:
    distribution: gentoo
    version: "2.17"
  eventLogger: file
  freeLocks: 2045
  hostname: masha
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.12.16-gentoo
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 48935530496
  memTotal: 67151851520
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: app-containers/aardvark-dns-1.12.2-r1
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.12.2
    package: app-containers/netavark-1.12.2-r1
    path: /usr/libexec/podman/netavark
    version: netavark 1.12.2
  ociRuntime:
    name: crun
    package: app-containers/crun-1.20
    path: /usr/bin/crun
    version: |-
      crun version 1.20
      commit: 9c9a76ac11994701dd666c4f0b869ceffb599a66
      rundir: /run/crun
      spec: 1.0.0
      +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: net-misc/passt-2025.01.21
    version: |
      pasta 2025.01.21
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: app-containers/slirp4netns-1.2.0
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.6.0
  swapFree: 0
  swapTotal: 0
  uptime: 406h 4m 29.00s (Approximately 16.92 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /vol/containers/storage
  graphRootAllocated: 21474836480
  graphRootUsed: 5049495552
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 120
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /vol/containers/storage/volumes
version:
  APIVersion: 5.4.1
  Built: 1741719773
  BuiltTime: Tue Mar 11 20:02:53 2025
  GitCommit: ""
  GoVersion: go1.23.6
  Os: linux
  OsArch: linux/amd64
  Version: 5.4.1

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[Luap99]

also see #18193, importing via xz is not supported

Was the resulting image on 5.3 before that actually usable?

cc @mtrmac

[mtrmac]

… and #25465 afterwards.

I think the image might have previously mostly worked: until recently, Podman basically didn’t care about the DiffIDs values in the config (which is why the reproducer in #18193 requires import+save+load, it’s only the load step that relies on DiffIDs being accurate).

But now we require DiffIDs to match, so this fails early.

Otherwise, #18193 still applies: it would be ~possible but awkward to support non-OCI-recognized compression formats, or c/image should detect non-gzip compression formats, and reject them if they are not implemented.

[babykart]

@Luap99 @mtrmac Thanks you for the precise responses. I confirm : the images are working well with podman-5.3.2.

podman images

REPOSITORY                                            TAG                     IMAGE ID      CREATED       SIZE
localhost/kubler-gentoo/stage3-amd64-hardened-openrc  latest                  f5e1eb40577b  2 days ago    1.19 GB
localhost/kubler-gentoo/stage3-amd64-hardened-openrc  20250309T170330Z        f5e1eb40577b  2 days ago    1.19 GB
localhost/kubler-gentoo/stage3-amd64-musl-hardened    latest                  631fd823c282  2 days ago    1 GB
localhost/kubler-gentoo/stage3-amd64-musl-hardened    20250309T170330Z        631fd823c282  2 days ago    1 GB
localhost/kubler-gentoo/portage                       20250309                38a1f7a739ab  3 days ago    341 MB
localhost/kubler-gentoo/portage                       latest                  38a1f7a739ab  3 days ago    341 MB

Reading the various tickets, I understand that load and import no longer support bzip2 and xz formats, but this seems to me to be a break with docker compatibility. I don't think theses archive formats are exotic. Won't this impact the adoption of podman?

[Luap99]

If we were able to import xz images fine then we should continue to support that as not doing so it clearly a breaking change. My assumption was the the resulting image was not usable already, but if we were able to run containers with that then this seems clearly unacceptable to break in a minor release. Docker compat is also certainly a factor for us.

@mtrmac Why does the compression format matter here at all? Import only takes a tarball and there is no actual image and/or layer data so why would c/image or c/storage care since whatever we store is the uncompress representation as single layer from the full tar content? Why would we keep any compression info from the tar used to import?
If we decompress the tar outside of podman then load it it seems to work fine.

[mtrmac]

The problematic piece of code in the tarball transport turns the input into a single image layer, and writes (what it thinks is) the uncompressed digest into the DiffID metadata field. And that is:

• disconnected from the “primary consumer” that extracts the layer on disk; that consumer can handle other formats, so the import mostly works, except for the metadata.
• only supporting gzip
• not all that robust, assuming that anything not gzip is uncompressed. (To be fair, this happens throughout the codebase — we have heuristics to recognize compression formats, but not a heuristic to validate that the outcome is a tar. Usually that’s unnecessary because parsing as a tar immediately follows anyway.)

Anyway, the breakage should be easy to fix. The incorrect metadata on the imported tarball (no way to express "xz") would remain, but, meh.

[Luap99]

Anyway, the breakage should be easy to fix. The incorrect metadata on the imported tarball (no way to express "xz") would remain, but, meh.

Which would be fine as it would not be a new regression.
I mean we could work around it by decompression the stream before handing to c/image I suppose if that would help? I don't think we need or want to actually express a xz image if it has no defined media types. But on a podman push I would think it should be able to compress using gzip or zstd to publish it somewhere.

[mtrmac]

I don’t think working around this at the Podman level would be any better — the tarball transports exists ~exclusively for {buildah,podman} import.

What needs to happen is something like containers/image#2795 , just as a demo.

Does anyone have time to test that, confirm that it does fixes this failure, and that a subsequent podman push / podman save produces reasonable output, particularly MIME types?

[Luap99]

Looks reasonable to me at least, though I don't pretend I know all the mime type/layout specifics so see for yourself:

$ bin/podman import ~/testimage.tar.xz test
Getting image source signatures
Copying blob 5329b0732ae3 skipped: already exists  
Copying config 801aac52ee done   | 
Writing manifest to image destination
sha256:801aac52ee1d3197e66fb2dbe18aeed5825bf88aadbe8beaa46bc97a19c0d8d7

$ bin/podman images
REPOSITORY                TAG         IMAGE ID      CREATED       SIZE
localhost/test            latest      801aac52ee1d  26 hours ago  12.2 MB

$ bin/podman push test oci:/tmp/test
Getting image source signatures
Copying blob cecadf112837 done   | 
Copying config 801aac52ee done   | 
Writing manifest to image destination
$ tree /tmp/test/
/tmp/test/
├── blobs
│   └── sha256
│       ├── 3862052efbfa8a003c8aa402438c38251768cafd39f07de49fcdb131684db3ad
│       ├── 801aac52ee1d3197e66fb2dbe18aeed5825bf88aadbe8beaa46bc97a19c0d8d7
│       └── eaa6e476693714fb62b53dcd5a018afb207761efff8181ba49521705089f1915
├── index.json
└── oci-layout

3 directories, 5 files
$ jq . /tmp/test/index.json 
{
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:eaa6e476693714fb62b53dcd5a018afb207761efff8181ba49521705089f1915",
      "size": 348
    }
  ]
}
$ jq . /tmp/test/blobs/sha256/eaa6e476693714fb62b53dcd5a018afb207761efff8181ba49521705089f1915
{
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:801aac52ee1d3197e66fb2dbe18aeed5825bf88aadbe8beaa46bc97a19c0d8d7",
    "size": 419
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:3862052efbfa8a003c8aa402438c38251768cafd39f07de49fcdb131684db3ad",
      "size": 5595088
    }
  ]
}
$ file /tmp/test/blobs/sha256/3862052efbfa8a003c8aa402438c38251768cafd39f07de49fcdb131684db3ad
/tmp/test/blobs/sha256/3862052efbfa8a003c8aa402438c38251768cafd39f07de49fcdb131684db3ad: gzip compressed data, original size modulo 2^32 12175360

$ podman save test -o test.tar
Copying blob cecadf112837 done   | 
Copying config 801aac52ee done   | 
Writing manifest to image destination
$ tar -tf test.tar 
cecadf112837f146298c7433211a515d0df853a8dbe21fb08f1327152c951be6.tar
801aac52ee1d3197e66fb2dbe18aeed5825bf88aadbe8beaa46bc97a19c0d8d7.json
9cb04319c576705ceb1d0fd7e756f9d64b2d22872487b6647031aa5eced5bcbb/layer.tar
9cb04319c576705ceb1d0fd7e756f9d64b2d22872487b6647031aa5eced5bcbb/VERSION
9cb04319c576705ceb1d0fd7e756f9d64b2d22872487b6647031aa5eced5bcbb/json
manifest.json
repositories
$ bin/podman load --input test.tar
Getting image source signatures
Copying blob cecadf112837 skipped: already exists  
Copying config 801aac52ee done   | 
Writing manifest to image destination
Loaded image: localhost/test:latest

[mtrmac]

Thanks! That looks correct, and I think that as long as we are not doing worse than Podman 5.3, that’s ~by definition good enough, at least for the 5.4 release branch. I just don’t want to take on more hard commitments for the next ~2 weeks.

[TomSweeneyRedHat]

Fixes: https://issues.redhat.com/browse/RHEL-85212