Golang amd64 build fails on arm host with segmentation faults

[ReToCode]

Issue Description

Building an amd64 image using podman on an ARM machine (Mac M1 Pro in my case), quite often results in segmentation fault (core dumped) at various places. This happens more often with more complex golang dependencies like kubernetes.

More details and reproducer here: https://github.com/ReToCode/golang-qemu-podman-reproducer

Steps to reproduce the issue

See https://github.com/ReToCode/golang-qemu-podman-reproducer

Describe the results you received

See github repo.

Describe the results you expected

Both builds should work equally.

podman info output

podman info
host:
  arch: arm64
  buildahVersion: 1.30.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc38.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 96.3
    systemPercent: 0.2
    userPercent: 3.49
  cpus: 8
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: coreos
    version: "38"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.3.11-200.fc38.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 10090602496
  memTotal: 16330194944
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.8.5-1.fc38.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.5
      commit: b6f80f766c9a89eb7b1440c0a70ab287434b17ed
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-12.fc38.aarch64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 3h 15m 10.00s (Approximately 0.12 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 7
    paused: 0
    running: 0
    stopped: 7
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 106769133568
  graphRootUsed: 34020192256
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 105
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.5.1
  Built: 1685123899
  BuiltTime: Fri May 26 19:58:19 2023
  GitCommit: ""
  GoVersion: go1.20.4
  Os: linux
  OsArch: linux/arm64
  Version: 4.5.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

MacBook Pro, M1 Pro

Additional information

I checked various issues here and in the golang repository (like golang/go#36981), but none of them really seem to match (or help) this exactly. Also I'm not super sure, if this is really a podman issue or an issue with building golang on QEMU.

If you need any additional info, you can also reach out to me on RH slack.

[afbjorklund]

Unfortunately this is a known issue, golang is not really stable when compiing under qemu user emulation.

Cross-compiling (with GOOS/GOARCH) is much faster and more predictable, but does require tool support.

Have you tried GOOS=linux GOARCH=amd64 go build instead ?

With the native (not emulated) go compiler for arm64, that is.

[ReToCode]

Unfortunately this is a known issue, golang is not really stable when compiing under qemu user emulation.

👍 That's the impression I got from reading the issues.

Cross-compiling (with GOOS/GOARCH) is much faster and more predictable, but does require tool support.

Yes, that is kind of the point. It is quite tedious to do this for multiple repositories, scripts, tools settings. It happens quite fast to get something wrong, like the base image is already stored, but in the wrong architecture, or building a binary using go cross-compilation that is different than the base image. Examples with GOOS/GOARCH:

Without explicitly telling podman which architecture to use:

podman build --no-cache .
STEP 1/6: FROM golang:1.19-bullseye
WARNING: image platform (linux/amd64) does not match the expected platform (linux/arm64)
STEP 2/6: ENV BASE=github.com/retocode/golang-qemu-podman-reproducer
--> cf4046f71e2f
STEP 3/6: WORKDIR ${GOPATH}/src/${BASE}
--> 41ccd4f0dc63
STEP 4/6: COPY . .
--> 0f91d8d3ed61
STEP 5/6: ENV GOFLAGS="-mod=vendor"
--> e02700825dca
STEP 6/6: RUN GOOS=linux GOARCH=amd64 go build $BASE
k8s.io/client-go/applyconfigurations/discovery/v1: /usr/local/go/pkg/tool/linux_amd64/compile: signal: segmentation fault (core dumped)
Error: building at STEP "RUN GOOS=linux GOARCH=amd64 go build $BASE": while running runtime: exit status 1

the user needs to get WARNING: image platform (linux/amd64) does not match the expected platform (linux/arm64)

When explicitly defining it:

podman build --platform=linux/amd64 --no-cache .
STEP 1/6: FROM golang:1.19-bullseye
...
STEP 2/6: ENV BASE=github.com/retocode/golang-qemu-podman-reproducer
--> 4476a7259de5
STEP 3/6: WORKDIR ${GOPATH}/src/${BASE}
--> c63cb1e28e95
STEP 4/6: COPY . .
--> 6253b6efb2fe
STEP 5/6: ENV GOFLAGS="-mod=vendor"
--> 13b3f91f4e01
STEP 6/6: RUN GOOS=linux GOARCH=amd64 go build $BASE
k8s.io/apimachinery/pkg/conversion/queryparams: /usr/local/go/pkg/tool/linux_amd64/compile: signal: segmentation fault (core dumped)
Error: building at STEP "RUN GOOS=linux GOARCH=amd64 go build $BASE": while running runtime: exit status 1

still breaks, even if the go binary already "would" be amd64 and the following

podman build --platform=linux/arm64 --no-cache .
STEP 2/6: ENV BASE=github.com/retocode/golang-qemu-podman-reproducer
--> 0e487044c4be
STEP 3/6: WORKDIR ${GOPATH}/src/${BASE}
--> 6dc819bc596f
STEP 4/6: COPY . .
--> d99a7dd2a93a
STEP 5/6: ENV GOFLAGS="-mod=vendor"
--> f271553bb541
STEP 6/6: RUN GOOS=linux GOARCH=amd64 go build $BASE
COMMIT

works, but now we have an amd64 binary based in arm64 base image.

I get that this is not easy to fix (and probably also not on podman to fix it), but the current behaviour is pretty bad from an UX perspective and quite hard to explain to devs. We probably should have some documentation around this.

[afbjorklund]

You probably want an arm64 builder image, and an amd64 result image.

The go image is comparatively huge, so you don't want to ship that anyway...

REPOSITORY                   TAG            IMAGE ID      CREATED        SIZE
docker.io/library/golang     1.19-bullseye  73799244f796  5 days ago     1.02 GB
docker.io/library/debian     bullseye-slim  02ad3ff027de  13 days ago    84 MB

And you can still use binfmt_misc, to execute unit tests etc with qemu-user

EDIT: swapped architectures, I am used to building from Intel and to ARM

[afbjorklund]

The crash is surprising, but it had probably cached the image from the last run `(docker didn't "know" about arch)

WARNING: image platform (linux/amd64) does not match the expected platform (linux/arm64)

So if you do have some old images lingering, they need to be cleared up - or even use explicit arch digests instead.

https://hub.docker.com/_/golang/tags?page=1&name=1.19-bullseye

[ReToCode]

The crash is surprising, but it had probably cached the image from the last run `(docker didn't "know" about arch)

Yes, that is what I meant with that example. The user has to catch the warning and "get" what is happening, then fix it by removing the existing image or use explicit tags.

You probably want an amd64 builder image, and an arm64 result image.

Yeah, we use a multi-stage build anyways. That one was just for simplicity in the reproducer.

So to summarise, yes it works, if the dev uses the right combination of GOOS/GOARCH combined with either explicit architecture tags in the FROM images or having the correct images lying around. I still think that is pretty bad UX for someone that just want to run a simple go build or just uses a build/test script/Makefile on mostly any OSS repository to do a PR.

[Luap99]

Moved to a discussion as this does not look like a podman bug, instead something in qemu-user-static.

[afbjorklund]

The brute force method is to give up on containers and use separate VMs (i.e. go from qemu-user to qemu-system), but Podman machine doesn't support that anymore...

[Luap99]

Wouldn't this make things much slower as the whole OS is emulated vs only the container process?

[afbjorklund]

It most definitely would, doing the cross-compile is the best and doing the usermode is the second best.

[afbjorklund]

Another alternative would be to reach out to a build cluster, and run a remote build on the right arch

[afbjorklund]

Anyway, you can start a Podman instance using Lima for the "wrong" architecture - if you want it.

https://github.com/lima-vm/lima/blob/master//docs/multi-arch.md

• Slow mode (qemu-system)

• Fast mode (qemu-user)

[ReToCode]

I tested this with the rosetta approach, this actually works like a charm and is pretty fast (although being experimental).
Using the golang native approach is still to be prefer, but maybe it helps someone in a similar situation: https://github.com/ReToCode/local-kind-setup/tree/main/lima.

[moonlight16]

I'm confused about the conclusions here. I'm trying to build a container images from my MacBook M2 (arm64) that runs a Golang application that will run on OpenShift (amd64) architecture. It seems we cannot use podman build --arch=linux/amd64, so we're supposed to use podman build --platform=linux/arm64 instead? But it was noted that this is building the base image for arm64? So how would we build a base image for linux/amd64?

[dajrivera]

For those looking how to do this you can use your native architecture and pass the GOOS and GOARCH environment variables: podman build --env=GOOS=linux --env=GOARCH=amd64 .... In your Containerfile, after your build step you can use FROM --platform=linux/amd64 ... for your run image and copy over the binaries.