Cannot allocate memory while running customize-image.sh for rockchip64 on GitHub Workflow.

sans-ltd · 2025-07-30T17:51:02Z

Build logs:

https://paste.armbian.de/memejarafo

Since about four weeks we encounter out of memory issues when building an Armbian image on GitHub with a customize-image.sh script. We had no big modifications on the build system since then, we have to rebuild from time to time to get the latest version of our application into the image.
We already use a more powerful runner (32GB RAM, 8 cores).

Our compile call:

./compile.sh \
 BOARD=rockpi-4b \
 BRANCH=edge \
 RELEASE=bookworm \
 KERNEL_CONFIGURE=no \
 BUILD_MINIMAL=yes \
 NETWORKING_STACK="network-manager" \
 COMPRESS_OUTPUTIMAGE=xz \
 KERNEL_BTF=no \
 SHARE_LOG=yes

Our build is based on the tag v25.8.0-trunk.260

Any ideas for this?
Thanks in advance

Florian

Edited 19 hours ago by sans-ltd

laibsch · 2025-07-31T01:43:14Z

Hello @sans-ltd, thank you for providing the logs. 32GB sounds like it should be plenty to compile the image successfully. Have you ever tried to reproduce this on bare metal? Would adding swap space to the github runner to mask the issue be an option? Does the runner always error out at the same step? It's a bit odd for apt to be running out of memory like this. Have you been able to monitor memory consumption during a failed build? Can you share your github workflow definition?

sans-ltd · 2025-07-31T10:43:33Z

We saw that Github has no swap activated. So we activated it using thejerrybao/setup-swap-space@v1 step with 64GB which should be plenty enough.

It happens sporadically, but when it happens it happens always at the same place, during this second apt update.
Here is something else interesting:

2025-07-31T09:58:48.5188156Z [0m[1;30m[🐳|[0m🔨[0m]   [0m               total        used        free      shared  buff/cache   available
2025-07-31T09:58:48.5189232Z [0m[1;30m[🐳|[0m🔨[0m]   [0mMem:            31Gi       3.8Gi        15Gi       2.3Gi        14Gi        27Gi
2025-07-31T09:58:48.5190178Z [0m[1;30m[🐳|[0m🔨[0m]   [0mSwap:           63Gi          0B        63Gi
2025-07-31T09:58:48.5643835Z [0m[1;30m[🐳|[0m🔨[0m]   [0mFilesystem      Size  Used Avail Use% Mounted on
2025-07-31T09:58:48.5645400Z [0m[1;30m[🐳|[0m🔨[0m]   [0mtmpfs            32G  2.3G   29G   8% /
2025-07-31T09:58:48.5646148Z [0m[1;30m[🐳|[0m🔨[0m]   [0mtmpfs            32G   12M   32G   1% /tmp
2025-07-31T09:58:48.5646870Z [0m[1;30m[🐳|[0m🔨[0m]   [0mtmpfs            32G     0   32G   0% /var/tmp
2025-07-31T09:58:48.5647587Z [0m[1;30m[🐳|[0m🔨[0m]   [0mtmpfs            32G     0   32G   0% /run/user/0
2025-07-31T09:58:48.5648279Z [0m[1;30m[🐳|[0m🔨[0m]   [0mtmpfs            64M     0   64M   0% /dev
2025-07-31T09:58:48.5648972Z [0m[1;30m[🐳|[0m🔨[0m]   [0m/dev/root       290G  105G  186G  37% /tmp/overlay
2025-07-31T09:58:48.5669589Z [0m[1;30m[🐳|[0m🔨[0m]   [0m------------------------------------------------------------------
2025-07-31T09:58:48.5670721Z [0m[1;30m[🐳|[0m🔨[0m]   [0m------------------------------------------------------------------------------------
2025-07-31T09:58:48.5671415Z [0m[1;30m[🐳|[0m🔨[0m]   [0mInstalling NodeRED
2025-07-31T09:58:49.1670560Z [0m[1;30m[🐳|[0m🔨[0m]   [0mHit:1 http://deb.debian.org/debian bookworm InRelease
2025-07-31T09:58:49.1677442Z [0m[1;30m[🐳|[0m🔨[0m]   [0mHit:2 http://github.armbian.com/configng stable InRelease
2025-07-31T09:58:49.1680378Z [0m[1;30m[🐳|[0m🔨[0m]   [0mHit:3 http://deb.debian.org/debian bookworm-updates InRelease
2025-07-31T09:58:49.1682868Z [0m[1;30m[🐳|[0m🔨[0m]   [0mHit:4 http://deb.debian.org/debian bookworm-backports InRelease
2025-07-31T09:58:49.1685589Z [0m[1;30m[🐳|[0m🔨[0m]   [0mHit:5 http://security.debian.org bookworm-security InRelease
2025-07-31T09:58:58.1575506Z [0m[1;30m[🐳|[0m🔨[0m]   [0mReading package lists...
2025-07-31T09:58:58.2559413Z [0m[1;30m[🐳|[0m🔨[0m]   [0mE: Could not read from /var/lib/apt/lists/deb.debian.org_debian_dists_bookworm-backports_InRelease - getline (12: Cannot allocate memory)
2025-07-31T09:58:58.2561244Z [0m[1;30m[🐳|[0m🔨[0m]   [0mE: The repository 'http://deb.debian.org/debian bookworm-backports InRelease' provides only weak security information.

We added a free -h and a df -h command. It seems that the memleak is in apt-get update because there is around 78GBi of RAM available.

No we did not build on bare metal for a long time.

laibsch · 2025-07-31T10:53:58Z

10 minutes ago, sans-ltd said:

No we did not build on bare metal for a long time.

Thank you for following up. I just did build on bare metal and checked on the result a few minutes ago and came here now to see you already responded. I can confirm this is not a github issue as I ran into the exact same problem on my Thinkpad 16GB RAM and 16GB swap. Never had an OOM issue with any armbian builds before. Thank you for bringing this to our attention.