Build environment - virtualization

[jgauthier]

Is anyone else running their build environment in a VM? Specifically hosted by QEMU?  I'm running into some strange things, and wondering if anyone has seen it.

I'm running an Ubuntu (20.04) in a VM using QEMU as the hypervisor.

 

Basically, the build generally works fine.  However, as I add more features/processes to the customization script, I am starting to see segmentation faults.

I see this with sudo and when I try to git pull something.

This is what I get:

 

qemu: uncaught target signal 11 (Segmentation fault) - core dumped
Segmentation fault (core dumped)

I got here by dumping into a bash script from 'customize-image.sh', so it's an arm-based chroot environment.

 

I can work around some things, by doing them on the board. and them moving over the directories (or whatever), but I can't always do that.

 

Interested if anyone else is doing this with this system platform.

 

QEMU emulator version 4.2.0 (qemu-4.2.0-7.fc32)
Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers

 

[Werner]

Not sure if I get that right. You are using QEMU to kind of nativly build armbian on arm?

[jgauthier]

5 minutes ago, Werner said:

Not sure if I get that right. You are using QEMU to kind of nativly build armbian on arm?

 

 

I'm using QEMU as my host VM platform and run several VMs in it for my own use.  One of those is an Ubuntu VM intended for building armbian.

I'm going to edit my post for clarification on that.

[jgauthier]

I built a quick Focal system on VirtualBox and ran into the same issue.  So, I realized it's not the qemu host that I was using previously, but the qemu instance running on the build environment.

 

/usr/bin/qemu-arm-static /bin/bash /tmp/customize-image.sh buster sunxi bananapim2zero no

 

I'm not sure this is an easily solvable problem.

[lanefu]

By chance can you share the customization script that is failing?

[lanefu]

I spun up a focal VM on my qemu-kvm host.. ran into same issue....but

I believe it's because you may be missing the binfmt kernel module that qemu requires for executing arm binaries during the customization process.

 

root@armbian-test-builder-qemu:/lib/modules# modprobe binfmt_misc
modprobe: FATAL: Module binfmt_misc not found in directory /lib/modules/5.4.0-1009-kvm

I switched kernels to linux-image-virtual-hwe-20.04 and then I was able to build successfulyl

 

lane@armbian-test-builder-qemu:~/GIT/build$ uname -a
Linux armbian-test-builder-qemu 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
lane@armbian-test-builder-qemu:~/GIT/build$ sudo lsmod|fgrep binfmt
binfmt_misc            24576  1

 

Spoiler

chroot: failed to run command ‘/bin/bash’: Exec format error
chroot: failed to run command ‘/bin/bash’: Exec format error
chroot: failed to run command ‘/bin/bash’: Exec format error
[ .... ] Installing [ linux-u-boot-current-orangepizero_20.08.0-trunk_armhf.deb ]
[ .... ] Installing [ linux-image-current-sunxi_20.08.0-trunk_armhf.deb ]
[ .... ] Installing [ linux-dtb-current-sunxi_20.08.0-trunk_armhf.deb ]
[ .... ] Installing [ armbian-firmware_20.08.0-trunk_all.deb ]
chroot: failed to run command ‘/bin/bash’: Exec format error
chroot: failed to run command ‘/bin/bash’: Exec format error
chroot: failed to run command ‘/bin/bash’: Exec format error
chroot: failed to run command ‘/bin/bash’: Exec format error
chroot: failed to run command ‘/bin/bash’: Exec format error
chroot: failed to run command ‘/bin/bash’: Exec format error
chroot: failed to run command ‘/bin/bash’: Exec format error
chroot: failed to run command ‘/bin/bash’: Exec format error
chroot: failed to run command ‘/bin/bash’: Exec format error
[ o.k. ] Enabling serial console [ ttyS0 ]
[ o.k. ] Enabling serial console [ ttyGS0 ]
cp: cannot stat '/home/lane/GIT/build/.tmp/rootfs-current-orangepizero-bionic-no-yes/etc/armbian-release': No such file or directory
[ o.k. ] Building kernel splash logo [ bionic ]
chroot: failed to run command ‘/bin/bash’: Exec format error
chroot: failed to run command ‘/bin/bash’: Exec format error
chroot: failed to run command ‘/bin/bash’: Exec format error
[ .... ] Installing additional packages [ EXTERNAL_NEW ]
chroot: failed to run command ‘/bin/bash’: Exec format error
[ o.k. ] Calling image customization script [ customize-image.sh ]
chroot: failed to run command ‘/bin/bash’: Exec format error
[ error ] ERROR in function customize_image [ image-helpers.sh:117 ]
[ error ] customize-image.sh exited with error (rc: 126)
[ o.k. ] Process terminated
[ o.k. ] Unmounting [ /home/lane/GIT/build/.tmp/rootfs-current-orangepizero-bionic-no-yes/ ]
[ error ] ERROR in function unmount_on_exit [ image-helpers.sh:66 ]
[ error ] debootstrap-ng was interrupted

 

[lanefu]

See It's noted right there in the documentation: https://docs.armbian.com/Developer-Guide_Build-Preparation/#what-do-i-need
 

...Just kidding I just updated the document.

[jgauthier]

It's not all executables,  just specific ones.    It's reproducible by putting a "bash" in your customization script and then running at least:

sudo
git

 

There may be others.

It can also be tested by mounting an existing disk image (say /mnt) then chroot'ing to /mnt.

The OS automatically runs 

/usr/bin/qemu-arm-static /bin/bash -i

when the chroot is executed. Then it's like being in the build environment.

# sudo ls
qemu: uncaught target signal 11 (Segmentation fault) - core dumped
Segmentation fault (core dumped)

 

So it's not the build environment at all, but an issue with qemu.   I'm just surprised no one has seen this much.  But maybe people aren't trying to highly customize and mass produce armbian images.
 

[lanefu]

right /usr/bin/qemu-arm-static /bin/bash -i requires the `binfmt_misc` kernel module... can you confirm your build vm has that module.

I literally replicated your issue by not having that module, and then switched to a kernel with that module, and built successful.  Please at least consider investigating the binfmt_misc module

[jgauthier]

Sorry about that.  I actually loaded the module before I posted.  Since it didn't resolve anything I assumed that it didn't make a difference so I didn't mention it.

 

Linux armbianbuild 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

lsmod|fgrep binfmt
binfmt_misc            24576  1

 

Then I mounted the raw image on /mnt

 

# chroot /mnt
# sudo ls
qemu: uncaught target signal 11 (Segmentation fault) - core dumped
Segmentation fault (core dumped)

 

[Przemysław Sztoch]

I have similar problem if I want to build debian bullseye armbian image under virtualized ubuntu (host).

Problem is in ldconfig arm64 binary launched by "Installing base system [ Stage 2/2 ]" (/debootstrap/debootstrap)

 

There is not problem if I want to build image for debian buster.

[Igor]

25 minutes ago, Przemysław Sztoch said:

There is not problem if I want to build image for debian buster.

For Debian Bullseye / SID and Ubuntu Hirsute target you need to run build system on Ubuntu 21.04. There is a bug in Qemu which is out of possible for us to do anything about. Upgrading that on Focal is probably nightmare. Also we never tried to support targets that are full of bugs / still in development.

Workaround: build on Ubuntu 21.04 or build natively on ARM64. But this part is again very experimental - just a few images can be build natively.

[Technicavolous]

on this page

https://docs.armbian.com/Developer-Guide_Build-Preparation/

 

links an 'mini.iso' for the official supported build system. Do you have such for 21.04?

[Werner]

We do not provide those isos. And it seems there are no mini isos for Hirsute available already.

https://www.google.com/search?q=ubuntu+hirsute+server+iso

[Przemysław Sztoch]

After upgrade to hirsute armbian builder works ok.