Build environment - virtualization


Recommended Posts

Is anyone else running their build environment in a VM? Specifically hosted by QEMU?  I'm running into some strange things, and wondering if anyone has seen it.

I'm running an Ubuntu (20.04) in a VM using QEMU as the hypervisor.

 

Basically, the build generally works fine.  However, as I add more features/processes to the customization script, I am starting to see segmentation faults.

I see this with sudo and when I try to git pull something.

This is what I get:

 

qemu: uncaught target signal 11 (Segmentation fault) - core dumped
Segmentation fault (core dumped)

I got here by dumping into a bash script from 'customize-image.sh', so it's an arm-based chroot environment.

 

I can work around some things, by doing them on the board. and them moving over the directories (or whatever), but I can't always do that.

 

Interested if anyone else is doing this with this system platform.

 

QEMU emulator version 4.2.0 (qemu-4.2.0-7.fc32)
Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers


 

Link to post
Share on other sites

Armbian is a community driven open source project. Do you like to contribute your code?

5 minutes ago, Werner said:

Not sure if I get that right. You are using QEMU to kind of nativly build armbian on arm?

 

 

I'm using QEMU as my host VM platform and run several VMs in it for my own use.  One of those is an Ubuntu VM intended for building armbian.

I'm going to edit my post for clarification on that.

Link to post
Share on other sites

I built a quick Focal system on VirtualBox and ran into the same issue.  So, I realized it's not the qemu host that I was using previously, but the qemu instance running on the build environment.

 

/usr/bin/qemu-arm-static /bin/bash /tmp/customize-image.sh buster sunxi bananapim2zero no

 

I'm not sure this is an easily solvable problem.

Link to post
Share on other sites

I spun up a focal VM on my qemu-kvm host.. ran into same issue....but

I believe it's because you may be missing the binfmt kernel module that qemu requires for executing arm binaries during the customization process.

 

root@armbian-test-builder-qemu:/lib/modules# modprobe binfmt_misc
modprobe: FATAL: Module binfmt_misc not found in directory /lib/modules/5.4.0-1009-kvm

I switched kernels to linux-image-virtual-hwe-20.04 and then I was able to build successfulyl


 

lane@armbian-test-builder-qemu:~/GIT/build$ uname -a
Linux armbian-test-builder-qemu 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
lane@armbian-test-builder-qemu:~/GIT/build$ sudo lsmod|fgrep binfmt
binfmt_misc            24576  1


 

Spoiler

chroot: failed to run command ‘/bin/bash’: Exec format error
chroot: failed to run command ‘/bin/bash’: Exec format error
chroot: failed to run command ‘/bin/bash’: Exec format error
[ .... ] Installing [ linux-u-boot-current-orangepizero_20.08.0-trunk_armhf.deb ]
[ .... ] Installing [ linux-image-current-sunxi_20.08.0-trunk_armhf.deb ]
[ .... ] Installing [ linux-dtb-current-sunxi_20.08.0-trunk_armhf.deb ]
[ .... ] Installing [ armbian-firmware_20.08.0-trunk_all.deb ]
chroot: failed to run command ‘/bin/bash’: Exec format error
chroot: failed to run command ‘/bin/bash’: Exec format error
chroot: failed to run command ‘/bin/bash’: Exec format error
chroot: failed to run command ‘/bin/bash’: Exec format error
chroot: failed to run command ‘/bin/bash’: Exec format error
chroot: failed to run command ‘/bin/bash’: Exec format error
chroot: failed to run command ‘/bin/bash’: Exec format error
chroot: failed to run command ‘/bin/bash’: Exec format error
chroot: failed to run command ‘/bin/bash’: Exec format error
[ o.k. ] Enabling serial console [ ttyS0 ]
[ o.k. ] Enabling serial console [ ttyGS0 ]
cp: cannot stat '/home/lane/GIT/build/.tmp/rootfs-current-orangepizero-bionic-no-yes/etc/armbian-release': No such file or directory
[ o.k. ] Building kernel splash logo [ bionic ]
chroot: failed to run command ‘/bin/bash’: Exec format error
chroot: failed to run command ‘/bin/bash’: Exec format error
chroot: failed to run command ‘/bin/bash’: Exec format error
[ .... ] Installing additional packages [ EXTERNAL_NEW ]
chroot: failed to run command ‘/bin/bash’: Exec format error
[ o.k. ] Calling image customization script [ customize-image.sh ]
chroot: failed to run command ‘/bin/bash’: Exec format error
[ error ] ERROR in function customize_image [ image-helpers.sh:117 ]
[ error ] customize-image.sh exited with error (rc: 126)
[ o.k. ] Process terminated
[ o.k. ] Unmounting [ /home/lane/GIT/build/.tmp/rootfs-current-orangepizero-bionic-no-yes/ ]
[ error ] ERROR in function unmount_on_exit [ image-helpers.sh:66 ]
[ error ] debootstrap-ng was interrupted

 

Link to post
Share on other sites

It's not all executables,  just specific ones.    It's reproducible by putting a "bash" in your customization script and then running at least:

sudo
git

 

There may be others.

It can also be tested by mounting an existing disk image (say /mnt) then chroot'ing to /mnt.

The OS automatically runs 

/usr/bin/qemu-arm-static /bin/bash -i

when the chroot is executed. Then it's like being in the build environment.

# sudo ls
qemu: uncaught target signal 11 (Segmentation fault) - core dumped
Segmentation fault (core dumped)

 

So it's not the build environment at all, but an issue with qemu.   I'm just surprised no one has seen this much.  But maybe people aren't trying to highly customize and mass produce armbian images.
 

Link to post
Share on other sites

right /usr/bin/qemu-arm-static /bin/bash -i requires the `binfmt_misc` kernel module... can you confirm your build vm has that module.

I literally replicated your issue by not having that module, and then switched to a kernel with that module, and built successful.  Please at least consider investigating the binfmt_misc module

Link to post
Share on other sites

Sorry about that.  I actually loaded the module before I posted.  Since it didn't resolve anything I assumed that it didn't make a difference so I didn't mention it.

 

Linux armbianbuild 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

lsmod|fgrep binfmt
binfmt_misc            24576  1

 

Then I mounted the raw image on /mnt

 

# chroot /mnt
# sudo ls
qemu: uncaught target signal 11 (Segmentation fault) - core dumped
Segmentation fault (core dumped)

 

Link to post
Share on other sites

25 minutes ago, Przemysław Sztoch said:

There is not problem if I want to build image for debian buster.


For Debian Bullseye / SID and Ubuntu Hirsute target you need to run build system on Ubuntu 21.04. There is a bug in Qemu which is out of possible for us to do anything about. Upgrading that on Focal is probably nightmare. Also we never tried to support targets that are full of bugs / still in development.

Workaround: build on Ubuntu 21.04 or build natively on ARM64. But this part is again very experimental - just a few images can be build natively.

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
 Share

1 1