Orange PI PC stuck on compiling headers

[jackbox]

I got Armbian (Jessie desktop) working on Orange PI PC. I did "sudo apt-get update" and then "sudo apt-get upgrade" and everything was ok until  it said "Compiling headers (please wait)." It has now been on this step for a very, very long time. Is it stuck or does this take a ridiculously long time to complete? Should I shut it down and start over or keep waiting? The green light is staying on steady.

[rreignier]

I have noticed the same behavior with my Orange Pi One with Armbian 5.20 upgrading to 5.24.
I have tried 4 times and let it work for one full day without success.

But I was suspecting my SD card or power supply when I did not see anyone on the forum with the same issue. And I am not able to test other SD cards and power supplies at the moment so I will try again later.

[@lex]

I have had similar problem when upgrading and solved this by removing the file /var/lib/dpkg/info/linux-headers-sun8i.postinst and typing:

sudo dpkg --configure -a

you may need to repeat the commands:

sudo apt-get update
sudo apt-get upgrade

*  update: this was Ubuntu 14.04

[jackbox]

I have noticed the same behavior with my Orange Pi One with Armbian 5.20 upgrading to 5.24.

I have tried 4 times and let it work for one full day without success.

But I was suspecting my SD card or power supply when I did not see anyone on the forum with the same issue. And I am not able to test other SD cards and power supplies at the moment so I will try again later.

 

I got success with a more powerful power supply. I used a Qualcomm certified QC 2.0 charger by Aukey. At 5 volts it can put out 3 amps. I reimaged the same sd card, used  that power supply and Armbian fully upgraded. I think these Orange PI PC's are super touchy about voltages. My Raspberry PI can get low voltage and just show the rainbow box in the corner but it keeps on going. The Orange PI PC seems to just freeze up whatever it was doing if the voltage pulls too low. Try it with a stronger power adapter and see if that works for you, it did for me. Finally got this thing working after many frustrating hours trying all sorts of images, etc.

[tkaiser]

I think these Orange PI PC's are super touchy about voltages. My Raspberry PI can get low voltage and just show the rainbow box in the corner but it keeps on going. The Orange PI PC seems to just freeze up whatever it was doing if the voltage pulls too low.

 

Well, it took Raspberry folks a very long time until they realized what a crappy idea it was to use the Micro USB jack to power the board. Unfortunately they never corrected their mistake and now we suffer from the same problem on many other SBC -- fortunately not on Orange Pis.

 

Undervoltage was and is one of the main reasons why instabilities occur with Raspberries. The reason is simple: Micro USB is crap, tiny contacts are responsible for high contact resistance and the connector is therefore specified for 1.8A max (please see the last sentence here what a silly idea it is to try to support 2.5A with this connector) and most if not all USB cables are crap too (cable resistance way too high). That's the reason there are so much special PSUs for RPi around that come with 5.15V or even 5.2V to compensate for the voltage drops.

 

Therefore with the 2nd gen Raspberries they invented voltage measurement 'in the firmware' and provide this even on GPIO35 (the ARM core(s) aren't first class citizens on Raspberries, everything is under control of the VideoCore IV). And if they now see that voltage drops below 4.65V then led warning is triggered and the VideoCore overlays this rainbow thingie. At the same time heavy throttling occurs to prevent freezes (affects IIRC CPU cores, GPU/VPU and DRAM). BTW: With latest firmware the rainbow stuff should be replaced with this instead:

 

 

All Orange Pi are based on cheap SoCs for OTT TV boxes that are not accompanied by circuitry that could do voltage measurements (unlike the tablet SoCs from Allwinner where this is possible and where a connected battery compensate for mains voltage drops perfectly). So more or less you get what you pay for and have to ensure yourself to properly power these boards. A good 5V/2A PSU is sufficient. Only when you try to attach tons of host powered USB peripherals you need to get close to 3A.

 

Some PSUs are crap, some cables are crap. Usually the problem is undervoltage and the problem with this is that this occurs under heavy load (Ohm's law: the higher the current needed, the more the voltage drops if there's too much resistance involved by either cable or contacts).

 

We already talked about an easy measure to signal this problem: Simply run a heavy task like cpuburn-a7 directly at firstrun. Everyone with insufficient power supply will never get beyond the first boot. And might get the idea way sooner that's a powering problem. Now this problem will be discovered way too late when compiling this kernel header stuff the first time or doing anything else a little bit more heavy.

 

I really don't know which variant is better but from my point of view I would prefer the 'DOA' method: Never be able to fully boot up Armbian with an insufficent powering setup.

 

BTW: You find a lot numbers regarding consumption and performance in a thread with the same name: https://forum.armbian.com/index.php/topic/1748-sbc-consumptionperformance-comparisons/

[Carlos Rodrigues]

Yes this is a topic from 2016 but i have the same issue on an OrangePi Zero...

 

I have tried every thing on the last 3 days:

• dpkg --configure -a
• changed de SD
• changed the USB Cable
• changed the PSU
• changed the OS from debian to Ubuntu

What am i missing?

 

 

[Igor]

25 minutes ago, Carlos Rodrigues said:

• changed de SD

If your SD card is not from this century, upgrade procedure can take whole day(s).

[Carlos Rodrigues]

Its a toshiba Excercia 8GB

[Igor]

18 minutes ago, Carlos Rodrigues said:

Its a toshiba Excercia 8GB

This tells absolutely nothing (due too huge numbers of fakes and I need to google around and trust 3rd party tests), while running this:

iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2

does. 

There is a walk around - just kill the kernel headers compilation process and continue.

[Carlos Rodrigues]

                kB  reclen    write  rewrite    read    reread    read     write     
          102400       4      751      720     7118     7114     6195      724
          102400      16     7028     7701     9874    10425     9888       62
          102400     512    15734    15619    21914    21913    21834     2003
          102400    1024    15827    15634    21989    21986    21952     3954
          102400   16384    15850    16242    22202    22200    22196    16204
 

[Igor]

Very slow for system operations, good enough for storing photos and video. Small files random write speed mater and not every SD card can handle this:

Your SD:

                                                    random    random
    kB  reclen    write  rewrite    read    reread    read     write
102400       4      751      720     7118     7114     6195      724
102400      16     7028     7701     9874    10425     9888       62

Samsung EVO32

102400       4     2886     3256     7620     7627     7625     3419
102400      16     9870    10306    14685    14672    14687     9314

Do the math. If you compare those numbers further to some SSD drive, things looks even more bizarre.

If you want to understand more about this issue, read this topic:
https://forum.armbian.com/index.php?/topic/954-sd-card-performance

Get proper SD card or use some workaround - freeze kernel packages that it won't upgrade since your SD card is not ready for small files manipulation in decent time. Upgrade is stress for system and you need to have solid SD card to make hassle free upgrades.

[Jean-Christian de Rivaz]

I have found that the cause of the endless "Compiling headers - please wait ..." message was that the 'make scripts' into the kernel headers directory (/usr/src/linux-header-<version>/) from the postinst package script detect clock skew and files with modification time in the future causing the redo of the target forever. This happen to me because I installed the kernel-headers package on a target that don't have a way to correctly set his real time clock and so has a real time in the past. The simplest way to fix this is to set a correct date on the target with the command 'date -s "<iso-8601-date>"', where the <iso-8601-date> is replaced by the output of the command 'date -Is' on a machine with a correct date. 

 

For example: On my workstation, 'date -Is' returns 2018-07-25T04:06:50+02:00, so on the target I use the command 'date -s "2018-07-25T04:06:50+02:00"' before installing the kernel-headers package.

 

I also found that a 'find -exec touch {} \;' into the kernel headers directory also fix the issue.

 

 

[Igor]

4 hours ago, Jean-Christian de Rivaz said:

I also found that a 'find -exec touch {} \;' into the kernel headers directory also fix the issue.

Good catch. Now we need to add it here (+ all other source packaging patches) and test. This method is picky on a shell and it doesn't work everywhere. It has to be done a bit differently.

[Jean-Christian de Rivaz]

Why not doing the 'make scripts' while creating the package on the host instead of doing it while executing the postinst on the target ? Could be the cleanest way to solve this issue. I mean, Debian linux-headers package don't do a 'make scripts' unless something explicitly tell to do this in a /etc/kernel/header_postinst.d/ file, so what's the initial goal of this extra operation on Armbian ?

Edited July 25, 2018 by Jean-Christian de Rivaz
Compare with Debian linux-headers package

[Igor]

Well. This workaround is since early days and its a step forward from not working at all to yes, you can use headers. We support a lot of different kernels, which are a decade apart. Stock Debian has a very simple job here, fixing one script while we can do just that. I agree that your proposal would be better but ... we are just a few vs "1000 issues". If someone can make and test on mainline I can port this to others ... when I finish operation "sand castle"



Wrote on mobile

[Igor]

2 hours ago, Jean-Christian de Rivaz said:

Debian linux-headers package don't do a 'make scripts' unless something explicitly tell to do this in a /etc/kernel/header_postinst.d/ file,

IIRC why we add this make scripts is ... most/many wireless drivers failed to build without running make scripts first. And people start to complain ... and we found a "solution". I think we have to keep this, just need to fix it properly and if possible without too much hassle.

[Jean-Christian de Rivaz]

I use this procedure to reproduce the issue:

systemctl stop ntp
dats -s "2017-07-25T12:34:27+00:00"
cd /usr/src/linux-headers-4.14.52-sunxi/
make scripts

This result into make doing an infinite loop while complaining about files that has modification time in the future.

Then I tried to use this command to fix the issue:

find scripts/ -type f -exec touch {} \;

This reduce the number of complains but still cause an infinite loop. I also tried to touch only make related files, without more success.  Actually the generic way to solve the issue is this command:

find -type f -exec touch {} \;

This fix the infinite loop, even if a few complains remains about some files outside of the actual directory. But this is very slow, especially on my low power Orange Pi Zero that is purposely running at the fixed very low 240 MHz processor frequency (USB powered with a lot of sensors).

But I have found that the following command both solve the issue and is very fast:

find -type f -exec touch {} +

The execution time go from 311 seconds to only 4.5 seconds,  about 70 time faster !

So I suggest to add this command into the linux-headers package postinst script, just before calling the 'make -s scripts'.

[Igor]

Implemented.

[Jean-Christian de Rivaz]

Many thanks Igor for your implementation. 

 

While I use armbian images since almost 2 years now, I am new to the armbian internals. I found a bit strange that the fix need to be ported to every single targets. Is that a design choice ?

[Igor]

8 hours ago, Jean-Christian de Rivaz said:

Is that a design choice ?

 

Unfortunately, it is a necessity due to source diversity. Packaging script from kernel 3.4.y is very different from the one from 4.17.y. Then there are small changes in between. If someone would dig into and clean up this mess, we could end up probably with 4 different patches + a patch that differentiate our DEV and NEXT kernel branch naming. Since all this mess works, upstream changes in this sections are extremely rare ... we don't touch it.