NanoPC-T4 latest-stable won't boot after initial package update

[smemsh]

Armbianmonitor:

http://ix.io/1Ty0

steps to reproduce:

1. download official stable images and extract
2. write either image (both fail in same way) to sdcard:
Armbian_5.91_Nanopct4_Debian_buster_default_4.4.179.img
Armbian_5.90_Nanopct4_Ubuntu_bionic_default_4.4.179.img
3. boot sdcard -> works fine

4. run armbian-config -> firmware update

5. (also tried "apt update && apt full-upgrade", same result)
6. reboot -> boot loop

happens every time, either the ubuntu or debian versions of armbian.  no changes made after booting beyond just doing the update.  reliably boot loops (it makes it to booting the kernel, then the board restarts after a while of no output)

 

note: I cannot see the kernel printk messages, because any time I add "console=ttyS2,1500000" last (either by using "console=serial" in armbianEnv.txt or adding it to extraargs), it also boot loops then (even pre-upgrade).  ttyS2 works fine as one of the consoles, and getty/login works (via minicom), but it cannot be the primary console (ie the last one specified) or it will boot loop.  it may need an earlyprintk argument or something... not sure how to proceed.  seems to work fine until updating it...

 

note: the armbianmonitor URL is from *before* reboot obviously since it will not make it to OS init after

[Igor]

23 hours ago, smemsh said:

happens every time, either the ubuntu or debian versions of armbian. 

Those images and upgrades were tested many many times, because support is expensive and "you" forget to cover that cost. I test this particular case 3x:

- updating installation which was already on eMMC

- one old version: Armbian_5.75_Nanopct4_Debian_stretch_default_4.4.174.7z

- the same as you did: Armbian_5.91_Nanopct4_Debian_buster_default_4.4.179.7z http://ix.io/1TCV

... and I can't reproduce. How can we explain this case?

 

Have you read this: https://docs.armbian.com/User-Guide_Getting-Started/?

 

Do you use decent/quality SD card?
Proper SD card writing method?

Official PSU?

[smemsh]

There's no reason to believe anything is wrong with the board.   Note that FriendlyELEC images work fine. The machine had been running on the sdcard for months (not really in use, but operating nonetheless), then I decided to start over again with the newest image (written with dd, 512 bytes at a time).  All I did was write the vendor image, boot it, run updates, and reboot.  That was enough to make it boot-loop.

 

I'm using a 12V 2A supply and a SanDisk 16GB EDGE A1 U1 Class 10, both were purchased from FriendlyELEC with the board.  The heat sink is cooled with a 40mm Noctua soldered to the 12V supply.   I'm booting from sdcard, not eMMC.  I'm just writing the stock server image from the downloads page, booting it, running updates, then rebooting, and it boot loops.

 

Is there some way to get kernel console output on the serial port? Again, if I put the serial port as primary console, it will boot loop (even with stock sdcard image that boots fine otherwise), so in my mind that's something to fix first, to get more information from the early printks.   (Note that adding some other seemingly innocuous things like either apparmor=0 or kernel.sysrq=1 to extraargs, also causes the "good" image to boot loop, but other options work fine like ipv6.disable=1 and selinux=0).  If I could just get the kernel boot messages shown on the serial port...

[Igor]

1 hour ago, smemsh said:

Note that FriendlyELEC images work fine.

And Armbian images also work fine.
 

1 hour ago, smemsh said:

written with dd, 512 bytes at a time

So you didn't read our user guide. Not sure it is related to this problem, but do it anyway. You will see more.

 

1 hour ago, smemsh said:

I'm just writing the stock server image from the downloads page, booting it, running updates, then rebooting, and it boot loops.

I did the same and using 2 different SD cards. Just to make sure. I made an image (with Etcher which verify, not with DD - this way DD should also be fine) boot it, create user, run apt update and apt upgrade following by reboot. No problems. (On T4 you have to hold a boot button to boot from SD card, otherwise it starts boot loader from eMMC. Probably this is what bothers you)

 

1 hour ago, smemsh said:

Is there some way to get kernel console output on the serial port?

Yes. There are 4 pins TX RX 5V GND (check manual or I think its also marked on PCB). UART is accessible with unusual speed: 1500000

 

1 hour ago, smemsh said:

Note that adding some other seemingly innocuous things like either apparmor=0 or kernel.sysrq=1 to extraargs, also causes the "good" image to boot loop, but other options work fine like ipv6.disable=1 and selinux=0

Selinux and apparmor is disabled since its broken and nobody has time to fix that. Anyway, you don't want this ... but at stock image this could be enabled. I am not sure.

[smemsh]

Thanks for testing the exact scenario with your board.  I'm very puzzled then, if it works for you.  To test your suspicion of corrupt sdcard write, after copying the image, I used dd again to copy it back (up to the same offset as the image size) and the md5sum of the copy is identical to that of the original image file.

 

As mentioned, I already am using the serial interface successfully, just not as the primary console, which means there's no way to see the kernel printk's.  I'm using a 1a86:7523,  "QinHeng Electronics HL-340 USB-Serial adapter" which is capable of this high baud rate.  I see initial boot messages, U-Boot output, can interrupt with a key, etc.   As well, once the system boots, it works with a getty, I can login, no console corruption.

 

What does NOT work is making the serial port *primary console* (ie, the last console= arg to appear on kernel command line) so we can see the kernel boot messages.   Try adding a line "console=serial" (normally it's "both" which results in console=tty1 as the last console= arg, as one can see in boot.cmd) to armbianEnv.txt and you should see for yourself the same result.  Apparently ttyS2 cannot be used as the primary console for some reason.  Are you saying this works for you? I can write the stock sdcard image and make no other changes besides that one line to armbianEnv.txt and it will boot loop.

[smemsh]

I tried booting off sdcard and then immediately installing to eMMC, rebooting, and *then* running upgrades.  Now it reboots ok after doing the upgrade on eMMC instead of the sdcard.  I can't explain this.  I had installed a stretch 5.0.0 nightly a while ago, but it didn't see my NVMe (Samsung 970 EVO).  At some point I was booting off sdcard again because the 4.4 installer can see my SSD, then left it alone a few months.  when I came back I started over with a fresh 4.4 sdcard image and then the above reported sequence happened.

Furthermore, console=serial in armbianEnv.txt seems to work now.  I am perplexed how this could work on eMMC, but not on sdcard.  I dont understand how that can be so, but everything is working.  Thanks greatly for your help!