Odroid C4 will not reboot after any sort of kernel update - have tried running nand-sata-install

[deelan]

Armbianmonitor:

http://ix.io/3G0U

Hi there!

 

I just got my hands on a brand new Odroid C4.

I flashed Armbian Bullseye (tried Buster too as Bullseye is marked unstable) onto an SD card.

 

The Issue I'm now facing is, after any sort of kernel update the board will not properly soft reboot anymore.

 

I can restart the freshly installed system (which I generated the ix link with) as many times as I want using /sbin/reboot, but one apt upgrade later the board hangs in limbo when I issue a soft reboot.

 

I can see the system itself shutting down, the blue light stopping to heartbeat, and then nothing happens anymore, until I replug power, then the board boots up fine.

 

I have tried to run nand-sata-install as described on the C4's board page (If you face issues with rebooting run this as root) but it does not help, and I'm not even sure whether it even flashes a new U-Boot at all.

 

Obviously, I have trouble using the board as a server like this, as I'd have to come home and replug it if I ever need to reboot so hopefully this can get fixed 

 

Should anyone require further data, I can always provide that.

Edited November 24, 2021 by deelan

[Igor]

4 hours ago, deelan said:

after any sort of kernel update the board will not properly soft reboot anymore.

 

Upgrade and reboot was tested many times with Focal (userspace anyway does not matter in this), but on HC4 which is almost identical ... C4 was not tested specifically. I wasted almost whole day for this.

 

4 hours ago, deelan said:

until I replug power, then the board boots up fine.

This bug is on and off coming back on Amlogic  ... Yes, this is not happening on Hardkernel images, because they didn't update boot loader since 2015. BTW. We are trying to setup a testing team to at least detect problems earlier, but that costs millions and end user donations are only in thousands. Then Armbian lacks personnel to fix problems once they are find ... 

 

4 hours ago, deelan said:

Should anyone require further data, I can always provide that.

That's enough. Just no specific anyone will be dealing with this. This is forum.

[Igor]

Try switching to nightly builds and update bootloader there. Both in armbian-config.

[sulfum]

Hi everybody!

 

I would like to add my two cent to this discussion.

 

I face similar problem with HC4 (not much difference with C4 as Igor pointed out). I run this board under Armbian Focal and got it updated a few hours ago and the board was not coming back online: no heart bit LED, HDDs spinning but heads seem to be parked, Ethernet showing 1 Gbps speed both on the board and on the switch. I had the same problem this Monday after I updated the board and the kernel but then network stack did not work until I restarted the board once again. I intend to use this board with 2x 4TB HDDs as a backup server for critical company data from the main ZFS@FeeBSD bare metal server and inability to do soft reset makes administration if this system less straightforward that usual.

 

At the moment of writing HC4 here rocks following software/firmware:

 

Quote

uname -a

Linux hc4bckp 5.10.81-meson64 #21.08.6 SMP PREEMPT Mon Nov 22 11:21:51 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

U-Boot

 

I have faced this issue with the HC4 server not being able to come online from a soft reboot a few time now. After some further research I came to a conclusion that it has little to do with kernel update itself but the board seems to boot somewhat differently from when it is a cold start as opposed to soft reboot. I have looked into kernel log and went through syslog, there is nothing bizarre there, all shutdown routines finish just fine: I had my encrypted RAID1 with LVM on top of it as one of the suspects but everything gets nicely unmount, LVM got stopped, and the whole system gets nicely shutdown. The interesting part seems to start at the boot: I hooked up a monitor and a keyboard to the server to see what is going on there. I cannot capture output in any proper way (had to film it with my smartphone) but, it it helps, I can arrange it tomorrow or next week as I have serial to USB adapter at home but not here at work. In short I see the system spits following messages:

 

Quote

Hit any key to stop autoboot: 0

switch to partitions #0, OK

mmc0 is current device

** No partiton table - mmc 0 **

Card did not respond to voltage select: -110

MMC Device 2 not fount

no mmc device at slot 2

starting USB...

Bus usb@ff500000: Register 30000140 MbrPorts 3

Starting the controller

USB XHCI 1.10

scanning bus usb@ff500000 for devices... 2 USB Device(s) found

     scanning usb for storage devices... 0 Storage device(s) found

Device 0: unknown device

Speed: 1000, full duplex

Boot broadcast 1

Boot broadcast 2

etc. till 5

 

and then

DHCP client bound to address 192.168.9.30 (3913 ms)

*** ERROR: 'serverip' not set

Cannot autoload with TFTPGET

missing environment variable: pxeuuid

missing environment variable: bootfile

Retrieving file: pxelinux.cfg/*VARIOUS FILE NAMES*

Speed: 1000, full duplex

***ERROR: 'serverip' not set

and this continues for a while until (what seems like) the option in pxelinux.cfg are exhausted and then it just yields and got stuck there. If I issue reset command in U-Boot the whole cycle continues. This does not happen upon cold boot.

 

And by the way, I boot Armbian with mainline U-Boot with the 'old' approach: I have deleted 4 partitions of petitboot (i.e. mtd0 through mtd3) can this somehow interfere with U-Boot here?

 

Is this helpful? Should I capture full U-Boot output via serial and post it here later?

 

Thank you for your time!

[deelan]

6 hours ago, Igor said:

Try switching to nightly builds and update bootloader there. Both in armbian-config.

I did that and it shows the same behaviour.

 

15 minutes ago, sulfum said:

I face similar problem with HC4 (not much difference with C4 as Igor pointed out).

There actually is a vital difference in the boot process as I've just noticed.

 

The Odroid HC4 has an internal 16MB flash as seen in the bottom of the block diagram

 

 

While the C4 lacks that flash and replaces that with the eMMC slot.

 

 

https://wiki.odroid.com/odroid-hc4/software/boot_sequence shos that the HC4 has Odroid's Petitboot living on the flash.

 

I'm not quite sure why you are only seeing U-Boot logs on your HC4 (does armbian somehow replace petitboot on the spi?)

 

So well, for now I'm not too sure what to make out of all of this, I can certainly try hooking up to serial and see if I can trace any kind of U-boot activity, but I don't understand why the stock image reboots fine all the time but running apt upgrade or switching to nightly (which contains a kernel update) messes things up so much.

 

 

17 minutes ago, sulfum said:

Card did not respond to voltage select: -110

 

As your log shows U-boot has successfully started up but it can't start the SD card. This is most likely a problem with the SD card not reacting well to a soft reboot, using wrong voltage levels and this could be fixable with the use of a different SD card or 

what's also been on my mind. To try and see wether theres any kind of SoC function or MMC kernel function to properly reset the SD-card before rebooting.

 

Another thing to add would be that ArchDroid has rebooted fine for me so it might be worth to consult with their devs about the best way to handle rebooting Amlogic SoCs.

 

 

[deelan]

So after a bit of googling I've came across this: 

 

https://lists.denx.de/pipermail/u-boot/2020-December/435196.html

 

Apparently the reboot issue has been documented back in 2020, and there was a simple DTS patch developed and merged into linux.

I could confirm that my C4 is running the fixed dts with GPIO_OPEN_DRAIN set on the tflash_vdd regulator.

 

However I obviously still experience the issue. 

 

What I want to try is applying some of the changes to the armbian kernel as Hardkernel did in theirs to fix the reboot problem (https://github.com/hardkernel/linux/commit/84628497332a5cd2154c92436ec86fad900fe0af).

I don't know what your stance on doing hacky patches like this is, but I still want to try it.

 

Unfortunately I can't build a kernel. It errors out even though I'm building without making any changes at all. I'm building in a VM on latest Ubuntu Hirsute server. I don't see an error, probably because it gets lots in all of the parallel make threads running.

[c0rnelius]

I can't be 100% sure, but I believe I saw a pull request for this some time ago where some one removed some things related to the reboot issue. I haven't scanned through the patch set as of late, but in my testing the following is needed.

 

Need to revert:

drivers/gpu/drm/meson/meson_drv.c

https://github.com/pyavitz/debian-image-builder/blob/feature/patches/amlogic/odroid/002-linux-odroid-patch-set.patch#L524

https://github.com/pyavitz/debian-image-builder/blob/feature/patches/amlogic/odroid/002-linux-odroid-patch-set.patch#L542

 

Add Odroid reboot:

https://github.com/pyavitz/debian-image-builder/blob/feature/patches/amlogic/odroid/002-linux-odroid-patch-set.patch#L2323

https://github.com/pyavitz/debian-image-builder/blob/feature/patches/amlogic/odroid/002-linux-odroid-patch-set.patch#L2533

 

If you review the pull request, you can see where the revert and odroid power reset patch was removed: https://github.com/armbian/build/pull/3154/files

 

As for mainline u-boot the only thing of real importance is the following revert: https://github.com/armbian/build/pull/3154/files#diff-65100acf19e202ac3f3980da554c205752ea3c67d08fc4a3b445d4397189d12fR36

 

With out it, the boards "especially the N2/+ will kernel panic, as the reserved memory "CMA pool won't be set correctly".  So the patch forces the boards to be marked nomap, hence populated after uboot hands off to the kernel.

 

[sulfum]

Hi everyone,

 

I did some testing in the past days (no UART boot logs though):

 

1. Deelan suggested that the boot problem may stem from uSD card due to the voltage error message. Hence, I have made two things:
   • Made an image of the current Armbian Fossa install (which is btw on a Kingston 32 GB uSD) and wrote it to a Samsung 32 GB uSD. Behavior did not change.
2. I made a fresh install of Armbian Focal and I chose for an older version 21.08.1: this release from 26-Aug-2021. In short, it, of course, booted up without any issues. I could soft reboot it several time without any issues... the I ran apt update && apt upgrade. The system was brought to the current state (i.e. 21.08.6). The system rebooted ad came online just fine after system/kernel update. All following attempts to reboot the system led to the dark silence. For the note, I have tested it with and without HDDs just in case. The result was the same.
3. For the sake of testing I installed Armbian Bullseye 21.08.1 on the Samsung uSD. The result was the same as for fresh install of Armbian Fossa: 21.08.1 booted up and I soft rebooted it about 10 times with no issues, when the system was updated to 21.08.6, it came online from a soft reboot the first time after update, any follow up attempt of a soft reboot led to dark silence and the system could be brought back online only by unplugging the system.

In summary, it looks like Cornelius is right and some of the patches, which were submitted and accepted in the meantime, rendered soft boot of the board/amlogic SoC impossible. I guess I have to cope with it as it is now and shutdown/cold boot the system after each kernel update. Hope that someone with knowledge of this business will get down to the core of the problems (e.g. revert some of the late changes).

 

If there is anything that I can test further.

[coocooc]

I have the same problem with Buster, as described in the hardkernel forum: https://forum.odroid.com/viewtopic.php?p=317345#p317345 (Igor had also answered in this thread).

The same is true for "Lost dog" and "394ijf9rju" there.

 

To add another data point: As described there, the boot behaviour for me depends on the second SATA slot being used or not. Most of you probably use this slot and hence run into this problem.

 

I'll try to build my own image now with the patches reverted that Cornelius mentioned. I'll report, if I succeed with this step.

Edited December 29, 2021 by coocooc

[mjamiol1]

On 12/29/2021 at 8:52 PM, coocooc said:

I have the same problem with Buster, as described in the hardkernel forum: https://forum.odroid.com/viewtopic.php?p=317345#p317345 (Igor had also answered in this thread).

The same is true for "Lost dog" and "394ijf9rju" there.

 

To add another data point: As described there, the boot behaviour for me depends on the second SATA slot being used or not. Most of you probably use this slot and hence run into this problem.

 

I'll try to build my own image now with the patches reverted that Cornelius mentioned. I'll report, if I succeed with this step.

 

Any luck?

[jtmpush18]

Has anyone been able to resolve this issue?

 

If so, how?

[Igor]

https://github.com/armbian/scripts/actions/runs/1709203416

 

  ___      _           _     _    ____ _  _   
 / _ \  __| |_ __ ___ (_) __| |  / ___| || |  
| | | |/ _` | '__/ _ \| |/ _` | | |   | || |_ 
| |_| | (_| | | | (_) | | (_| | | |___|__   _|
 \___/ \__,_|_|  \___/|_|\__,_|  \____|  |_|  
                                              
Welcome to Armbian 22.02.0-trunk.0014 Focal with bleeding edge Linux 5.15.15-meson64

No end-user support: untested automated build

System load:   2%           	Up time:       40 min	
Memory usage:  6% of 3.73G  	IP:	       10.0.30.121
CPU temp:      43°C           	Usage of /:    33% of 15G    	

 

Reboots ok.

[SHA_ndy]

The issue is still persistent on the latest bullseye release.

[Tuffi]

For me as well, a reboot is *not* possible.

  ___      _           _     _   _   _  ____ _  _
 / _ \  __| |_ __ ___ (_) __| | | | | |/ ___| || |
| | | |/ _` | '__/ _ \| |/ _` | | |_| | |   | || |_
| |_| | (_| | | | (_) | | (_| | |  _  | |___|__   _|
 \___/ \__,_|_|  \___/|_|\__,_| |_| |_|\____|  |_|

Welcome to Armbian 22.02.1 Focal with Linux 5.10.102-meson64

System load:   2%               Up time:       1 day 1:12
Memory usage:  25% of 3.70G     IP:            192.168.156.21
CPU temp:      49°C             Usage of /:    20% of 15G
storage/:      1% of 30G

 

Tried also to update the bootloader.

sudo armbian-config
	System > Install > "5 Install/Update the bootloader on SD/eMMC"

 

Without success.

Any tips what to do?

 

Thanks,

Tuffi

[Slycat]

New HC4 owner, switching from a Cubox-i4pro (just very old, usbs are dead, want to avoid using HDD enclosures)

 

Was running into these same issues and the one important thing that solved all of this is, only after removing petit bootloader

armbian-config ---> System -> Install -> Install/Update the bootloader on SPI/Flash

using "install/update the bootloader on SD/eMMC" would not be possible, let alone necessary, on the HC4

 

/sbin/reboot works without fail at this moment, tested on the latest kernel after apt update && apt upgrade

Linux OdroidServ 5.15.30-meson64 #trunk.0012 SMP PREEMPT Sat Mar 19 16:17:31 UTC 2022 aarch64 GNU/Linux

 

 

P.S.: @Igor the /etc/fancontrol config was not working as there is no "fan1_input" in /sys/class/hwmon/hwmon2. Fancontrol.service was failing as it could not find this pointer

Plus, pwmconfig yields "There are no fan-capable sensor modules installed".

My solution was to comment that line in the /etc/fancontrol, restart fancontrol, and ran sysbench to warm up the cpu and the fan worked as intended

 

P.P.S.: seem like zram is ignoring the max syslog limit and is not properly being flushed with --vacuum-size, repopulates after reboot

#!/bin/bash is already set in armbian-truncate-logs so I'll try to trace back where this fails

[Igor]

4 hours ago, Slycat said:

My solution was to comment that line in the /etc/fancontrol, restart fancontrol, and ran sysbench to warm up the cpu and the fan worked as intended

Indeed, my second NAS is cooling again Information added to  https://www.armbian.com/odroid-hc4/ @Technicavolous
 

4 hours ago, Slycat said:

seem like zram is ignoring the max syslog limit and is not properly being flushed with --vacuum-size, repopulates after reboot

#!/bin/bash is already set in armbian-truncate-logs so I'll try to trace back where this fails

There has been some recent fixes close to this topic. Dunno if they cover this.

[Tuffi]

vor 9 Stunden schrieb Slycat:

armbian-config ---> System -> Install -> Install/Update the bootloader on SPI/Flash

 

 

Thank you very much. This was working for me, and my HC4 is now rebooting again. Thanks.

 

How did you get Kernel 5.15? For me on Kernel 5.10.102-meson64 apt update doesn't shows anything new.

 

Regards,

Tuffi

 

 

 

[Igor]

8 minutes ago, Tuffi said:

How did you get Kernel 5.15? For me on Kernel 5.10.102-meson64 apt update doesn't shows anything new.

Kernel 5.15.y comes with EDGE kernel branch. In theory it should be able to switch within armbian-config but some features might not work ... Download and make a separate image and try before.

[Tuffi]

Thanks for the fast answer.

That means I do not have to switch to Kernel 5.15 yet, because you still support Kernel 5.10?

[Slycat]

Reboot continued to work fine

However, I cannot come back online after a power cycle. With or without the button press+hold

 

I'll attempt a clean install later to be sure I did not fudge the sd card

Edit: disregard, bad fstab. Fully operational with cold/hard boot without button press

Edited April 5, 2022 by Slycat
Test results

[SHA_ndy]

Unfortunately this is still persistent for me. I did

 

armbian-config ---> System -> Install -> Install/Update the bootloader on SPI/Flash

 

but still rebooting is not possible. I have populated both SATA slots with a 1TB Samsung SSD. I can see a red LED flickering three times when HC4 is trying to reboot. But rebooting doesn't happen...

 

I’m running Armbian 22.08.8 Bullseye with Linux 5.10.147-meson64.

[Igor]

20 hours ago, SHA_ndy said:

I’m running Armbian 22.08.8 Bullseye with Linux 5.10.147-meson64.

Try EDGE kernel with 5.19.y. I don't see those problems on my device.

[SHA_ndy]

Is it possible to update the kernel via armbian-config?

[krokodyl]

With system on A1 microsd 'reboot' works like a charm.

It doesn't work on V10 microsd.

I have tried armbian and dietpi. Results are the same.

nand-sata-install doesn't help.

[Ash]

For me, this is an SD card issue. I attached the UART pins to my PC and the bootloader fails with messages about the SD card not working. I've recompiled my kernel for the Odroid C4 and reversing some of the changes as suggested by @Cornelius fixes the reboot problem. The C4 reboot hander needs the SD card reset code to reset voltages for UHS-I cards, so that they can be detected properly by the bootloader. The patch can't be simply reversed because some of the upstream power management code has been changed.

 

I will try and get my patches merged into the mainline kernel, but in the meantime I'll submit something to the Armbian project.

 

Changing to a different SD card (A1 perhaps, or non UHS-I?) or to an eMMC card will likely solve this problem, but of course they might cost money!

[SHA_ndy]

In the latest armbian release (bullseye 22.11.1 with kernel 5.19) armbian-config seems to be updated, too. armbian-config lacks the "Install" section now, so it is impossible to install the bootloader to another media like SPI flash or SATA.
I'll try another SD for the moment...

[deelan]

Reporting back after way too much time passed.

 

I played around with other distros that used an older version of U-Boot and seemingly did not suffer from the reboot issue, however the reboot issue started coming back sporadically with newer kernels.

Having read about pure UHS-I cards causing the issue I decided to buy a new SD card for my Odroid C4 and look at that, the reboot issue disappeared completely even with the 6.0 nightly kernel build.

Safe to say I'm moving back to Armbian now having abandoned it previously due to this issue.

 

 

Old card on the left (Class 10 UHS-I)
New card on the right (UHS-I, UHS class 3, A1, V30)

 

Old card does not reboot while new one does.

[sn0re]

My god, I couldn't believe it. I am running a Nextcloudpi instance on my Odroid HC4 and used the SanDisk Industrial SDHC 16 GB card (class 10, UHS-I) that was sent with it when I bought it some years ago. Reboots didn't work so I started searching around. I now cloned it to a 64GB SDXC UHS-I C10 V30 U3 A2 I had lying around and the reboots on Debian bookworm with Armbian 6.6.47 work like a charm now. Thank you, @deelan!

 

I decided to write this comment for better search engine coverage, since I searched around quite a bit and maybe not everyone is aware that the NCP images use Armbian. I hope this is okay, sorry for unearthing the thread.

[g4715]

Well, I have the opposite experience. I was happily running Odroid HC4 with Transcend Premium 400X 16GB class 10 HC-I SDHC card for a few years w/o worrying about the reboot issue until I decided to change the SD card, as the old one started slowing down. I have tried Sandisk Ultra XC 64GB A1, as well as Transcend 16GB SDHC Ultimate 600x, but they both have issues with soft reboot.

 

I have also tried cloning Armbian from the old card to one of the new cards, but I was still facing issues with the new card. I have also tried doing a fresh install on the old card, and then I was not facing issues, so apparently this is related to SD card compatibility.

I have connected a monitor to capture the logs after I hit reboot, and this is what I get:

 

I hope someone will have a working solution for this, as all the advices I have found so far on this forum are not helping, and it seems that my HDD started having issues after the forceful shut-downs. 😕

[meathill]

@ deelan thanks a lot for the hint with the specific Kingston industrial card! Before that i tried 6 different micro sd cards without success in fixing the reboot issue. (Sandisk Ultra, Extreme, Extreme Pro from different years/specs and Samsung Pro/Evo cards). Bought exactly the same kingston industrial with 16gb and now reboots perfectly fine.

 

Edit: I use the Odroid HC4.

Edited November 19 by meathill