Odroid C4 will not reboot after any sort of kernel update - have tried running nand-sata-install


deelan
 Share

2 2

Recommended Posts

Armbianmonitor:

Hi there!

 

I just got my hands on a brand new Odroid C4.

I flashed Armbian Bullseye (tried Buster too as Bullseye is marked unstable) onto an SD card.

 

The Issue I'm now facing is, after any sort of kernel update the board will not properly soft reboot anymore.

 

I can restart the freshly installed system (which I generated the ix link with) as many times as I want using /sbin/reboot, but one apt upgrade later the board hangs in limbo when I issue a soft reboot.

 

I can see the system itself shutting down, the blue light stopping to heartbeat, and then nothing happens anymore, until I replug power, then the board boots up fine.

 

I have tried to run nand-sata-install as described on the C4's board page (If you face issues with rebooting run this as root) but it does not help, and I'm not even sure whether it even flashes a new U-Boot at all.

 

Obviously, I have trouble using the board as a server like this, as I'd have to come home and replug it if I ever need to reboot so hopefully this can get fixed :(

 

Should anyone require further data, I can always provide that.

Edited by deelan
Link to post
Share on other sites

Donate and support the project!

4 hours ago, deelan said:

after any sort of kernel update the board will not properly soft reboot anymore.

 

Upgrade and reboot was tested many times with Focal (userspace anyway does not matter in this), but on HC4 which is almost identical ... C4 was not tested specifically. I wasted almost whole day for this.

 

4 hours ago, deelan said:

until I replug power, then the board boots up fine.


This bug is on and off coming back on Amlogic :( ... Yes, this is not happening on Hardkernel images, because they didn't update boot loader since 2015. BTW. We are trying to setup a testing team to at least detect problems earlier, but that costs millions and end user donations are only in thousands. Then Armbian lacks personnel to fix problems once they are find ... 

 

4 hours ago, deelan said:

Should anyone require further data, I can always provide that.


That's enough. Just no specific anyone will be dealing with this. This is forum.

Link to post
Share on other sites

Hi everybody!

 

I would like to add my two cent to this discussion.

 

I face similar problem with HC4 (not much difference with C4 as Igor pointed out). I run this board under Armbian Focal and got it updated a few hours ago and the board was not coming back online: no heart bit LED, HDDs spinning but heads seem to be parked, Ethernet showing 1 Gbps speed both on the board and on the switch. I had the same problem this Monday after I updated the board and the kernel but then network stack did not work until I restarted the board once again. I intend to use this board with 2x 4TB HDDs as a backup server for critical company data from the main ZFS@FeeBSD bare metal server and inability to do soft reset makes administration if this system less straightforward that usual.

 

At the moment of writing HC4 here rocks following software/firmware:

 

Quote

uname -a

Linux hc4bckp 5.10.81-meson64 #21.08.6 SMP PREEMPT Mon Nov 22 11:21:51 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

U-Boot

 

I have faced this issue with the HC4 server not being able to come online from a soft reboot a few time now. After some further research I came to a conclusion that it has little to do with kernel update itself but the board seems to boot somewhat differently from when it is a cold start as opposed to soft reboot. I have looked into kernel log and went through syslog, there is nothing bizarre there, all shutdown routines finish just fine: I had my encrypted RAID1 with LVM on top of it as one of the suspects but everything gets nicely unmount, LVM got stopped, and the whole system gets nicely shutdown. The interesting part seems to start at the boot: I hooked up a monitor and a keyboard to the server to see what is going on there. I cannot capture output in any proper way (had to film it with my smartphone) but, it it helps, I can arrange it tomorrow or next week as I have serial to USB adapter at home but not here at work. In short I see the system spits following messages:

 

Quote

Hit any key to stop autoboot: 0

switch to partitions #0, OK

mmc0 is current device

** No partiton table - mmc 0 **

Card did not respond to voltage select: -110

MMC Device 2 not fount

no mmc device at slot 2

starting USB...

Bus usb@ff500000: Register 30000140 MbrPorts 3

Starting the controller

USB XHCI 1.10

scanning bus usb@ff500000 for devices... 2 USB Device(s) found

     scanning usb for storage devices... 0 Storage device(s) found

Device 0: unknown device

Speed: 1000, full duplex

Boot broadcast 1

Boot broadcast 2

etc. till 5

 

and then

DHCP client bound to address 192.168.9.30 (3913 ms)

*** ERROR: 'serverip' not set

Cannot autoload with TFTPGET

missing environment variable: pxeuuid

missing environment variable: bootfile

Retrieving file: pxelinux.cfg/*VARIOUS FILE NAMES*

Speed: 1000, full duplex

***ERROR: 'serverip' not set


and this continues for a while until (what seems like) the option in pxelinux.cfg are exhausted and then it just yields and got stuck there. If I issue reset command in U-Boot the whole cycle continues. This does not happen upon cold boot.

 

And by the way, I boot Armbian with mainline U-Boot with the 'old' approach: I have deleted 4 partitions of petitboot (i.e. mtd0 through mtd3) can this somehow interfere with U-Boot here?

 

Is this helpful? Should I capture full U-Boot output via serial and post it here later?

 

Thank you for your time!

Link to post
Share on other sites

6 hours ago, Igor said:

Try switching to nightly builds and update bootloader there. Both in armbian-config.

I did that and it shows the same behaviour.

 

15 minutes ago, sulfum said:

I face similar problem with HC4 (not much difference with C4 as Igor pointed out).

There actually is a vital difference in the boot process as I've just noticed.

 

The Odroid HC4 has an internal 16MB flash as seen in the bottom of the block diagram

 

odroid-hc4_block.png

 

While the C4 lacks that flash and replaces that with the eMMC slot.

 

c4_blockdiagram_rev1.0.png

 

https://wiki.odroid.com/odroid-hc4/software/boot_sequence shos that the HC4 has Odroid's Petitboot living on the flash.

 

I'm not quite sure why you are only seeing U-Boot logs on your HC4 (does armbian somehow replace petitboot on the spi?)

 

So well, for now I'm not too sure what to make out of all of this, I can certainly try hooking up to serial and see if I can trace any kind of U-boot activity, but I don't understand why the stock image reboots fine all the time but running apt upgrade or switching to nightly (which contains a kernel update) messes things up so much.

 

 

17 minutes ago, sulfum said:

Card did not respond to voltage select: -110

 

As your log shows U-boot has successfully started up but it can't start the SD card. This is most likely a problem with the SD card not reacting well to a soft reboot, using wrong voltage levels and this could be fixable with the use of a different SD card or 

what's also been on my mind. To try and see wether theres any kind of SoC function or MMC kernel function to properly reset the SD-card before rebooting.

 

Another thing to add would be that ArchDroid has rebooted fine for me so it might be worth to consult with their devs about the best way to handle rebooting Amlogic SoCs.

 

 

Link to post
Share on other sites

So after a bit of googling I've came across this: 

 

https://lists.denx.de/pipermail/u-boot/2020-December/435196.html

 

Apparently the reboot issue has been documented back in 2020, and there was a simple DTS patch developed and merged into linux.

I could confirm that my C4 is running the fixed dts with GPIO_OPEN_DRAIN set on the tflash_vdd regulator.

 

However I obviously still experience the issue. 

 

What I want to try is applying some of the changes to the armbian kernel as Hardkernel did in theirs to fix the reboot problem (https://github.com/hardkernel/linux/commit/84628497332a5cd2154c92436ec86fad900fe0af).

I don't know what your stance on doing hacky patches like this is, but I still want to try it.

 

Unfortunately I can't build a kernel. It errors out even though I'm building without making any changes at all. I'm building in a VM on latest Ubuntu Hirsute server. I don't see an error, probably because it gets lots in all of the parallel make threads running.

image.png.284bdd59ef211dda96c4661fb61c7dc6.png

Link to post
Share on other sites

I can't be 100% sure, but I believe I saw a pull request for this some time ago where some one removed some things related to the reboot issue. I haven't scanned through the patch set as of late, but in my testing the following is needed.

 

Need to revert:

drivers/gpu/drm/meson/meson_drv.c

https://github.com/pyavitz/debian-image-builder/blob/feature/patches/amlogic/odroid/002-linux-odroid-patch-set.patch#L524

https://github.com/pyavitz/debian-image-builder/blob/feature/patches/amlogic/odroid/002-linux-odroid-patch-set.patch#L542

 

Add Odroid reboot:

https://github.com/pyavitz/debian-image-builder/blob/feature/patches/amlogic/odroid/002-linux-odroid-patch-set.patch#L2323

https://github.com/pyavitz/debian-image-builder/blob/feature/patches/amlogic/odroid/002-linux-odroid-patch-set.patch#L2533

 

If you review the pull request, you can see where the revert and odroid power reset patch was removed: https://github.com/armbian/build/pull/3154/files

 

As for mainline u-boot the only thing of real importance is the following revert: https://github.com/armbian/build/pull/3154/files#diff-65100acf19e202ac3f3980da554c205752ea3c67d08fc4a3b445d4397189d12fR36

 

With out it, the boards "especially the N2/+ will kernel panic, as the reserved memory "CMA pool won't be set correctly".  So the patch forces the boards to be marked nomap, hence populated after uboot hands off to the kernel.

 

Link to post
Share on other sites

Hi everyone,

 

I did some testing in the past days (no UART boot logs though):

 

  1. Deelan suggested that the boot problem may stem from uSD card due to the voltage error message. Hence, I have made two things:
    • Made an image of the current Armbian Fossa install (which is btw on a Kingston 32 GB uSD) and wrote it to a Samsung 32 GB uSD. Behavior did not change.
  2. I made a fresh install of Armbian Focal and I chose for an older version 21.08.1: this release from 26-Aug-2021. In short, it, of course, booted up without any issues. I could soft reboot it several time without any issues... the I ran apt update && apt upgrade. The system was brought to the current state (i.e. 21.08.6). The system rebooted ad came online just fine after system/kernel update. All following attempts to reboot the system led to the dark silence. For the note, I have tested it with and without HDDs just in case. The result was the same.
  3. For the sake of testing I installed Armbian Bullseye 21.08.1 on the Samsung uSD. The result was the same as for fresh install of Armbian Fossa: 21.08.1 booted up and I soft rebooted it about 10 times with no issues, when the system was updated to 21.08.6, it came online from a soft reboot the first time after update, any follow up attempt of a soft reboot led to dark silence and the system could be brought back online only by unplugging the system.

In summary, it looks like Cornelius is right and some of the patches, which were submitted and accepted in the meantime, rendered soft boot of the board/amlogic SoC impossible. I guess I have to cope with it as it is now and shutdown/cold boot the system after each kernel update. Hope that someone with knowledge of this business will get down to the core of the problems (e.g. revert some of the late changes).

 

If there is anything that I can test further.

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
 Share

2 2