Odroid HC1 does not boot with kernel 5.4.6

[jshc1]

My HC1 after changing from 4.14.157 to 5.4.6 using armbian-config is not able to get up ... zero response, no boot
Does HC1 work with kernel 5 and boot, can anyone confirm?

[Igor]

3 minutes ago, jshc1 said:

My HC1 after changing from 4.14.157 to 5.4.6 using armbian-config is not able to get up ... zero response, no boot

It must boot, but this change was not tested many times. Perhaps you need to power cycle, worse case installation is broken somehow - boot scripts were not successfully installed.

[jshc1]

This situation has been around for quite a long time with 5.x and HC1.
Using armbian-config theoretically the installation is the same as in the case of 4.14.x
At the end of the installation, automatic reboot is made and the HC1 does not get up again. Hard power on / off also does nothing. The blue idode will light up for a moment and afterwards go off and nothing happens as if something was wrong with the boot. I have performed the installation 5.x a dozen times always the same so a random error during installation I rather rule out. What's worse, this behavior has always been with 5.x, earlier I just treated it as an error with the early version 5.x but now with 5.4.6 it's a bit too far to ignore it.

Any ideas?

[Igor]

57 minutes ago, jshc1 said:

Any ideas?

Check if everything is in place:

- dtb exits and its in correct location

- UUID is correct
- did you only boot from SD but have root on SSD. Did you leave /boot mounted before upgrading

 

I assume you don't have serial console? How can I replicate this - which image to start, what to do to make this happen?

[jshc1]

16 hours ago, Igor said:

Check if everything is in place:

- dtb exits and its in correct location

- UUID is correct
- did you only boot from SD but have root on SSD. Did you leave /boot mounted before upgrading

 

I assume you don't have serial console? How can I replicate this - which image to start, what to do to make this happen?

Everything looks similar between 5.4 and 4.14 ...

Everything is on SD. Nothing modified.
4.14 works normally without problems I do uptime 20-30 days. Only there is always this problem with boot at 5.x
As if something was wrong with armbian-config and the installation procedure or with the 5.x kernel for HC1.

I log in. I run armbian-config and System / Other / I change the kernel ... the typical 5.4 installation procedure begins and the old kernel is removed, after which the ssh connection is broken and after a while it gets reboot and HC1 does not get up anymore. Only the blue LED is lit for a while as if it was looking for a boot and goes off, nothing happens. Red and green behave normally.

dtb exists and points to the right place.

I did the test, restored working 4.14.157 boot partition from backup and manually copied 5.4.6 and changed the links ... the effect is the same no boot and just change the links back to 4.14.157 and it is boot.
It seems that specifically with 5.x for hc there is something wrong ...

UUID is also ok.

Unfortunately, I do not have a cable that would fit the odroid, I only have a regular uart.

[jshc1]

I can send you the whole boot partition if it helps...

[Igor]

3 hours ago, jshc1 said:

I can send you the whole boot partition if it helps...

 

It helps best if you do something you know but its project related, not necessarily this problem - perhaps solve some easy script bug or help with developing and maintaining https://github.com/armbian/config or diagnostics tools https://github.com/armbian/autotests? Its in our interest that images works, but when we have to deal with everything but the problem ...

My HC1 with freshly current (not DEV) build 5.4.y kernel boots but I noticed slight stability issue - sometimes restarts at loading kernel, but it always come up. Perhaps kernel is just clocked too optimistically? Out of my quick stress test one could conclude that the problem only appears when the board / PSU are still cold  

Spoiler

[ o.k. ] Try if we can login and send CTRL C [ 10.0.10.118 ]
[ o.k. ] Host found [ 08:16:26 ]
[ o.k. ] Stressing [1] for 100s [ 10.0.10.118 ]
[ warn ] Rebooting in 5 seconds [ 10.0.10.118 ]
Connection to 10.0.10.118 closed by remote host.
[ warn ] Ping failed [ 08:18:42 ]
[ warn ] Ping failed [ 08:18:52 ]
[ warn ] Ping failed [ 08:19:02 ]
[ warn ] Ping failed [ 08:19:12 ]
[ warn ] Ping failed [ 08:19:15 ]
[ o.k. ] Host found [ 08:19:16 ]
[ o.k. ] Stressing [2] for 100s [ 10.0.10.118 ]
[ warn ] Rebooting in 5 seconds [ 10.0.10.118 ]
Connection to 10.0.10.118 closed by remote host.
[ warn ] Ping failed [ 08:21:33 ]
[ o.k. ] Host found [ 08:21:33 ]
[ o.k. ] Stressing [3] for 100s [ 10.0.10.118 ]
[ warn ] Rebooting in 5 seconds [ 10.0.10.118 ]
Connection to 10.0.10.118 closed by remote host.
[ warn ] Ping failed [ 08:23:49 ]
[ o.k. ] Host found [ 08:23:49 ]
[ o.k. ] Stressing [4] for 100s [ 10.0.10.118 ]
[ warn ] Rebooting in 5 seconds [ 10.0.10.118 ]
Connection to 10.0.10.118 closed by remote host.
[ warn ] Ping failed [ 08:26:06 ]
[ warn ] Ping failed [ 08:26:16 ]
[ o.k. ] Host found [ 08:26:16 ]
[ o.k. ] Stressing [5] for 100s [ 10.0.10.118 ]
[ warn ] Rebooting in 5 seconds [ 10.0.10.118 ]
Connection to 10.0.10.118 closed by remote host.
[ warn ] Ping failed [ 08:28:32 ]
[ o.k. ] Host found [ 08:28:32 ]
[ o.k. ] Stressing [6] for 100s [ 10.0.10.118 ]
[ warn ] Rebooting in 5 seconds [ 10.0.10.118 ]
Connection to 10.0.10.118 closed by remote host.
[ warn ] Ping failed [ 08:30:49 ]
[ warn ] Ping failed [ 08:30:59 ]
[ o.k. ] Host found [ 08:30:59 ]
[ o.k. ] Stressing [7] for 100s [ 10.0.10.118 ]
[ warn ] Rebooting in 5 seconds [ 10.0.10.118 ]
Connection to 10.0.10.118 closed by remote host.
[ warn ] Ping failed [ 08:33:15 ]
[ o.k. ] Host found [ 08:33:15 ]
[ o.k. ] Stressing [8] for 100s [ 10.0.10.118 ]
[ warn ] Rebooting in 5 seconds [ 10.0.10.118 ]
Connection to 10.0.10.118 closed by remote host.
[ warn ] Ping failed [ 08:35:32 ]
[ warn ] Ping failed [ 08:35:42 ]
[ o.k. ] Host found [ 08:35:42 ]
[ o.k. ] Stressing [9] for 100s [ 10.0.10.118 ]
[ warn ] Rebooting in 5 seconds [ 10.0.10.118 ]
Connection to 10.0.10.118 closed by remote host.

 

Boot log: http://ix.io/26ge

Regarding kernel switching ... that features might need some hardening and DEV kernels are as is. They work or not.
 

[jshc1]

2 hours ago, Igor said:

My HC1 with freshly current (not DEV) build 5.4.y kernel boots but I noticed slight stability issue - sometimes restarts at loading kernel, but it always come up. Perhaps kernel is just clocked too optimistically? Out of my quick stress test one could conclude that the problem only appears when the board / PSU are still cold  

Strange, since your hc is booting hmmm ... If I find the time, I'll try a clean installation.
Whether it's warm or cold I can't see the difference. I have a maximum clock frequency of 2Ghz and 1.5Ghz, boot problems or stability I didn't have until 5.x
The same thing happened at 5.0 but then I thought it was the initial problem of underdevelopment and I hoped that it would be solved over time but remained with me to this day.
But if you say that there are problems with stability, then probably I will stay for now at 4.14. Any plans for 4.14.161 in the near future? Because .157 is a bit old ...

Currently, arm-conf only shows me DEV when it comes to 5.x

 

 

PS
in the attachment both boot partitions ... Working correctly 4.14.157 and problematic with 5.4.6

Kernels.4_14.157-5.4.6.tar

[Igor]

48 minutes ago, jshc1 said:

Any plans for 4.14.161 in the near future? Because .157 is a bit old ...

If you find out why this is happening, then yes: https://github.com/armbian/build/issues/1681

[nihilista]

Hey, just a little question: On my other boards i can update u-boot within armbian-config, but not on odroid HC1 (system on ssd). Is that normal behavior or did i miss something?

Not a big thing as i can do that with dd...btw. Maybe i have to test with 5.4.y kernel, don't know i need newer u-boot...

[Igor]

There are no changes in u-boot since year(s). I was just about to check how far we can get with a pure mainline.

 

1 hour ago, nihilista said:

On my other boards i can update u-boot within armbian-config, but not on odroid HC1

It's a small bug which is waiting for PR. I have no time and it doesn't bothers me.

[Igor]

I found last known working configuration ... it something build script related, change from old 5.98 -> v19.11. 

Compile with:

LIB_TAG="v19.08" = stable (default branch)

LIB_TAG="master" = unstable

 

Moving on later or tomorrow.

[jshc1]

I saw this behavior on one of 5.4.x
I changed to nightly and I probably used current and I had the same symptom as TK, the blue diode is lit all the time and no boot, on DEV, however, as I wrote above, the diode stops lighting and there is no boot.
Of course, I'm talking about an update, maybe with a clean installation something is better and works just like yours

[Igor]

Fixed https://github.com/armbian/build/pull/1716

[nihilista]

Thx, good to know...and yes, its just a small bug ;-)

[jshc1]

So... .161 works ok.

I checked again 5.4 and still no boot. I checked the clean Buster 5.4 installation and the same, 30 seconds heartbeat, and darkness. There is no ssh, hc dead ...
I checked clean 4.14 stretch installations and change to 5.4 again darkness.
My HC1 just has 5.x allergies I don't know how your HC1 works with 5.x

[jshc1]

On 1/3/2020 at 1:53 PM, nihilista said:

Maybe i have to test with 5.4.y kernel

Good luck with 5.4 My HC1 absolutely does not want to boot with 5.4

 

 

 

[nihilista]

Mmm...maybe i just wait a bit...but normally i have a backup before testing ;-)

[nihilista]

I can confirm, no boot with latest 5.4.7 kernel....

[Igor]

5 hours ago, nihilista said:

I can confirm, no boot with latest 5.4.7 kernel....

Me too. I had few successful boots but now it also crashed. I will try few things to see if I can fix this.
 

Spoiler

    2.372461] Internal error: : 1406 [#1] PREEMPT SMP ARM
[    2.376209] Modules linked in:
[    2.379243] CPU: 6 PID: 151 Comm: kworker/6:1 Not tainted 5.4.8-odroidxu4 #19.11.7
[    2.386779] Hardware name: SAMSUNG EXYNOS (Flattened Device Tree)
[    2.392858] Workqueue: events deferred_probe_work_func
[    2.397965] PC is at kbase_reg_read+0x20/0x80
[    2.402293] LR is at kbase_backend_gpuprops_get+0x1c/0x204
[    2.407751] pc : [<c0656d04>]    lr : [<c0656fd8>]    psr: 600c0013
[    2.413990] sp : c204dcd0  ip : 00000000  fp : ee06e760
[    2.419188] r10: ee25dc40  r9 : c400939c  r8 : c22a2ff0
[    2.424388] r7 : c204dd04  r6 : 00000000  r5 : 00000000  r4 : 00000000
[    2.430887] r3 : c22a3000  r2 : 00000000  r1 : 00000000  r0 : c22a2000
[    2.437388] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[    2.444492] Control: 10c5387d  Table: 422c406a  DAC: 00000051
[    2.450212] Process kworker/6:1 (pid: 151, stack limit = 0x(ptrval))
[    2.456538] Stack: (0xc204dcd0 to 0xc204e000)
[    2.460870] dcc0:                                     c22a2ff0 c22a2000 c204dd3c c0656fd8
[    2.469017] dce0: c22a2ff0 c0e04f48 00000000 c22a2000 c22a2ff0 c063b10c ee23f2b4 c02978f8
[    2.477162] dd00: ee23f2c0 00002000 f0800000 c029aaa0 00000000 ffffe000 ff800000 c0ee6f28
[    2.485307] dd20: 00000000 ee23f300 c0e04f48 00000000 c0ea3a80 00000000 c0ea3a80 00000000
[    2.493453] dd40: 00000001 0000065f 38e38e39 c0007c78 f1c77000 f1c76fff ffe00000 ee25dd40
[    2.501598] dd60: fffff000 ee25dc40 ee06e760 00000001 ee25dd00 00000001 ffffffff 00002cc2
[    2.509744] dd80: 00000cc0 ee25dd40 ee06e760 c0297b38 00000001 ee095cb0 ff800000 ffffffff
[    2.517889] dda0: 00000cc0 da013c4a 23c34600 c22a2000 00000000 00000000 c22a2000 c22a2ff0
[    2.526035] ddc0: c400939c ee25dc40 ee06e760 c063b2e4 ee06e760 c0674ec0 c22a2000 c22a2000
[    2.534180] dde0: 00000000 00000000 ee095c00 c22a2000 c400939c ee25dc40 ee06e760 c0656eac
[    2.542326] de00: c22a3000 c22a2000 00000000 c064a6cc 00000000 c40093d8 00000000 ee095c10
[    2.550471] de20: 00000000 ee095c10 00000000 c0e6e360 00000000 c0e6e360 00000004 00000000
[    2.558617] de40: eedd6c80 c066a5b4 c0f00138 ee095c10 c0f0013c c0668590 ee095c10 c0e6e360
[    2.566762] de60: c06689d8 c0e04f48 00000001 c0ea7950 00000000 c0668820 c0e6e360 c204ded4
[    2.574908] de80: ee095c10 00000000 c204ded4 c06689d8 c0e04f48 00000001 c0ea7950 00000000
[    2.583053] dea0: eedd6c80 c066697c c0ea7950 eeac606c c208beb8 da013c4a ee095c10 ee095c10
[    2.591198] dec0: c0e04f48 ee095c54 c0e6ebd0 c06682e8 ee8f2000 ee095c10 00000001 da013c4a
[    2.599344] dee0: ee095c10 ee095c10 c0e6ee40 c0e6ebd0 00000000 c0667660 ee095c10 c0e6ebbc
[    2.607489] df00: c0e6ebbc c0667afc c0e6ebf4 ee16c000 eedd6c80 eedd9e00 00000000 c0140acc
[    2.615635] df20: ffffe000 eedd6c80 c0e03d00 ee16c000 eedd6c80 ee16c014 c0e03d00 eedd6c98
[    2.623781] df40: ffffe000 00000008 eedd6c80 c0141110 ffffe000 c0ea7256 c0b72250 00000000
[    2.631926] df60: ffffe000 ee3a4500 ee170080 00000000 c204c000 ee16c000 c0140e7c ee98fea4
[    2.640072] df80: ee3a451c c0147ab0 00000000 ee170080 c0147960 00000000 00000000 00000000
[    2.648217] dfa0: 00000000 00000000 00000000 c01010e8 00000000 00000000 00000000 00000000
[    2.656363] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    2.664508] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
[    2.672662] [<c0656d04>] (kbase_reg_read) from [<c0656fd8>] (kbase_backend_gpuprops_get+0x1c/0x204)
[    2.681671] [<c0656fd8>] (kbase_backend_gpuprops_get) from [<c063b10c>] (kbase_gpuprops_get_props+0x28/0x134)
[    2.691548] [<c063b10c>] (kbase_gpuprops_get_props) from [<c063b2e4>] (kbase_gpuprops_set+0x1c/0x28c)
[    2.700732] [<c063b2e4>] (kbase_gpuprops_set) from [<c0656eac>] (kbase_backend_early_init+0x3c/0x84)
[    2.709834] [<c0656eac>] (kbase_backend_early_init) from [<c064a6cc>] (kbase_platform_device_probe+0x2a8/0xc08)
[    2.719886] [<c064a6cc>] (kbase_platform_device_probe) from [<c066a5b4>] (platform_drv_probe+0x48/0x98)
[    2.729241] [<c066a5b4>] (platform_drv_probe) from [<c0668590>] (really_probe+0x234/0x34c)
[    2.737472] [<c0668590>] (really_probe) from [<c0668820>] (driver_probe_device+0x60/0x168)
[    2.745705] [<c0668820>] (driver_probe_device) from [<c066697c>] (bus_for_each_drv+0x58/0xb8)
[    2.754196] [<c066697c>] (bus_for_each_drv) from [<c06682e8>] (__device_attach+0xd0/0x13c)
[    2.762428] [<c06682e8>] (__device_attach) from [<c0667660>] (bus_probe_device+0x84/0x8c)
[    2.770573] [<c0667660>] (bus_probe_device) from [<c0667afc>] (deferred_probe_work_func+0x64/0x90)
[    2.779503] [<c0667afc>] (deferred_probe_work_func) from [<c0140acc>] (process_one_work+0x218/0x5c8)
[    2.788600] [<c0140acc>] (process_one_work) from [<c0141110>] (worker_thread+0x294/0x59c)
[    2.796744] [<c0141110>] (worker_thread) from [<c0147ab0>] (kthread+0x150/0x154)
[    2.804110] [<c0147ab0>] (kthread) from [<c01010e8>] (ret_from_fork+0x14/0x2c)
[    2.811299] Exception stack(0xc204dfb0 to 0xc204dff8)
[    2.816326] dfa0:                                     00000000 00000000 00000000 00000000
[    2.824472] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    2.832617] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000
[    2.839207] Code: e0833001 e5935000 f57ff04f e2803a01 (e5d33590)
[    2.845272] ---[ end trace 29569194585851c3 ]---
[    2.948999] mmc1: new ultra high speed SDR104 SDHC card at address 0001
[    2.955040] mmcblk1: mmc1:0001 EB1QT 29.8 GiB
[    2.960646]  mmcblk1: p1

 

 

[nihilista]

Ah, will be fine if that get fixed. No really need for using 5.x kernel, but as it fully supports my dual dvb tuner (4.14 did not) it would be fine to have a working 5.x  kernel. Many thx for all the good work !

[Igor]

16 minutes ago, nihilista said:

Ah, will be fine if that get fixed. No really need for using 5.x kernel, but as it fully supports my dual dvb tuner (4.14 did not) it would be fine to have a working 5.x  kernel. Many thx for all the good work !

Sadly this was not a solution  No quick fix but serious debugging ... when possible.

[Igor]

20 hours ago, jshc1 said:

I don't know how your HC1 works with 5.x

Actually I only does sometimes. But when it does, it works, doesn't hang on numerous boots ... can't reproduce it easy, but I just did. No 5.4.y in production for now.

[jshc1]

12 hours ago, Igor said:

No 5.4.y in production for now.

Can you tell me such a thing ... where is 162 you wrote about, because arm-conf only gives me a maximum of 161. I'm probably blind.

It would also be good to delete 5.4 for now, so that no one accidentally gets into a problem and complains about it as with TK recently

[Igor]

1 hour ago, jshc1 said:

where is 162 you wrote about

Self-build.

 

1 hour ago, jshc1 said:

It would also be good to delete 5.4 for now, so that no one accidentally gets into a problem and complains about it as with TK recently

That was a different case. TK had issues with a stable 4.14.y kernel build. 5.4.y is noted as testing/experimental.

[unixispower]

I can confirm a similar issue using the newest Buster minimal image (Armbian_19.11.7_Odroidxu4_buster_current_5.4.3_minimal) on the HC2. I am able to boot into a freshly written image, but cannot reboot successfully. When running `reboot now` or rebooting when prompted by `armbian-config` selections I hear the installed HDD spin down, see the heartbeat LED go solid, then start blinking as if a reboot happened -- but the system doesn't seem to come back up. The attached HDD never spins back up and I cannot connect via SSH until I pull the power and replug. After needing to manually replug twice (I have tested this a few times now) the HC2 fails to boot altogether with a solid blue heartbeat LED. I have both the HC1 and HC2, but I won't have access to my HC1 until tonight. I would be happy to provide any logs or try to troubleshoot with some direction.

[Igor]

5 hours ago, unixispower said:

I would be happy to provide any logs or try to troubleshoot with some direction.

We have logs, "just" no time / resources to debug since we have to deal with numerous other things. Help there is better then providing us what we already have  We need to move on with this: https://github.com/armbian/autotests just there is nobody who will lead the project - on a long run it will help us with the problem we have right now. Then https://github.com/armbian/config receive virtually no help with maintaining from community which means its still a security problematic but highly useful and used tool. When there will be more help on general project needs, there will be more time for things like this.

Use 4.14.y and try if you can do something elsewhere.

[cyagon]

Can also confirm the problem with the newest 5.4x Kernel. The prior 5.4x kernel worked without problems for me (im using it because of the newest btrfs features).
It boots up normally, SSH is up for a few seconds until the connection fails and my HC1 and becomes unreachable.

It also has current problems when you apply the HC1 DTB via armbian-config, then it will not boot either.
 

[nihilista]

Anyone tested new kernel 5.4.19 ? Changelog says "Odroid XU4 current with kernel 5.4.y seems unstable", but what does than mean? Till now it wasn't unstable, it didn't work at all ;-)

And why did same changelog say "Odroid XU4 legacy kernel images instability" ? Is 5.4.x now more stable (because it "seems") than 4.14.x?

I'm bit confused... Btw, its no problem to make an image/backup and test and i know, developers couldnt test all....but little information will be fine :-)

[Igor]

6 hours ago, nihilista said:

Odroid XU4 legacy kernel images instability

 

Default kernel is more or less stock kernel. If Hardkernel breaks it, its broken ... We had troubles elsewhere - a compiler version definition fell out when we move to new branches: default -> legacy, next -> current. Since we didn't compile the kernel with some particular (old) compiler, kernel was unstable. I need to kill a day or two before finding out what was the problem ...

Kernel 5.4.y boots for me on Odroid XU4 with eMMC or from SD card. Also on HC1 ... but people report troubles and we were not able to sucesfully fix them yet. Which is why we label them as "Odroid XU4 current with kernel 5.4.y seems unstable"

[nihilista]

OK, thx. So i will test latest 5.4 kernel...maybe that works with latest. Before it won't boot at all ;-)