Odroid HC4 not recognising/partially recognising SSDs

[leafybasil]

I recently installed the latest precompiled Bullseye image on my Odroid HC4 and plugged in two Samsung 870 QVO 8TB SSDs.  After configuring the system I noticed that they were not present, checking output from `/var/log/syslog` I can see the "SATA Llink down" messages for both channels.  After several reboots I tried seating them after booting, both drives cause some messages about a soft reset failing, then eventually within 30-60 seconds are recognised until I reboot.

 

I then one Crucial MX500 4TB SSDs which was recognised at first, after some reboots it would no longer be present, or strangely, be present with a much lower capacity (1TB).  When seating the drive after boot, it instantly becomes available/recognised.

 

I went back to trying an old 250GB Kingston SSD (SA400S37240G) which is detected every single boot, just fine.  The same is also true for some old Seagate/Western Digital HDDs.

 

I eventually tried some other distros, namely DietPI which behaves exactly the same.  I eventually tried all available precompiled Armbian 5.x and 4.x images but they all behaved the same way.

 

Finally I tried the Hardkernel Odroid Ubuntu image, and this seems to work fine, for all of the drives listed above.  Noting it's a 4.9 kernel, I assumed I might get the same functionality from Armbian if I compiled it myself, however the build system seems specifically geared toward current/edge, judging by the configs, this is usually 6.x and 5.x.  After some fiddling I was able to start the process of building a 4.9 image, but due to other expectations the compilation fails much later on.

 

I've seen posts here and elsewhere online about faulty NCQ for the Samsung drives, apparently fixed with my firmware version and also blacklisted by the driver anyway.  I've seen posts about the Crucial MX500 and other drives appearing and disappearing between boots.  Those people had access to even earlier kernels and managed to downgrade to keep things working, but they are no longer available.  So I'm at a loss.  Any advice would be appreciated.

 

armbianmonitor: https://paste.armbian.com/gexecuzuja

[Amissogen]

hy leafybasil,

 

I think we are all in the same situation. In my case, I have freeze kernel at  Armbian 22.05.3 Bullseye with Linux 5.10.123-meson64 . Waiting new release for see if they can "repair" this problem.

 

 

[leafybasil]

@amissogen glad you posted, I thought I had tried all the 5.x Armbian images, but it turns out I did not, and I now have one that works thanks to you.

 

Issues around this stuff go back a year or two, so it's not hugely likely it would get fixed unless someone with particular knowledge tries.  I will comment on the hardkernel/odroid forums too and add another voice.  I have tried looking into this myself, but there are too many moving pieces and I know nothing about any of them, so it's challenging to say the least.

[charly_53]

for me the latest working image was Armbian 22.08.1 with kernel 5.10.139.

Did someone test the new 23.02 image wiith kernel 6.1?

[Amissogen]

I'm afraid to try it I have all the home automation on this server

[leafybasil]

At the time of posting I had tested all released images, and used the framework to build current and edge.  None of them worked.  I also used petitboot to install Debian in case that had fixes, but it did not.

I have picked up a spare SD card so I can periodically test new releases.  It's just a media NAS for me so as long as my drives remain safe I can test various images.

[charly_53]

Thanks for the info. I'll stick with kernel 5.10.139 for now

[Amissogen]

Tried with a kingston kc600. No luck. No recognized neither...

 

charly_53, how can I install armbian 23.02 with kernel freze at 5.10.139? I can only freeze all, not only kernel

[charly_53]

- install or downgrade to armbian 22.08.1 (it comes with kernel 5.10.139)

- set the following packages on hold :

   linux-dtb-current-meson64
   linux-image-current-meson64

you can do it with the following  command: sudo apt-mark hold <package-name>

then you can upgrade to the current armbian version

 

[poulsen84]

Anyone  tried with the new "bookworm"? Any change?

[Yan Pashkovsky]

Hi!

 

Yep, I've tried bookworm, both 6.1 and 6.2 kernels, in both cases my ssd Samsung 870 is only seen after hotplug. Also 2nd time I was switching kernel using armbian-config - the bootloader got broken, I gave up and installed image from hardkernel with 4.9 kernel. 

[ChrisO]

Sorry, it will be quite long 🙂

I have three HC4 devices:

odroid1:
Linux odroid1 5.10.57-meson64 #21.05.8 SMP PREEMPT Mon Aug 9 12:44:31 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
ata-Samsung_SSD_870_QVO_1TB
ata-Samsung_SSD_870_QVO_1TB

odroid2:
Linux odroid2 5.19.17-meson64 #22.08.7 SMP PREEMPT Thu Oct 27 18:34:08 CEST 2022 aarch64 aarch64 aarch64 GNU/Linux
ata-Samsung_SSD_870_QVO_1TB
ata-Samsung_SSD_870_QVO_1TB

They run the kernel which recognized both discs AND there was ZFS support when I installed them.

odroid3, currently used for some experimenting:
Linux odroid3 6.1.63-current-meson64 #1 SMP PREEMPT Mon Nov 20 10:52:19 UTC 2023 aarch64 GNU/Linux

odroid3 recognises those disks:
ata-Samsung_SSD_840_EVO_120GB_S1D5NEAD710633R
ata-Samsung_SSD_750_EVO_250GB_S33SNWAH687472R

I replaced the ata-Samsung_SSD_840_EVO_120GB_S1D5NEAD710633R
by another
Samsung SSD_860_EVO_250GB
It is not recognized, and in the /var/log/kern.log I see this:
odroid3 kernel: [    2.692110] ata1: SATA max UDMA/133 abar m512@0xfc700000 port 0xfc700100 irq 27
odroid3 kernel: [    2.692124] ata2: SATA max UDMA/133 abar m512@0xfc700000 port 0xfc700180 irq 27
odroid3 kernel: [    3.005134] ata1: SATA link down (SStatus 0 SControl 300)
odroid3 kernel: [    3.478833] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)

Disconnected the ata-Samsung_SSD_750_EVO_250GB_S33SNWAH687472R
and rebooted
NO ssd!
 /var/log/kern.log:
odroid3 kernel: [    2.901128] ata1: SATA link down (SStatus 0 SControl 300)
odroid3 kernel: [    3.213129] ata2: SATA link down (SStatus 0 SControl 300)

Tried both SATA slots, same results.
Tried another Samsung SSD_860_EVO_250GB, same results, no SSD. The same "SATA link down" message.

I bought a Crucial 1TB just to change manufacturer:
ata-CT1000MX500SSD1_2249E68F3955
Works in both SATA slots.
Rebooted again with
ata-Samsung_SSD_750_EVO_250GB_S33SNWAH687472R
and
ata-CT1000MX500SSD1_2249E68F3955

The last one in not there, the same messages in /var/log/kern.log.
Two days later I rebooted it to try libata.force=noncqtrim kernel parameter and both disks are there.
Removed libata.force=noncqtrim from armbianEnv.txt rebooted again, both disks are there.
Then shutdown, power off/on and boot, no CT1000MX500SSD1.

I tried the SD-card from odroid3 in the odroid2 and there both Samsung_SSD_870_QVO_1TB are there.
Rebooted few times, just in case 😉 they were always there.
Then disconnecte those two disks and connected the CT1000MX500SSD one: no disk after reboot.

Put the SD-card in odroid1, only one disk, Samsung_SSD_870_QVO_1TB, is there.
Replaced one of Samsung_SSD_870_QVO_1TB by CT1000MX500SSD, both are not there:
kern.log:
odroid3 kernel: [    2.940933] ata1: SATA link down (SStatus 0 SControl 300)
odroid3 kernel: [    3.252903] ata2: SATA link down (SStatus 0 SControl 300)

I like the HC4 (when it works 😉 so I ordered another one hoping it will not have any problems.
Got it today (I must be one of the best HK's custommers: H3, H3+, 4 x HC4 🙂

booted with a EVO 860 250GB SSD:
uname -a
Linux odroid4 6.1.63-current-meson64 #1 SMP PREEMPT Mon Nov 20 10:52:19 UTC 2023 aarch64 GNU/Linux

No SSD there. Re-plug it (I know the riscs) and
lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda           8:0    0 232.9G  0 disk 

Rebooted.
lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda           8:0    0 232.9G  0 disk 

Rebooted again and the SSD is there. Aha, rebooted, not powered off/on.

init 0
remove power
10 seconds
connect power
The SSD is there.

init 0
Connected the CT1000MX500SSD, boot.
lsblk -> no SSD.

Somehow it doesn't make any sense, I don't see any regularity in all this.

Did again hotplug
~$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda           8:0    0 931.5G  0 disk 

Some time ago I already reported the problem with SSD's not being recognized.
And the strange thing is, some of the events are quite random. The SSD is there, than not there. 

Now I would say: HC4 and kernels newer than 5.19.17-meson64 don't want to cooperate correctly.
Who is the culprit? I don't know, don't have the necessary skills to debug it.
However, I would love to provide all infos needed which HK needs to analyze the problem. Screenshots, etc.

BTW, rebooted odroid4 again, no SSD.

Regards,
Chris