NanoPi M4V2 + SataHAT + OMV - pcie fail to train after reboot

[markon]

Armbianmonitor:

http://ix.io/2HSE

Few days ago i saw an issue after i added rule to auto-reboot device once a day, every time it's reboot it's losses connection with sata hat - no more sata controler after typing "lspci" and some nasty output after "dmesg | grep pcie".
I tried reboot manually, when i reboot just after powering on (like up to 2 min), pcie goes back up and there is no issue, but if device stay on for longer and i reboot it then there is pcie training issue and mervell sata controller don't get recognized anymore, further rebooting don't fix the problem, i even tried to connect drives power straight through SATA Hat, but it had changed nothing.
The only fix is to shutdown device instead of reboot, after shutdown and powering it on using power button on hat or board itself (without physically disconnecting power to the board) its going up correctly with pcie visible.

I searched here and there about this issues but i didn't saw anything helpful (some hardware issues related with nvme risers on rockpi forum, but it shouldn't be the case here because without rebooting hardware works great).

Someone saw this issue before or know how to solve it ?

Overall board works great, except of this issue and of crashing on mainline (at least when i tried it few months back, so thats why i'm staying with legacy, someone knows if it's already heve been fixed ?).
 

[canardo]

This does happen here sometimes as well - a few m4v2 not bringing up sata after remote reboot.

We are on latest kernels (legacy only for rk3399) and it does not happen all the time (no OMV, I do not have a log to share immediately).

 

Recycling power always helps, or not rebooting

 

[TRS-80]

2 hours ago, canardo said:

We are on latest kernels (legacy only for rk3399)

 

I don't understand this comment.  Are you on latest (which is, at time of writing, 5.9.y for M4 V2) or legacy (which would be 4.4.y at same link)?

 

It's a particularly relevant detail, because I was just last night reading some article where guy built one of these setups, and as of 2020-10-16 he issued an update stating:

 

Quote

Despite the CPU tuning improvements I mentioned in my previous update, I’ve continued to have a few stability issues with Kernel 5.x. After a while, I’ve decided to reinstall Armbian Buster with Kernel 4.4.213-rk3399 (legacy) and it has been smooth sailing ever since.

 

If anyone else has any feedback regarding this (one way or the other) please comment!

Edited December 18, 2020 by TRS-80

[canardo]

On 12/18/2020 at 1:06 PM, TRS-80 said:

 

I don't understand this comment.  Are you on latest (which is, at time of writing, 5.9.y for M4 V2) or legacy (which would be 4.4.y at same link)?

 

It's a particularly relevant detail, because I was just last night reading some article where guy built one of these setups, and as of 2020-10-16 he issued an update stating:

 

 

If anyone else has any feedback regarding this (one way or the other) please comment!

 

Apologies for being unclear - we are on legacy kernels for rk3399 (4.4.x).

Since we update regularly, this should be close to the latest legacy (plain buster & buster-backports).

Edited December 19, 2020 by TRS-80
clean up quotes

[TRS-80]

"Latest legacy" then, it all makes sense now. 

 

Also, in that case, the article I linked does not appear to be relevant...  So, must be something else...

 

EDIT:  I completely forgot until now, I remember reading about V2 have some kernel instability perhaps...  And if that's true it is probably going to take some time and effort to track that down.  In fact if I recall correctly, the recommendation (for NAS use, anyway) was to stick with M4 V1 because of that.

 

One more data point, I don't know if related or not.  @NicoD is a big fan of these, in fact they are his favorite for desktop use-case at the moment.  But he also run legacy kernel, with JMCC (or someone) media script I think.  But even there I read about some problems lately (and then more problems even more recently).  I thought they were worked out though.  Maybe the fixes are only on nightly / dev branch so far?

 

I don't own this device, just passing along whatever I can recall since I read the forums a lot.

 

If anyone have any more feedback or experiment with trying various things to help instability, by all means please share your findings!

Edited December 19, 2020 by TRS-80
add "more problems even more recently"

[markonmoto2]

I also tried to go with mainline earlier this year but as you both mentioned stability issues get me back to legacy ;P i didn't have time to investigate that further, temporary fix is "do not reboot", in case of mainline kernel there is probably still an issue with it because there is still some reports here and there that are no older than 1-2months, and there is hardly any evidence of it resolved, except of various fix by locking clock speeds and changing governor's which in the and usually still ends with instabilities

v1 is more stable, so it even can be hardware level bug in case of v2... the issue persist from early days and there is still no clear resolution for this (i have v2 almost from first days of its availability) especially that rockpi with also ddr4 works without a hitch.

Sent from my Redmi 4X using Tapatalk

[i5Js]

I've installed a dev kernel (Linux 5.10.0-rc7-rockchip64)... with 5.9 I've suffered a few kernel panics in a short period of time. 

[i5Js]

Well. with dev kernel I'm suffering kp... I don't know honestly If it's because the nvme hat or it has a hardware issue.

[i5Js]

I’ve updated the kernel to 5.10.12, and it’s failing and suffering randomly reboots all the time. 

I’ve uploaded the armbianmonitor output to http://ix.it/2OOZ I can’t see which is the problem of theses reboots on the logs.

[canardo]

Hi all - small update from my side: I confirm that using later kernels the remote reboot/training issue seems resolved.

These are nanopim4v2 hosts running either 4.4.x or 5.10.x kernels.

 

At the time of writing for the interested: either

- kernel 5.10.21-rockchip64 #21.02.3 SMP PREEMPT Mon Mar 8

- kernel 4.4.213-rk3399 #2 SMP Mon Mar 8