H6 Famous Reboot problem

[martinayotte]

Hi everyone !

 

I'm starting this new dedicated thread because I've got some new headache about this issue that we tried to figured out since months...

 

I've received my PineH64 last week, and with this board I don't faced the reboot hang behaviour !

 

Today, since I've committed the OPi3 WIP scripts, and facing same reboot hang, I've decided to to try booting PineH64 image on this OPi3 board, but unfortunately reboot hang still occurs ...

 

This brings me to a conclusion that it is not a software issue, but maybe hardware one ... I will dig on that during next few days by comparing PMIC and other between all those boards ...

 

But I remember that @jernej said that Reboot worked on one of his H6 board ...

@jernej can you confirm which board ? Do you have some boards where you still face reboot hang issue ? Can you try to mix image like I did to give clues ?

 

 

 

[lanefu]

Ive got a oneplus how can i help debug?

[martinayotte]

2 minutes ago, lanefu said:

Ive got a oneplus how can i help debug?

I presume you faced the Reboot issue ? ...

Try some other H6 images, even if it missed some hardware, try to stop U-Boot to get into comand prompt , if "reset" hang, the issue is faced ...

What is strange to me is the issue doesn't seems to exist on PineH64, so, maybe hardware issue ...

Digging hardware comparison tomorrow ...

[lanefu]

18 minutes ago, martinayotte said:

I presume you faced the Reboot issue ? ...

Try some other H6 images, even if it missed some hardware, try to stop U-Boot to get into comand prompt , if "reset" hang, the issue is faced ...

yep reboots always failed and had to powercycle.

[jernej]

2 hours ago, martinayotte said:

can you confirm which board ?

I mostly work on pineH64 model A. I also have pre-production samples of model A and B, on which I can test. But since they aren't produced, it may not matter. I still think it something power/frequency related.

[martinayotte]

6 hours ago, jernej said:

I mostly work on pineH64 model A

Ok ! So, you don't any board, such OPi, where you see the issue, right ?

[jernej]

2 minutes ago, martinayotte said:

So, you don't any board, such OPi, where you see the issue, right ?

no (not yet)

 

It would be interesting to know what bin they belong to. If there is difference, e.g. slow/normal/fast, then default settings might not work for all.

[martinayotte]

18 minutes ago, jernej said:

It would be interesting to know what bin they belong to

Do you mean ATF bin ? From my knowledge, all are using bl31.

 

But, as I said, my recent trial to use PineH64 image, which reboot work fine, on an OPi board, which still boot properly, the reboot still doesn't work ...

[jernej]

1 minute ago, martinayotte said:

Do you mean ATF bin ?

No, I actually mean binning. Read here: https://irclog.whitequark.org/linux-sunxi/2019-02-28#24137228;

[martinayotte]

18 minutes ago, jernej said:

No, I actually mean binning. Read here

I'm not familiar with this ... What would be the relation with different H6 based boards ?

 

Also, as an additional symptom that differ between those boards : without have any kernel involved, only the same U-Boot at command line, "reset" command hangs on the OPi but working on PineH64.

 

[jernej]

Just now, martinayotte said:

I'm not familiar with this ... What would be the relation with different H6 based boards ?

Take "bin" as a speed grade. Every chip gets tested in factory and based on electrical characteristics it is put in different bin. According to BSP SID driver, there are 3 bins: slow, normal and fast. BSP DT has actually 3 different dvfs tables, one for each bin. As I said, it would be interesting to know if H6 chips between OrangePis and PineH64s comes from different bins. If there is no rule, e.g. same board model and different bins, then something else is wrong, probably difference in HW design.

[martinayotte]

1 hour ago, jernej said:

it would be interesting to know if H6 chips between OrangePis and PineH64s comes from different bins.

Am I right by looking at sunxi-sid.c code that "sunxi_soc_bin" register location would be 0x0300621c ?

If Yes, then "devmem2" is reporting the same 0x00000020 value on all the H6 ...

[jernej]

2 hours ago, martinayotte said:

Am I right by looking at sunxi-sid.c code that "sunxi_soc_bin" register location would be 0x0300621c ?

It seems so.

[dolphs]

hmm all this seems similar to what I experience on my Orange Pi One Plus board once I reboot.

The board gets stuck ( 5.0.2 dev image ) and powering on device manually is the only way to get it back to business.

 

I was hoping as a fall back WoL could be used,

but alas as currently it lists "  (D)isabled "

 

# ethtool eth0 |grep Wake-on
        Supports Wake-on: d
        Wake-on: d
 

Even after installing " apt-get -t stretch-backports install firmware-realtek " it does not show anything else as " d " .

Then I discovered RTL8211 is not included ( yet ) in to this package, so back to square one again ...

How to get this board properly rebooted ( aka telinit 6/ shutdown -r now , etc etc )

 

@martinayotte - Perhaps you found anything useful after comparing the OPi's and PINE H64 hw?

BTW can I update to 5.0.4 kernel or should I build a new image in my dev environment?

 

thanks

 

 

[dreddit]

Hello martinayotte the reboot works on the OP3 image from the Orange pi can the script/blobs work for Kernel 5.0,  the sys_config is at git hub here https://github.com/orangepi-xunlong/OrangePiH6_external/blob/master/sys_config/OrangePiH6_3_sys_config.fex

[martinayotte]

5 hours ago, dreddit said:

Hello martinayotte the reboot works on the OP3 image from the Orange pi

No, we can't mix stuff from a Legacy build and Mainline one ...

[dreddit]

Thanks for the update.

[dolphs]

Hi - I just tried ubuntu server built by Xunlong.

Indeed, as "dreddit" already wrote earlier, I can also confirm my "  OrangePi One Plus " reboots perfectly fine.

This is good news in terms it should be possible to make things work, but reading comments above there are some hurdles to overcome?

 

Now if you would please excuse my ignorance, but could someone be so kind to explain why from link provided,  part of the code cannot be adapted except "stuff cannot be mixed" .

I am not a developer, just a silly end user and therefore I cannot fully oversee the efforts that need to be done,

 

eg. would it be a patch to kernel 5.x versions, is it a feature that needs to be incorporated and addressed first at kernel mailinglist, etc etc.

Please enlighten me, it is appreciated and aye I do foresee a solution in the long run keeping my patience.

cheers

 

[johanvdw]

I don't think you need the whole config, the commit which fixes reboot is just this one:

https://github.com/orangepi-xunlong/OrangePiH6_external/commit/4c2440c02e29f84aef7a296f90e170f38e12c248

[lanefu]

1 hour ago, johanvdw said:

I don't think you need the whole config, the commit which fixes reboot is just this one:

https://github.com/orangepi-xunlong/OrangePiH6_external/commit/4c2440c02e29f84aef7a296f90e170f38e12c248

 

To help clarify

 

Xunlong used a "legacy" method for configutation called fex.  It is unique to allwinner android kernels.   Fex is not used with linux mainline kernels, which is what armbian uses.  

[dolphs]

thanks for enlighten me, so it is fex that has been used in their stuff. 

I did find just now documentation here at armbian docs

cheers

 

[lanefu]

If I were to try to figure this out..... where would I start?

Device Tree in Kernel? then see how like another Pi is done, then stare at OpiOnePlus schematics until i have a breakthrough?

or is it a u-boot thing?

[martinayotte]

12 hours ago, lanefu said:

or is it a u-boot thing?

I'm pretty sure it is a U-Boot thing.

Even the U-Boot "reset" command is stalling.

I've tried to debug the reset_cpu() in arch/arm/mach-sunxi/board.c where it set some Watchdog register and loop infinitely, but it seems that watchdog never kicks in.

I will get back to this debug task when I get change, "time is the missing ingredient", by comparing with PineH64 which seems to reboot without hanging.

 

EDIT : Ok ! I've invested a bit of time last hour ... Bad news : taking the u-boot from PineH64 and write it to OPi3 SDCard revealed that it is still hanging there ...

So, I start doubting about the hardware itself ... I will try the same with BSP u-boot ...

[dolphs]

rats - it would have been too easy, thanks for the feedback although it is somewhat disappointing.

The watchdog being used is " sunxi_wdt ", right? 

hmm just a log shot using the software watchdog instead,

remember a post somewhere which describes something like "  echoing "  the watchdog with value 1 and then it reboots after so many seconds ( 20? ).

not sure if that is the key to success, as said just a long shot. thanks!

[martinayotte]

2 hours ago, martinayotte said:

Bad news : taking the u-boot from PineH64 and write it to OPi3 SDCard revealed that it is still hanging there ...

I must have done something wrong, because after another trial, it works !!!

59 minutes ago, dolphs said:

rats - it would have been too easy, thanks for the feedback although it is somewhat disappointing.

So, I will investigate to figure out "what are the differences between those 2 u-boots", because I didn't found any yet ...

 

EDIT : Bingo ! The missing thing is CONFIG_NR_DRAM_BANKS=1, although the documentation isn't explaining why, but it seems that diffs between OPis and PineH64 revealed this.

I will do full image for OPiLite2 and OPiOnePlus and see if this issue is definitively gone ...

[dolphs]

omg, let's bump dev to 5.05 ( current kernel ) and build a fresh debian image.

Mr martinayotte, I should like to express my respect to you for your work!

Will check back later as duty calls ( cooking for the wife ;-) )

[lanefu]

@martinayotte 

 

You're a machine!

[martinayotte]

3 hours ago, martinayotte said:

EDIT : Bingo ! The missing thing is CONFIG_NR_DRAM_BANKS=1, although the documentation isn't explaining why, but it seems that diffs between OPis and PineH64 revealed this.

I will do full image for OPiLite2 and OPiOnePlus and see if this issue is definitively gone ...

False Victory ...

Trying the same recipe with OPiLite2 and OPiOnePlus didn't work ...

Probably the the previous False success was because I've ran PineH64 u-boot than overwritting it with new one on the same SDCard.

I will digg more ...

[jernej]

25 minutes ago, martinayotte said:

Probably the the previous False success was because I've ran PineH64 u-boot than overwritting it with new one on the same SDCard.

I will digg more ...

If I'm not mistaken, that OPi fex fix did some changes around PMU. Can you check those settings?

[martinayotte]

40 minutes ago, martinayotte said:

False Victory ...

Here is the explanation why it was working earlier, but only for awhile :

I've booted a BSP U-Boot and stopped it without unplugging the power, then using any Mainline U-Boot, the "reboot/reset" is working fine since the BSP code still in memory, until I'm unpowering the board.

I've just did the same sequence with OPiOne, and the "reboot/reset" is working again there too ...

 

So, back to SquareOne !

 

18 minutes ago, jernej said:

If I'm not mistaken, that OPi fex fix did some changes around PMU. Can you check those settings?

Can you bring me some light ? I'm currently digging a whole while been in the fog without any shovel ...

[jernej]

4 minutes ago, martinayotte said:

Can you bring me some light ? I'm currently digging a whole while been in the fog without any shovel ...

Now that I checked changes again, it just about power key connected to PMU, so probably that's not it.

 

So, does reset with PineH64 U-Boot work on OPi or not? If not, then it has to be some peripheral settings...