[RockPro64] Armbian upstream kernel/uboot test?

[anonymoustemp]

Hi,

 

I got my RockPro64 board several days ago, and got *UPSTREAM* kernel (v5.1-rc5) and uboot (v2019.04) running on it using Archlinux ARM. (Sorry guys, I like upstream and Alarm more)

The pull request for uboot-rockpro64 can be found here:

https://github.com/archlinuxarm/PKGBUILDs/pull/1691

 

The upstream kernel is in the official core repor of Archlinuxarm (and that's why I like Alarm so much).

 

It boots and runs mostly OK on full upstream kernel/u-boot setup.

On rc-kernel it even has HDMI output working, but no hardware acceleration. I'm waiting panfrost as the open-source driver, so far it doesn't matter much as I don't plan to use it with X11/wayland anyway.

But it's awesome that you can just run GNOME3 on wayland, using llvmpipe.

And I even compiled mesa with panfrost natively on that board, so the future looks super bright.

 

However the biggest problem I hit is, upstream (5.0.8 and 5.1-rc5) kernel has random panic.

The panic itself is Async SError, and mostly happens for memory heavy operations (memtester and heavy file operations like checking out the whole kernel code).

The last panic I hit has no meaningful kernel calltrace, just el0_error_naked (user space triggered, and memtester is running)

 

At first I thought it's faulty ram, but when using the rockchip kernel from Armbian, it no longer crashes any more.

 

I have no clue what's going wrong, device tree? idbloader? trust.img? Uboot? Or just upstream kernel sucks? (I really hate to admit the last one)

 

Is there any upstream kernel precompiled in armbian for me to test so I could rule out one more problem?

 

Thanks.

Edited April 22, 2019 by anonymoustemp

[Igor]

1 hour ago, anonymoustemp said:

and got *UPSTREAM* kernel (v5.1-rc5) and uboot (v2019.04)

Most likely still waste of time.
 

1 hour ago, anonymoustemp said:

kernel has random panic

Which is normal and expected. RK3399 mainline support is far away from stable.
 

1 hour ago, anonymoustemp said:

Is there any upstream kernel precompiled in armbian for me to test so I could rule out one more problem?

https://dl.armbian.com/rockpro64/nightly/
IMO that's the best for RK3399 but last few builds are apparently completely broken. I would suggest you do use legacy kernel if you plan to do something useful with this board. For at least by the end of this year.

 

If you want to join debugging and development, get our build tools (sorry, but no direct ArchLinux output, Ubuntu/Debian only) which are attached to most promising up-streaming branch. Precompiled kernels can also be found in repository beta.armbian.com ...

[anonymoustemp]

20 minutes ago, Igor said:

Which is normal and expected. RK3399 mainline support is far away from stable.

That's what I didn't expect at all.

 

Shouldn't upstream get more review and tests? Or it's a completely different case for ARM world?

 

Quote

most promising up-streaming branch.

That branch looks pretty promising, although some commits don't meet upstream standard just by a quick glance.

 

But still it provides a pretty good clue for me to look into the random hang.

It pretty much limits the modification to a reviewable size, which is a huge leap compared to legacy kernel.

 

Thank you very much for the info.

[Igor]

3 hours ago, anonymoustemp said:

Shouldn't upstream get more review and tests? Or it's a completely different case for ARM world?

More people means more pressure (more wasted time) to already small and scattered developers base. I estimate that ratio is already very toxic, at least 1000 end-users on 1 developer. Most of end-users have no clue how much work is needed and they are usually reporting stupid, repeated and irrelevant things. We, on the other hand, are wasting time with educating, communicating to and support people and people are frustrated that things doesn't just work. And we are frustrated since its hard and expensive to deal with this complexity and people are not grateful ...  even just a few regulars are investing 20-50 hours per day into the project which is public and everyone benefits at the end. This time is covered way way below 1% which means all project costs are covered from our private budget. And users again expect a services worth millions of dollars. 

 

RK3399 is new and more complex. Much more than 10 years old Raspberry Pi design. Each ARM SoC is like a new architecture. Most of the kernels out there are private branches (legacy builds), mainlining is slow, expensive and mainly our cost. Some vendors financially support this process, most don't. I am mainly involved into Allwinner scene and there we have produced 300+ patches (just a few are distro specifics) on top of already patched mainline source. And (mostly minor) problems still exists. For some SoCs, Armbian and related groups (linux-sunxi, http://linux-meson.com(backed also by Amlogic), ... are the one bringing those boards up. Development eventually landed in the mainline, but late and as you already figured out, some features will never be accepted due to low quality or they remained "in development". We do accept them and sometimes polish, fix, maintain further. Since most of this development is done by amateurs with little support infrastructure and little help from public, work from development branches need year(s) to find the way to the mainline. Delay differs from vendor to vendor.

 

3 hours ago, anonymoustemp said:

It pretty much limits the modification to a reviewable size, which is a huge leap compared to legacy kernel.

Armbian mainline support goes this way - 1st stage, we attach or create own development branch and stay on that until difference with mainline becomes manageable. Then base source is switched to mainline and diff is added with patches. When things progress, patches are getting removed when they start to appear in the mainline. Ofc in reality this looks a lot more messy, but you get the idea. It's much more manageable.

[anonymoustemp]

23 minutes ago, Igor said:

More people means more pressure (more wasted time) to already small and scattered developers base. I estimate that ratio is already very toxic, at least 1000 end-users on 1 developer. Most of end-users have no clue how much work is needed and they are usually reporting stupid, repeated and irrelevant things. We, on the other hand, are wasting time with educating, communicating to and support people and people are frustrated that things doesn't just work. And we are frustrated since its hard and expensive to deal with this complexity and people are not grateful ...  even just a few regulars are investing 20-50 hours per day into the project which is public and everyone benefits at the end. This time is covered way way below 1% which means all project costs are covered from our private budget. And users again expect a services worth millions of dollars.  

Thank you very much for such information.

As I'm completely new to SBC community, and the huge different from my previous kernel experience indeed surprises me.

 

Now it totally makes sense.

 

25 minutes ago, Igor said:

Armbian mainline support goes this way - 1st stage, we attach or create own development branch and stay on that until difference with mainline becomes manageable. Then base source is switched to mainline and diff is added with patches. When things progress, patches are getting removed when they start to appear in the mainline. Ofc in reality this looks a lot more messy, but you get the idea. It's much more manageable.

Makes sense for the situation.

 

I'll check the branch to see if I can find something related and test.

 

And explore the armbian repo to find some clue with the Uboot/firmware (which is my current assumption of the strange crash).

 

Thank you again for the detailed explanation on the workflow.

 

 

[anonymoustemp]

Hopes this will be the last question.

 

What's the armbian recommended test suit for SBCs?

 

As I have only focused on fs so far, the main suits are fstests, LTP and bcc (for error injection and performance analyse).

It would be a good idea to have a simple suit to test stability (maybe just LTP?)

 

Thanks.

[Igor]

19 minutes ago, anonymoustemp said:

And explore the armbian repo to find some clue with the Uboot/firmware

 

Here I can't provide you any clues. Not working much with RK3399. Just tune in, do some forum research - you will find a lot of useful info.

[chwe]

Never used a suite for that.. mostly tools.. and those depending on what I want to test.. (e.g. simple cpu-burn isn't a bad thermal tester).

 

Are you currently on RCs for your rockpro? Didn't this guy just hit mainline?

http://lists.infradead.org/pipermail/linux-rockchip/2019-April/023867.html

 

ram on rk3399 is a painful story and the main reason why we have two boardfamilies here.. one based on upstream u-boot (rk3399), and one based on bsp u-boot (rockchip64). Yours is part of the second..

 

[off-topic]

@Igor you need something to laugh..

Spoiler

 

 

[anonymoustemp]

5 minutes ago, chwe said:

Are you currently on RCs for your rockpro? Didn't this guy just hit mainline?

http://lists.infradead.org/pipermail/linux-rockchip/2019-April/023867.html

Not in rc5 nor rc6 yet.

 

6 minutes ago, chwe said:

ram on rk3399 is a painful story and the main reason why we have two boardfamilies here.. one based on upstream u-boot (rk3399), and one based on bsp u-boot (rockchip64). Yours is part of the second..

Just saw the rockchip wiki mentioning it's *UBoot* initialize the memory controller, so bad Uboot build and bad that upstream patch.

 

I'm just taking the shortcut to use the armbian idbloader/uboot combo and running Archlinux Arm kernel to test if the v5.0.8 works.

 

Will report back soon.

[anonymoustemp]

The armbian idbloader/uboot/trust combo survives at least one loop.

While my bad upstream uboot build can not even survive 10 tests of a single loop.

 

Armbian community/build rocks!

 

And here is my feedback to Armbian:

 

Upstream kernel works relatively well with armbian uboot.

 

What I have tested:

1. boot (WORKING)
2. SDcard (WORKING)
3. git checkout for mesa (WORKING)
4. memtester, still running at least one loop passed (WORKING)
5. base system upgrade (WORKING)
6. ethernet (WORKING)
7. USB2.0 and 3.0 port (WORKING)
8. PCIE slot, no root complex, although PCIE power is on (NOT WORKING)

What I haven't tested yet but will test soon:

1. Memory performance (upstream alarm vs armbian, armbian uboot)
2. LLVMpipe wayland/X11
3. Maybe some basic audio test on that 3.5mm jack
4. eMMC (will take some time for that purchase)

What I won't test:

1. Blob X11/wayland, I just hate blob, especially when I just want a NAS build
2. BT/Wireless, Just don't want to spend extra money on a NAS.
3. GPIO headers, the same reason.

 

Now it's my turn to "steal" some juice from the armbian/ayumi uboot branch for alarm.

 

But AFAIK for nas build, I still need to use armbian legacy kernel for a while.

 

[chwe]

going through the links spotted something for @martinayotte (and maybe @TonyMac32)

 

https://github.com/archlinuxarm/PKGBUILDs/pull/1691/commits/dfe2881b2eaf32a5b2289005c675888345684fd7#diff-2347b909a334e35e45d31e5bb4961a3f

 

there's a a patchset for upstream u-boot for the ram part.. as OP says it's not running stable.. but at least they got upstream u-boot up.. maybe worth a look as well.. to clean the rk u-boot mess asap.

[anonymoustemp]

2 hours ago, chwe said:

going through the links spotted something for @martinayotte (and maybe @TonyMac32)

 

https://github.com/archlinuxarm/PKGBUILDs/pull/1691/commits/dfe2881b2eaf32a5b2289005c675888345684fd7#diff-2347b909a334e35e45d31e5bb4961a3f

 

there's a a patchset for upstream u-boot for the ram part.. as OP says it's not running stable.. but at least they got upstream u-boot up.. maybe worth a look as well.. to clean the rk u-boot mess asap.

Unfortunately, that's from myself and the you can see the original author mentioned that it's just a basic setup.

 

In fact, upstream Uboot, even just using evb board config, can boot on that RockPro64.

While USB doesn't work in that case and no sdcard support, only eMMC and ethernet, and I haven't tested eMMC.

 

With that commit, I ran into the ram issue.

 

[Mathias]

@anonymoustemp for me, no matter which kernel I use, USB3 is not stable. With Armbian's 4.4.173, I end up with kernel NULL pointer when performing "heavy" I/O on USB3 (here "heavy" means reading or writing a few Gb of data). With Ayufan's builts (4.20 or 5.0 kernels), I end up with filesystem corruption (and the system must be restarted because it is so messed up).

 

@IgorThanks a lot for your explanations! I was actually wondering if I could offer to write some RockPro64 documentation somewhere in the wiki, Bu I am not sure where it would fit best. Would creating a RockPro64 "hardware notes" section be acceptable to the Armbian community?

 

Thanks anyway for the good work!

Mathias

[Igor]

42 minutes ago, Mathias said:

Would creating a RockPro64 "hardware notes" section be acceptable to the Armbian community?

Thanks!

If you plan to keep them concise, then this https://www.armbian.com/rock64/ is best place. Here

you can check what is generally expected. If you agree, you get a password on email, the rest - technical details if needed - are written in the dashboard when you login.

[chwe]

13 hours ago, anonymoustemp said:

In fact, upstream Uboot, even just using evb board config, can boot on that RockPro64.

While USB doesn't work in that case and no sdcard support, only eMMC and ethernet, and I haven't tested eMMC.

that's why it's a good starter.. we had even issues in getting the board booting with upstream u-boot.

 

2 hours ago, Mathias said:

@anonymoustemp for me, no matter which kernel I use, USB3 is not stable. With Armbian's 4.4.173, I end up with kernel NULL pointer when performing "heavy" I/O on USB3 (here "heavy" means reading or writing a few Gb of data). With Ayufan's builts (4.20 or 5.0 kernels), I end up with filesystem corruption (and the system must be restarted because it is so messed up).

are you sure it's an software issue? Could also be bad SATA-USB enclosure (assuming that's what you connect to USB3) and or bad powering..

[Mathias]

@chweI've used the enclosure (based on JMicron JMS 578 chip) on my laptop to transfer almost 400G of data without any problems twice (with UAS mode, thanks to an updated firmware ) but as soon as it is connected to my RockPro, I can not transfer/read more than a few Gb (ie the nightly rsync of my hosted Nextcloud to my backup system was enough to trigger the bug). I've tried disabling uas (blacklisting it and checking that it was really used as "mass storage") but this was not enough to make it stable. So far, I've reverted to moving my Nextcloud to an sdcard (so the rsync does not find anything on the USB3 drive that needs syncing). If you want, I can provide a backtrace (I ssh to it and start copying data around, so when it goes down, I still have the backtrace in my ssh terminal).

 

@IgorI would love to step in as board "maintainer" but so far, I don't have enough days as registered member... (and I can not promise to test all new stuff regularly, since I also use my board as a server, so I can not always afford to bring it down at any given time). In the mean time, I can start a thread in the forums to collect some links and information about the board.

[Igor]

3 minutes ago, Mathias said:

I would love to step in as board "maintainer" but so far, I don't have enough days as registered member... (and I can not promise to test all new stuff regularly, since I also use my board as a server, so I can not always afford to bring it down at any given time). In the mean time, I can start a thread in the forums to collect some links and information about the board.

You got email with credentials  Welcome!

Changes are usually very slow and when you find out that some important problems are solved, you edit/remove that part of listed bugs. Bug list is not that big anymore.

[chwe]

@Mathias, congrats to your new role.. somehow similar I got mod here..

 

Well, my interest in your project is minor.. and just to be clear. I don't claim that there aren't any software bugs for USB3 on those boards (I simply don't know it). But a driver/DTS issue might be hard to debug (depending on your kernel experience - at least for me it would be). Whereas troubleshooting hardware related bugs might be easier.

30 minutes ago, Mathias said:

based on JMicron JMS 578 chip

I assume it's the one provided by pine in their store? From what I heard (and this is mostly @tkaiser review of it) It should be one of the better choices.

 

looking into schematics.. USB should allow at least 5V/1A on each port as long as the PSU can deliver it (http://files.pine64.org/doc/rockpro64/rockpro64_v21-SCH.pdf p3). Did you check dmesg if there's some obvious errors. E.g. losing link multiple times during heavy load or try the same adapter on an USB2 of the RockPro this would nail it down to USB3 (software or hardware - cause powering for those are the same on the RockPro).

[Mathias]

@IgorThanks! Now I need to deliver... :-)

 

@chweNo, I'm using a "noname" enclosure that is based around the JMS578. But again, it just works great when connected to my laptop (with Debian Buster) and always makes trouble on the RockPro. I wanted to make sure this is not a problem with power, so I used a 6A power supply (universal power supply designed for laptops) and got the same trouble. But I'll try again with this power supply and the "stable" 4.4 kernel and either come back with a big smile on my face or with a backtrace...

[chwe]

did you check it's not a cable issue (maybe your rockpro64 usb3 isn't as tight as the one on your notebook. It can be a software issue.. just check the hardware possibilities as well.

[Mathias]

I have replaced the cable by another one (freshly purchased and tested on my laptop), I still have the same problems. I've not been able to capture the backtrace this time, but at least I have some kernel messages showing how the kernel is fighting to make it work: some messages early on and then more later on (it fully crashed a few minutes later). Again, this is with kernel 4.4.174 and a JMS578 (patched firmware) based enclosure...

[Mathias]

I finally managed to understand why neither pine64 nor Ayufan could reproduce this bug: this has to do with usb autosuspend. On a freshly booted system, USB3 works perfectly fine. On a system that has been idle for a day (so the USB had time to suspend), the problem is back with tons of error messages as soon as I try to copy a few hundred Mb. This was also mentioned in this thread.

 

[edit] The autosuspend is not the culprit... This has to do with some underpower, see this (long) thread. Sorry for the fake news and the disappointment!