Research Some experiences with Odroid-N2+ on Debian Bullseye

[qwr]

I first used Kingston endurance SD card with Armbian Buster image. This failed due some incompability with Odroid N2 UHS implementation, but Sandisk extreme SD card worked with same image.

After successful boot runned nand-sata-install to ADATA SP 550NS38 SSD connected using VL817 SATA Adaptor (2109:0715).
Upgraded the Debian to Bullseye and installed Mesa 21.2.0 packages from experimental repositories, arm64 firefox-esr and armhf Chromium via multiarch to use widevine extracted from ChromeOS images (local err.ee web broadcasts some programs with widevine encryption). This was in the end of august. It mostly worked, but had 3 weird problems that I think to have solved now having some free time.

 

1. The USB boot hanged up about half of times after kernel detected the SATA drive. Turns out that UAS is culprit, disabling UAS with usb-storage.quirks=2109:0715:u made it now boot every time without problems (modified /boot/boot.ini).
2. About 2-3 minutes after boot the compositor froze (no matter which - tried Xorg compton, wayland Weston and Wayfire (self compiled). Turns out armbian has /etc/udev/rules.d/hdmi.rules which runs /usr/local/bin/hdmi-hotplug. For some reason the hdmi-hotplug script stalls on my Odroid installation and when systemd finally killed it, something went wrong and the compositors froze. Diverting the hdmi.rules fixed it.
3. This was/is weirdest - when TV connected via HDMI was off and turned on, then sometimes the board would shutdown. Found from auth log "auth.log.1:Dec 25 19:04:59 piix systemd-logind[1641]: Power key pressed." Like WTF, i don't have any "power key" and why it gets invoked when TV is turned on. Set the *Key=ignore in /etc/systemd/logind.conf and it seems fine now, no idea what went wrong there. The kernel even don't have CEC enabled (CONFIG_CEC_MESON_AO and CONFIG_CEC_MESON_G12A_AO are not set for some reason in the armbian kernels).

 

A bit later I also saw 5.15.8-meson64 kernel OOPS "Unable to handle kernel execute from non-executable memory" at regmap_update_bits_base+0x74/0x98, meson_clk_cpu_dyndiv_set_rate+0xf4/0x118. As I hadn't seen this before, downgraded to 5.13.12-meson64. I plan to report the OOPS with full kernel dmesg output, when I'll have a bit more time (probably the https://bugzilla.kernel.org/ would be right place?).

 

Currently it seems to work fine, with the 5.13.12-meson64 kernel and mesa/panfrost 21.2.1-2 (from snapshot.debian.org). Weston is used as Wayland compositor. It is important to note that I have 1080p TV, and all video decoding is done on CPU without using the Amlogic VPU acceleration, as the VPU driver is currently both broken in the kernel and unsupported in unpatched userspace. I expected this when choosing the board. Fortunately the 4xA73@2.4GHz is fast enough for decoding most 1080p videos and panfrost is fine for doing the video output after decoding. AFAIK playing 4K videos is currently not possible on Odroid N2 with mainline kernel and the vendors proprietary VPU decoder isn't supported by any software in open source distributions like Debian. Interesting is that Firefox seems to handle the video decoding pathway better for Youtube in this configuration,
while Chromium occasionally stutters on some Youtube videos, these seem to be fine with Firefox. Luckily this seems to not affect the site where I needed widevine (that works only with Chromium in practice). In september widevine upgraded to version needing patched libc. I got patched armhf libc6 package from apt.xbian.org repository.

 

The ALSA configuration is weird, amixer scontrols | wc gives 50 controls, and if misconfigured, then the thing doesn't give any audio output. I found the following script from somewhere that "fixes" it into working state:

 

amixer sset 'FRDDR_A SINK 1 SEL' 'OUT 1'
amixer sset 'FRDDR_A SRC 1 EN' 'on'
amixer sset 'TDMOUT_B SRC SEL' 'IN 0'
amixer sset 'TOHDMITX I2S SRC' 'I2S B'
amixer sset 'TOHDMITX' 'on'
amixer sset 'FRDDR_B SINK 1 SEL' 'OUT 2'
amixer sset 'FRDDR_B SRC 1 EN' 'on'
amixer sset 'TDMOUT_C SRC SEL' 'IN 1'
amixer sset 'TOACODEC SRC' 'I2S C'
amixer sset 'TOACODEC OUT EN' 'on'
amixer sset 'TOACODEC Lane Select' '0'
amixer sset 'ACODEC' '255'
amixer sset 'FRDDR_C SINK 1 SEL' 'OUT 3'
amixer sset 'FRDDR_C SRC 1 EN' 'on'
amixer sset 'SPDIFOUT SRC SEL' 'IN 2'

 

After that ALSA pcm "hw:0,1" output works. I configured dmixed and removed pulseaudio, as it seemed to complicate things with no benefit when mpc and browser run as different users.

 

Armbian is great (couldn't have the board working so well without armbian) and merry holidays for you

Hoping this helps others having Odroid N2(+) board.

Edited December 29, 2021 by qwr

[ostanislaw]

Hi,
I've recently flashed new Odroid N2+ with dietpi and armbian, I'm going to build kubernetes training cluster with some personal home services like syncthing, myjdownloader, pyload, NFS etc. The first is nice distro but it seems to hang out on intensive file operations (maybe related with Sandisk Extreme Pro A2 card - I was not aware that A2 might be not supported well) in all of about 5 tests of trying to install texlive-full in ubuntu docker container it just hangs and board is restarted by watchdog.
Generally I'm happy with armbian debian and it's not crashing/hanging at all.

I don't remember if armbian had issues with default kernel but I've installed the edge and it's completely stable. Mostly because I've read that at 5.13 there were some improvements for odroid n2+.

Please try the edge kernel:

ostanislaw@odroid1:~$ uname -a
Linux odroid1 5.15.5-meson64 #trunk.70 SMP PREEMPT Thu Nov 25 14:05:30 UTC 2021 aarch64 GNU/Linux
ostanislaw@odroid1:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 11 (bullseye)
Release:	11
Codename:	bullseye
ostanislaw@odroid1:~$ sudo apt list --installed *meson64*
[sudo] password for ostanislaw: 
Listing... Done
linux-dtb-current-meson64/bullseye,now 21.08.6 arm64 [installed]
linux-dtb-edge-meson64/bullseye,now 21.11.0-trunk.70 arm64 [installed]
linux-image-current-meson64/bullseye,now 21.08.6 arm64 [installed]
linux-image-edge-meson64/bullseye,now 21.11.0-trunk.70 arm64 [installed]

Today I tried also ubuntu hirsuite headless but I think it hanged once and had 100% issue with reboots, needed to power cycle the board.
After I've removed current packages keeping only edge, and updated u-boot to edge it was not booting anymore (don't know which of them caused issue), nothing displayed at hdmi.
Reflashed again to bullseye but I'll keep current line of kernel dtb etc to see if they are reliable. Reboot issue is rather rarely reproducing, probably happened only once before full upgrade.

[Igor]

4 hours ago, ostanislaw said:

Reboot issue

... is here for some time and it will probably stay since we already lost insane amount of expensive private time fighting this problem.
 

5 hours ago, ostanislaw said:

I don't remember if armbian had issues with default kernel

 

If you mean their stock 4.9.y kernel? Well, it has its pros but I would not run such complicated things with it. Like Kubernetes cluster. That kernel is good for desktop and gaming, for home users ... 

 

5 hours ago, ostanislaw said:

After I've removed current packages keeping only edge, and updated u-boot to edge it was not booting anymore (don't know which of them caused issue), nothing displayed at hdmi.

Hard to say for this specific case, but there are plans to improve upgrade process on higher level. Just low interest from community is preventing this to proceed

https://armbian.atlassian.net/browse/AR-572
 

4 hours ago, ostanislaw said:

Reflashed again to bullseye

 

Won't make any difference since hardware interface (kernel + bootloader) defines troubles with reboot. We provide the same binary kernel for all those variants (Debian X, Ubuntu Y), including 3rd party distros like Dietpi. 

Some background info:

https://docs.armbian.com/#what-is-armbian

https://docs.armbian.com/User-Guide_FAQ/

[MichaIng]

Am 29.12.2021 um 17:25 schrieb qwr:

amixer sset 'FRDDR_A SINK 1 SEL' 'OUT 1'
amixer sset 'FRDDR_A SRC 1 EN' 'on'
amixer sset 'TDMOUT_B SRC SEL' 'IN 0'
amixer sset 'TOHDMITX I2S SRC' 'I2S B'
amixer sset 'TOHDMITX' 'on'
amixer sset 'FRDDR_B SINK 1 SEL' 'OUT 2'
amixer sset 'FRDDR_B SRC 1 EN' 'on'
amixer sset 'TDMOUT_C SRC SEL' 'IN 1'
amixer sset 'TOACODEC SRC' 'I2S C'
amixer sset 'TOACODEC OUT EN' 'on'
amixer sset 'TOACODEC Lane Select' '0'
amixer sset 'ACODEC' '255'
amixer sset 'FRDDR_C SINK 1 SEL' 'OUT 3'
amixer sset 'FRDDR_C SRC 1 EN' 'on'
amixer sset 'SPDIFOUT SRC SEL' 'IN 2'

 

Many thanks for this. I played a bit more around with these and found the following logic:

• With TOACODEC OUT EN and TOHDMITX one can enable analogue 3.5mm output and digital HDMI output.
• With TOACODEC SRC resp. TOHDMITX I2S SRC one can select the I2S device used for 3.5mm output respectively HDMI output. I cannot get I2S A working (because there is no TDMOUT_A at all), but I2S B and I2S C work for both HDMI and 3.5mm each. I assign it like you did, I2S B to TOHDMITX I2S SRC and I2S C to TOACODEC SRC.
• Now, with TDMOUT_B SRC SLT resp. TDMOUT_C SRC SLT one can assign I2S B resp. I2S C to an ALSA device, i.e. which one will be hw:0,0, hw;0,1 or hw:0,2. Since there is no TDMOUT_A, I cannot assign I2S A to any ALSA device. I set TDMOUT_B SRC SLT (HDMI) to IN 0 (device 0) and TDMOUT_C SRC SLT to IN 1 (device 1).
• FRDDR_[ABC] SRC [123] EN needs to additionally enable the I2S sources on the ALSA device, but somehow it is associated in a confusing way: FRDDR_A is related to ALSA device 0, FRDDR_B to ALSA device 1 etc, SRC 1 is related to I2S A, SRC 2 to I2C B etc. Would have been easier to understand when those would be named FRDDR_[012] SRC [ABC] so that device index and I2C identifier match. However, hence I need to enable FRDDR_A SRC 2 and FRDDR_B SRC 3, matching the TDMOUT_[BC] SRC SLT assignments above.
• Finally I think its the output channels which need to be split for the used ALSA devices, FRDDR_[ABC] SINK [123] SEL. Basically, at least when all channels are used, the three sinks for the used ALSA devices need to be split. With my 2.0/2.1 test audio hardware I just use OUT 0, OUT 1 resp. OUT 2 for each FRDDR_[AB] SINK 1, 2 resp. 3.
• All other controls seem to have no effect in my case, I guess its about SPDIF input and assigning capture devices/microphones, as of TODDR (to) and FRDDR (from) hence to which audio device input is sent compared what we assigned from which audio device the stream is received and sent to which sinks/channels, or so.

[ostanislaw]

On 1/13/2022 at 5:57 PM, Igor said:

... is here for some time and it will probably stay since we already lost insane amount of expensive private time fighting this problem.
 

OK, I understand. But anyway it seems for current armbian bullseye, fully updated the issue not reproduces. At least it was not reproduced at 3 boards for 5-10 reboots I did.

On 1/13/2022 at 5:57 PM, Igor said:

If you mean their stock 4.9.y kernel? Well, it has its pros but I would not run such complicated things with it. Like Kubernetes cluster. That kernel is good for desktop and gaming, for home users ... 

No, I mean all current (not edge versions) installed by default on armbian:
# apt list --installed "*current*"
Listing... Done
linux-dtb-current-meson64/bullseye,now 21.08.6 arm64 [installed]
linux-image-current-meson64/bullseye,now 21.08.6 arm64 [installed]
linux-u-boot-odroidn2-current/bullseye,now 21.08.1 arm64 [installed]
 

More experiences to share (ARMBIAN bullseye @ ODROID N2+ for all 3 SBC):
1. I've installed k3s on 3 nodes (master + 2 worker nodes).
2. The master node has 2TB HDD connected by USB. So It's one of the cheapest toshiba Canvio Basics 2TB.
It's sharing it with NFS.
And I'm very happy with it's performance it seems it writes 50-100MB/s for a couple of big files. And combined with async on server side it takes advantage of 4GB ram for buffering writes so I'm really able to write a lot of small files and they are dropped to the NAS really quickly 100MB/s where 1Gb/s Ethernet is the only limit. Great, not worth investing in SSD IMHO.
But as alternative NAS solution I think odroid hc4 + maybe HDD+SSD might also be a good idea. But it doesn't have that cool big passive heatsink mounted at factory 
3. I'm considering this impressive home cluster template to experiment with:
https://github.com/k8s-at-home/template-cluster-k3s
4. Image my cluster attached. I'm going to just add 1GB raspberry pi3 on top of it.

[qwr]

Small update. My Odroid-N2+ has been stable for nearly month with the 5.15 Armbian kernels, with one caveat. It has had mpd playing most of the time with occasional video playing from browser, so it has had light load almost continuously.

 

The caveat is, that using schedutil cpu frequency governor brings out some latent bug causing something unholy in the kernel. Only way to reproduce was about an hour of playing some video, and usually it hanged totally with nothing logged by kernel anywhere. If there was a trace, it was different each time. It didn't seem to depend on kernel version either (I think I tried both 5.13 and 5.15). So I can't even make any reasonable bug report out of it. I have uboot from last year, so it could be some already fixed initialization problem raising its head. Only useful infomation out of that is, if anybody is having unexplained hangs/crashes with Odroid-N2 using schedutil, then switching to another governor might help.

 

However, with performance (Armbians default configuration) and ondemand governors its not reproducible.

[usual user]

On 12/29/2021 at 5:25 PM, qwr said:

AFAIK playing 4K videos is currently not possible on Odroid N2 with mainline kernel

For me the hardware decoder (v4l2h264dec) is working for 4K.

video-pipeline-4K.pdf