[[Fixed] Changing Amlogic GPU RAM allocation through the DTB?]

I have a number of headless AML S905 / S912 1GB Ram TV boxes running a bastardised Armbian for systems monitoring. I have a 200MB zram compressed block device setup to store temporary data which gets processed locally then synced to a command server every hour because the eMMC's on these cheap boxes do not last very long under sustained write scenarios. I would like to increase the zram filesystem size to 500MB so I only have to sync daily instead of hourly but because 25% of the RAM is allocated to the GPU/VPU I dont have enough available ram

 

All the devices I have seen seem to allocate approx 250MB for the GPU which is obviously wasted in a headless scenario. I would like to reduce this to 32MB or 64MB which would give enough system RAM left over for a 500MB zram allocation

 

Can anyone point me in the right direction of what I need to add or modify in the DTS files? 

 

So far my digging has me here:

Im guessing this is or can be accomplished through modifying the DTB's for these devices but having looked through the DTS source files for various AML S9xx and RK33xx boxes I cant find anywhere where the GPU RAM allocation is defined. Im a complete noob when it comes to device trees so I have been studying @balbes150 Amlogic_s905-kernel github repo

 

Using a Nexbox A95X as an example I can see that the dtb image gets built from the source code of the following dts files:

 

https://github.com/150balbes/Amlogic_s905-kernel/tree/master/arch/arm64/boot/dts/amlogic

meson-gxl-s905x-nexbox-a95x.dts
meson-gxl-s905x.dtsi
meson-gxl.dtsi
meson-gx.dtsi
meson-gxl-mali.dtsi

 

Potentially Interesting looking entries:

In the meson-gxl-s905x-nexbox-a95x.dts file I found the following, but modifying it seems to just reduce total RAM available to the SoC

memory@0 {
    device_type = "memory";
    reg = <0x0 0x0 0x0 0x80000000>;
};

 

In meson-gx.dtsi I found vdec and vpu declarations. They seem to declare 5 separate memory regions but nothing seems to match the size of what has been reserved

vdec: video-decoder@c8820000 {
    compatible = "amlogic,gx-vdec";
    reg = <0x0 0xc8820000 0x0 0x10000>,
          <0x0 0xc110a580 0x0 0xe4>;
    reg-names = "dos", "esparser";

    interrupts = <GIC_SPI 44 IRQ_TYPE_EDGE_RISING>,
         	    <GIC_SPI 32 IRQ_TYPE_EDGE_RISING>;
    interrupt-names = "vdec", "esparser";

    amlogic,ao-sysctrl = <&sysctrl_AO>;
    amlogic,canvas = <&canvas>;
};

vpu: vpu@d0100000 {
    compatible = "amlogic,meson-gx-vpu";
    reg = <0x0 0xd0100000 0x0 0x100000>,
          <0x0 0xc883c000 0x0 0x1000>,
          <0x0 0xc8838000 0x0 0x1000>;
    reg-names = "vpu", "hhi", "dmc";
    interrupts = <GIC_SPI 3 IRQ_TYPE_EDGE_RISING>;
    #address-cells = <1>;
    #size-cells = <0>;
    amlogic,canvas = <&canvas>;

    /* CVBS VDAC output port */
    cvbs_vdac_port: port@0 {
        reg = <0>;
    };

    /* HDMI-TX output port */
    hdmi_tx_port: port@1 {
        reg = <1>;
            hdmi_tx_out: endpoint {
            remote-endpoint = <&hdmi_tx_in>;
        };
    };
};

 

 

[Armbian on X99 Tv Box]

It works fine with @balbes150 h96max dtb

 

Im using one as a nextcloud server. Even wireless and bluetooth work if you copy the firmware from /usr/lib/firmware/rkwifi to /usr/lib/firmware/bcrm

 

I havent figured out booting from eMMC yet so I have u-boot / kernel on an SD card and the root fs on the eMMC. Or you can just boot and run direct from SD. Im using a 256GB SATA SSD on the USB C port and a 12TB hard disk on the USB 3 A port. Performance is pretty good

 

Try @balbes150 RK3399 Armbian 5.91 TV Box image, it works great. The only issue I found was that there was no display on HDMI initially, the console was set to go to a serial debug port that doesn't exist on that device. After you write the SD card image, mount the boot partition first and modify the extlinux.conf file and add "console=tty1" to get a tty over HDMI and "console=ttyFIQ0,1500000n8" to get a serial console on the debug port. My extlinux.conf file for the X99 is:

 

LABEL Armbian
  LINUX /Image
  INITRD /uInitrd
  FDT /rk3399-h96max.dtb
  APPEND root=/dev/mmcblk1p1 rootflags=data=writeback rw console=tty1 console=ttyFIQ0,1500000n8 no_console_suspend consoleblank=0 fsck.fix=yes fsck.repair=yes net.ifnames=0

 

DTB that Im using attached

rk3399-h96max.dtb

[2.5gbe USB rtl]

Updated driver / kernel works perfect with both of my rk3399 TV boxes, Orange Pi RK3399 and my OP1 SoC ASUS chromebook

[2.5gbe USB rtl]

tvbox2.lan: aarch64, H96MaxTVBox-RK3399, USB RTL8156 (RTL driver), Custom userland, Kernel 4.4-rk

 

Test 5: Netcat UDP, tvbox2.lan --> faphq.lan

root@tvbox2:~# dd if=/dev/zero bs=100M count=10 | nc -v -u faphq.lan 9999
Connection to faphq.lan 9999 port [udp/*] succeeded!
10+0 records in
10+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 10.112 s, 99.7 MB/s

Test 6: Netcat UDP, faphq.lan --> tvbox2.lan

root@faphq:~# dd if=/dev/zero bs=100M count=10 | nc -v -u tvbox2.lan 9999
Connection to tvbox2.lan 9999 port [udp/*] succeeded!
10+0 records in
10+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 10.109 s, 99.7 MB/s

Using the 4.4-rk kernel and realtek driver I cant get it to connect at 2.5Gb at all

 

I have also just done a quick test between 2 Linux Mint 19.1 x86_64 boxes with the RTL8156's and kernel 5.0 and surprisingly they connect at 2.5Gb line rate and data transfer speeds are as they should be:

1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.9438 s, 273 MB/s

Ran out of time for any further testing tonight sorry 

[2.5gbe USB rtl]

Test 3: Netcat UDP, tvbox1.lan --> faphq.lan

root@tvbox1:~# dd if=/dev/zero bs=100M count=10 | nc -v -u faphq.lan 9999
Connection to faphq.lan 9999 port [udp/*] succeeded!
10+0 records in
10+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 9.98239 s, 107 MB/s

Test 4: Netcat UDP, faphq.lan --> tvbox1.lan

root@faphq:~# dd if=/dev/zero bs=100M count=10 | nc -v -u tvbox1.lan 9999
Connection to tvbox1.lan 9999 port [udp/*] succeeded!
10+0 records in
10+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 23.008 s, 46.9 MB/s

First test on the RK3399 device is roughly half the performance of the Jetson, again the recieve performance is much lower than the send. This is on kernel 4.19 (I already had a config I knew would work). Im compiling a new 5.2 kernel now so I can test like for like kernel versions for cdc_ncm

 

Edit: Performance on Kernel 5.2 is identical to 4.19 +/- 2%. Same asynchronous transfer speed and still half the performance of the Jetson

[2.5gbe USB rtl]

Ive started some testing but there is clearly a problem with this NIC or driver. I will post the Jetson Nano results now as it will take me another hour or so to finish compiling and setting up tests for the rockchip TV Box based boards as I want to try an older kernel with the official realtek driver. Im not even bothering with iperf yet

 

Test environment and server testing against is as follows:

Network: Netgear XS712T-200NES 10GB switch
Config: Everything as defaults, No cpu affinity
faphq.lan: x86_64, i9-7980XE, Intel 10GB ethernet, Linux Mint 19.1, Kernel 5.0 (Home Server)

 

Clients config:

jetson.lan: aarch64, Jetson Nano, USB RTL8156 (cdc_ncm), Ubuntu 18.04, Kernel 5.2
tvbox1.lan: aarch64, H96MaxTVBox-RK3399, USB RTL8156 (cdc_ncm), Custom userland, Kernel 4.19
tvbox2.lan: aarch64, H96MaxTVBox-RK3399, USB RTL8156 (RTL driver), Custom userland, Kernel 4.4-rk

 

Test 1: Netcat UDP - jetson.lan --> faphq.lan

root@jetson:~# dd if=/dev/zero bs=100M count=10 | nc -v -u faphq.lan 9999
Connection to faphq.lan 9999 port [udp/*] succeeded!
10+0 records in
10+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.4171 s, 244 MB/s

Test 2: Netcat UDP - faphq.lan --> jetson.lan

root@faphq:~# dd if=/dev/zero bs=100M count=10 | nc -v -u jetson.lan 9999
Connection to jetson.lan 9999 port [udp/*] succeeded!
10+0 records in
10+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 10.980 s, 98.2 MB/s

Edit: On the jetson it automatically negotiated the correct 2.5Gb link speed, not 1Gb but obviously there is a huge difference in performance depending on the direction

 

[2.5gbe USB rtl]

It doesnt seem related to cpu affinity from your output. Its certainly not entropy. Very strange behaviour. Did you try using netcat to get a baseline outside of iperf?

 

I actually stopped by at CPC on my way home today and picked up a couple of the PSG91497 Pro signal adaptors you have. They seemed too cheap to be true! I dont have any RK devices with USB3 unfortunately, apart from Android TV boxes so I cant really do an RK specific test (unless I can find an armbian or libreelec build for them). I will try them tonight on a Jetson Nano and any other USB3 arm devices I have at home

 

 

[2.5gbe USB rtl]

You probably dont need haveged and your current entropy pool is fine and full but if it empties when iperf is running then it might temporarily block random traffic generation

 

Also iperf does sometime give unexpected results when using "-R" as opposed to "-s", just listening as a server so it might be worthwhile using netcat to get an alternative baseline and see what performance you get firing a 100MB+ file over it

 

Im not concerned about the cdc_ncm usb0: network connection: connected syslog flood, as long as it doesnt have disconnects. The flooding is seen on all RTL8156 USB devices I have used when using cdc_ncm as opposed to the Realtek driver. For kernels >4.15 cdc_ncm is the only option anyway

[2.5gbe USB rtl]

The imbalance in directional performance can be caused by quite a few different things, including bad cables still (I know it shouldnt according to spec, but on real equipment, especially cheaper stuff it does)

 

Can you do the same tests, configured as follows, to give me a better idea of whats going on please?

 

1. Connect the 2 RTL8156 nics directly with a single cable (The RTL8156 is MDI / MDI-X compliant so it should automatically go into crossover mode without needing a 4 pair crossover cable)
2. Test with reversed cable direction (I know this seems stupid but humour me)
3. Force the RTL8156 on the RockPi side into 1GB

 

Before testing make sure that haveged is started and running and while the tests are running can you watch for:

• CPU use and specifically look to see if 1 particular CPU core maxes out
• dmesg for cdc_ncm disconnects (how many?)
• dmesg for "entropy", "CRNG" or "crng" (less likely)

 

 

[2.5gbe USB rtl]

Looks like a faulty cable from the inconsistent results pattern, especially the retries

 

I see this same pattern quite a lot with 2.5/10/40GBe devices. The driver thinks the ethernet cable is constantly up and down, many times a second sometimes

 

Can you grep dmesg output for "cdc_ncm" and see if you have lots of disconnects and reconnects. If so its almost certainly the cable

[Need Armbian SIP client on the Cubieboard 1 Alwinner A10]

6 hours ago, Irshat said:

Good that you answered me. I'm not a very good programmer, so please help me. I can install linphonec, but I do not know how to approach the GPIO.

Armbian version and base? Sunxi or mainkine kernel?
What scripting languages are you familiar with?

[Need Armbian SIP client on the Cubieboard 1 Alwinner A10]

Use Linphone as the sip client controlled via the linphonec command line interface. Use bash, python, php or whatever you are comfortable with to script linphonec and interact with the GPIO's. Its pretty simple

[[Fixed] Random pauses on S905 / X based hardware]

11 minutes ago, amirul said:

If it is entropy, is haveged running? I found that on kernel >5 it needs --data-16 in /etc/default/haveged DAEMON_ARGS to run

Yes it was all caused by lack of entropy. And yes your fix of modifying /etc/default/haveged is better than my fix of modifying the systemd unit file

 

Just set the DAEMON_ARGS="-w 1024 --data=16" in /etc/default/haveged for Kernels >5

 

 

[[Fixed] Random pauses on S905 / X based hardware]

Fixed. Ill post the solution so if anyone else is having the same problem they can fix it

 

Basically haveged is not starting because it cant detect the CPU cache size and so cant feed itself all the variables it needs to start up

 

Looking through the armbianmonitor output I noticed that lscpu / cpuid is unable to detect the CPU cache sizes at all

#lscpu:
Architecture: aarch64
Byte Order: Little Endian CPU(s): 4
On-line CPU(s) list: 0-3 Thread(s) per core: 1 
Core(s) per socket: 4 
Socket(s): 1 
Vendor ID: ARM 
Model: 4 
Model name: Cortex-A53 
Stepping: r0p4 
CPU max MHz: 1512.0000
CPU min MHz: 100.0000
BogoMIPS: 48.00 
L1d cache: unknown size 
L1i cache: unknown size 
L2 cache: unknown size 
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid

I think the S905 series has a 16kb L1 cache (correct me if Im wrong) so changing the haveged systemd unit to hard code a 16kb data size fixes the problem. Performance of encrypted traffic closely matches that of non encrypted traffic now

 

Fix is simple: just add --data=16 to the systemd unit arguments on the daemon start line

Modify /etc/default/haveged to include --data=16 on the DAEMON_ARGS line

# Configuration file for haveged
# Options to pass to haveged:
#   -w sets low entropy watermark (in bits)
DAEMON_ARGS="-w 1024 --data=16"

Edited fix as @amirul suggested the correct way to modify haveged args

[[Fixed] Random pauses on S905 / X based hardware]

Update 1:

Done some more extensive testing using different network protocols. The problem only happens with encrypted data traffic. Non encrypted traffic actually flows normally and without any problems

 

HTTP, FTP, POP, IMAP all work fine. HTTPS, SCP, Samba, Wireguard, OpenVPN all fail.

 

SSH actually works fine surprisingly, maybe there is not enough data transferred in a typical SSH session to trigger the problem

 

Update 2:

armbianmonitor -u finally finished on one of the devices. It took almost 3 hours for armbianmonitor -u to complete:  http://ix.io/1Pa4

 

Update 3:

Tested an Odroid-C2 board (S905) and the problem is still there

 

Are there any known RNG issues on the S905 and its variants? This is starting to look like a lack of available entropy for the RNG system

[[Fixed] Random pauses on S905 / X based hardware]

Hi All

 

I started playing with Armbian recently with some old S905 / S905X based TV Box devices I had lying around. Im working with a T95 S1, an A95X and an MXQ Pro and using the images from @balbes150 yandisk repo (thanks!)

 

Mostly everything seems to work fine except when transferring data to or from the devices (I havent tested WiFi or BT). My problem is that I cant use the ethernet to transfer any files. Network data transfers start as expected then get gradually slower and slower until after 10-15 seconds at which point data transfer stops and the device becomes unresponsive for a few minutes. Eventually I can SSH back into the device but any attempts at transferring data over the network fail as before. I have tried a USB ethernet adaptor which is normally used quite heavily on an RK3399 based device and works fine on the RK3399 armbian build but the results were the same which probably rules out a network driver issue. Anybody seen anything like this before?

 

One other thing I noticed was that all the devices had the same ethernet MAC addresses after flashing the @balbes150 images (00:15:18:01:81:31) which was strange and obviously caused a lot of network problems initially when they are all on the same LAN subnet. Is this hardcoded in the images or is it a quirk of S905 based hardware?

 

I get the same results on all 3 devices. I cant post an armbianmonitor -u because it hangs when uploading to xi.xo. Is there a way to get the local output and manually push it up to xi.xo?

 

Thanks

qstaq