qstaq

I have a number of headless AML S905 / S912 1GB Ram TV boxes running a bastardised Armbian for systems monitoring. I have a 200MB zram compressed block device setup to store temporary data which gets processed locally then synced to a command server every hour because the eMMC's on these cheap boxes do not last very long under sustained write scenarios. I would like to increase the zram filesystem size to 500MB so I only have to sync daily instead of hourly but because 25% of the RAM is allocated to the GPU/VPU I dont have enough available ram All the devices I have seen seem to allocate approx 250MB for the GPU which is obviously wasted in a headless scenario. I would like to reduce this to 32MB or 64MB which would give enough system RAM left over for a 500MB zram allocation Can anyone point me in the right direction of what I need to add or modify in the DTS files? So far my digging has me here: Im guessing this is or can be accomplished through modifying the DTB's for these devices but having looked through the DTS source files for various AML S9xx and RK33xx boxes I cant find anywhere where the GPU RAM allocation is defined. Im a complete noob when it comes to device trees so I have been studying @balbes150 Amlogic_s905-kernel github repo Using a Nexbox A95X as an example I can see that the dtb image gets built from the source code of the following dts files: https://github.com/150balbes/Amlogic_s905-kernel/tree/master/arch/arm64/boot/dts/amlogic meson-gxl-s905x-nexbox-a95x.dts meson-gxl-s905x.dtsi meson-gxl.dtsi meson-gx.dtsi meson-gxl-mali.dtsi Potentially Interesting looking entries: In the meson-gxl-s905x-nexbox-a95x.dts file I found the following, but modifying it seems to just reduce total RAM available to the SoC memory@0 { device_type = "memory"; reg = <0x0 0x0 0x0 0x80000000>; }; In meson-gx.dtsi I found vdec and vpu declarations. They seem to declare 5 separate memory regions but nothing seems to match the size of what has been reserved vdec: video-decoder@c8820000 { compatible = "amlogic,gx-vdec"; reg = <0x0 0xc8820000 0x0 0x10000>, <0x0 0xc110a580 0x0 0xe4>; reg-names = "dos", "esparser"; interrupts = <GIC_SPI 44 IRQ_TYPE_EDGE_RISING>, <GIC_SPI 32 IRQ_TYPE_EDGE_RISING>; interrupt-names = "vdec", "esparser"; amlogic,ao-sysctrl = <&sysctrl_AO>; amlogic,canvas = <&canvas>; }; vpu: vpu@d0100000 { compatible = "amlogic,meson-gx-vpu"; reg = <0x0 0xd0100000 0x0 0x100000>, <0x0 0xc883c000 0x0 0x1000>, <0x0 0xc8838000 0x0 0x1000>; reg-names = "vpu", "hhi", "dmc"; interrupts = <GIC_SPI 3 IRQ_TYPE_EDGE_RISING>; #address-cells = <1>; #size-cells = <0>; amlogic,canvas = <&canvas>; /* CVBS VDAC output port */ cvbs_vdac_port: port@0 { reg = <0>; }; /* HDMI-TX output port */ hdmi_tx_port: port@1 { reg = <1>; hdmi_tx_out: endpoint { remote-endpoint = <&hdmi_tx_in>; }; }; };

It works fine with @balbes150 h96max dtb Im using one as a nextcloud server. Even wireless and bluetooth work if you copy the firmware from /usr/lib/firmware/rkwifi to /usr/lib/firmware/bcrm I havent figured out booting from eMMC yet so I have u-boot / kernel on an SD card and the root fs on the eMMC. Or you can just boot and run direct from SD. Im using a 256GB SATA SSD on the USB C port and a 12TB hard disk on the USB 3 A port. Performance is pretty good Try @balbes150 RK3399 Armbian 5.91 TV Box image, it works great. The only issue I found was that there was no display on HDMI initially, the console was set to go to a serial debug port that doesn't exist on that device. After you write the SD card image, mount the boot partition first and modify the extlinux.conf file and add "console=tty1" to get a tty over HDMI and "console=ttyFIQ0,1500000n8" to get a serial console on the debug port. My extlinux.conf file for the X99 is: LABEL Armbian LINUX /Image INITRD /uInitrd FDT /rk3399-h96max.dtb APPEND root=/dev/mmcblk1p1 rootflags=data=writeback rw console=tty1 console=ttyFIQ0,1500000n8 no_console_suspend consoleblank=0 fsck.fix=yes fsck.repair=yes net.ifnames=0 DTB that Im using attached rk3399-h96max.dtb

Updated driver / kernel works perfect with both of my rk3399 TV boxes, Orange Pi RK3399 and my OP1 SoC ASUS chromebook

tvbox2.lan: aarch64, H96MaxTVBox-RK3399, USB RTL8156 (RTL driver), Custom userland, Kernel 4.4-rk Test 5: Netcat UDP, tvbox2.lan --> faphq.lan root@tvbox2:~# dd if=/dev/zero bs=100M count=10 | nc -v -u faphq.lan 9999 Connection to faphq.lan 9999 port [udp/*] succeeded! 10+0 records in 10+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 10.112 s, 99.7 MB/s Test 6: Netcat UDP, faphq.lan --> tvbox2.lan root@faphq:~# dd if=/dev/zero bs=100M count=10 | nc -v -u tvbox2.lan 9999 Connection to tvbox2.lan 9999 port [udp/*] succeeded! 10+0 records in 10+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 10.109 s, 99.7 MB/s Using the 4.4-rk kernel and realtek driver I cant get it to connect at 2.5Gb at all I have also just done a quick test between 2 Linux Mint 19.1 x86_64 boxes with the RTL8156's and kernel 5.0 and surprisingly they connect at 2.5Gb line rate and data transfer speeds are as they should be: 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.9438 s, 273 MB/s Ran out of time for any further testing tonight sorry

Test 3: Netcat UDP, tvbox1.lan --> faphq.lan root@tvbox1:~# dd if=/dev/zero bs=100M count=10 | nc -v -u faphq.lan 9999 Connection to faphq.lan 9999 port [udp/*] succeeded! 10+0 records in 10+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 9.98239 s, 107 MB/s Test 4: Netcat UDP, faphq.lan --> tvbox1.lan root@faphq:~# dd if=/dev/zero bs=100M count=10 | nc -v -u tvbox1.lan 9999 Connection to tvbox1.lan 9999 port [udp/*] succeeded! 10+0 records in 10+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 23.008 s, 46.9 MB/s First test on the RK3399 device is roughly half the performance of the Jetson, again the recieve performance is much lower than the send. This is on kernel 4.19 (I already had a config I knew would work). Im compiling a new 5.2 kernel now so I can test like for like kernel versions for cdc_ncm Edit: Performance on Kernel 5.2 is identical to 4.19 +/- 2%. Same asynchronous transfer speed and still half the performance of the Jetson

Ive started some testing but there is clearly a problem with this NIC or driver. I will post the Jetson Nano results now as it will take me another hour or so to finish compiling and setting up tests for the rockchip TV Box based boards as I want to try an older kernel with the official realtek driver. Im not even bothering with iperf yet Test environment and server testing against is as follows: Network: Netgear XS712T-200NES 10GB switch Config: Everything as defaults, No cpu affinity faphq.lan: x86_64, i9-7980XE, Intel 10GB ethernet, Linux Mint 19.1, Kernel 5.0 (Home Server) Clients config: jetson.lan: aarch64, Jetson Nano, USB RTL8156 (cdc_ncm), Ubuntu 18.04, Kernel 5.2 tvbox1.lan: aarch64, H96MaxTVBox-RK3399, USB RTL8156 (cdc_ncm), Custom userland, Kernel 4.19 tvbox2.lan: aarch64, H96MaxTVBox-RK3399, USB RTL8156 (RTL driver), Custom userland, Kernel 4.4-rk Test 1: Netcat UDP - jetson.lan --> faphq.lan root@jetson:~# dd if=/dev/zero bs=100M count=10 | nc -v -u faphq.lan 9999 Connection to faphq.lan 9999 port [udp/*] succeeded! 10+0 records in 10+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.4171 s, 244 MB/s Test 2: Netcat UDP - faphq.lan --> jetson.lan root@faphq:~# dd if=/dev/zero bs=100M count=10 | nc -v -u jetson.lan 9999 Connection to jetson.lan 9999 port [udp/*] succeeded! 10+0 records in 10+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 10.980 s, 98.2 MB/s Edit: On the jetson it automatically negotiated the correct 2.5Gb link speed, not 1Gb but obviously there is a huge difference in performance depending on the direction

It doesnt seem related to cpu affinity from your output. Its certainly not entropy. Very strange behaviour. Did you try using netcat to get a baseline outside of iperf? I actually stopped by at CPC on my way home today and picked up a couple of the PSG91497 Pro signal adaptors you have. They seemed too cheap to be true! I dont have any RK devices with USB3 unfortunately, apart from Android TV boxes so I cant really do an RK specific test (unless I can find an armbian or libreelec build for them). I will try them tonight on a Jetson Nano and any other USB3 arm devices I have at home

You probably dont need haveged and your current entropy pool is fine and full but if it empties when iperf is running then it might temporarily block random traffic generation Also iperf does sometime give unexpected results when using "-R" as opposed to "-s", just listening as a server so it might be worthwhile using netcat to get an alternative baseline and see what performance you get firing a 100MB+ file over it Im not concerned about the cdc_ncm usb0: network connection: connected syslog flood, as long as it doesnt have disconnects. The flooding is seen on all RTL8156 USB devices I have used when using cdc_ncm as opposed to the Realtek driver. For kernels >4.15 cdc_ncm is the only option anyway

The imbalance in directional performance can be caused by quite a few different things, including bad cables still (I know it shouldnt according to spec, but on real equipment, especially cheaper stuff it does) Can you do the same tests, configured as follows, to give me a better idea of whats going on please? Connect the 2 RTL8156 nics directly with a single cable (The RTL8156 is MDI / MDI-X compliant so it should automatically go into crossover mode without needing a 4 pair crossover cable) Test with reversed cable direction (I know this seems stupid but humour me) Force the RTL8156 on the RockPi side into 1GB Before testing make sure that haveged is started and running and while the tests are running can you watch for: CPU use and specifically look to see if 1 particular CPU core maxes out dmesg for cdc_ncm disconnects (how many?) dmesg for "entropy", "CRNG" or "crng" (less likely)

Looks like a faulty cable from the inconsistent results pattern, especially the retries I see this same pattern quite a lot with 2.5/10/40GBe devices. The driver thinks the ethernet cable is constantly up and down, many times a second sometimes Can you grep dmesg output for "cdc_ncm" and see if you have lots of disconnects and reconnects. If so its almost certainly the cable

Armbian version and base? Sunxi or mainkine kernel? What scripting languages are you familiar with?

Use Linphone as the sip client controlled via the linphonec command line interface. Use bash, python, php or whatever you are comfortable with to script linphonec and interact with the GPIO's. Its pretty simple

Yes it was all caused by lack of entropy. And yes your fix of modifying /etc/default/haveged is better than my fix of modifying the systemd unit file Just set the DAEMON_ARGS="-w 1024 --data=16" in /etc/default/haveged for Kernels >5

Fixed. Ill post the solution so if anyone else is having the same problem they can fix it Basically haveged is not starting because it cant detect the CPU cache size and so cant feed itself all the variables it needs to start up Looking through the armbianmonitor output I noticed that lscpu / cpuid is unable to detect the CPU cache sizes at all #lscpu: Architecture: aarch64 Byte Order: Little Endian CPU(s): 4 On-line CPU(s) list: 0-3 Thread(s) per core: 1 Core(s) per socket: 4 Socket(s): 1 Vendor ID: ARM Model: 4 Model name: Cortex-A53 Stepping: r0p4 CPU max MHz: 1512.0000 CPU min MHz: 100.0000 BogoMIPS: 48.00 L1d cache: unknown size L1i cache: unknown size L2 cache: unknown size Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid I think the S905 series has a 16kb L1 cache (correct me if Im wrong) so changing the haveged systemd unit to hard code a 16kb data size fixes the problem. Performance of encrypted traffic closely matches that of non encrypted traffic now Fix is simple: just add --data=16 to the systemd unit arguments on the daemon start line Modify /etc/default/haveged to include --data=16 on the DAEMON_ARGS line # Configuration file for haveged # Options to pass to haveged: # -w sets low entropy watermark (in bits) DAEMON_ARGS="-w 1024 --data=16" Edited fix as @amirul suggested the correct way to modify haveged args

Update 1: Done some more extensive testing using different network protocols. The problem only happens with encrypted data traffic. Non encrypted traffic actually flows normally and without any problems HTTP, FTP, POP, IMAP all work fine. HTTPS, SCP, Samba, Wireguard, OpenVPN all fail. SSH actually works fine surprisingly, maybe there is not enough data transferred in a typical SSH session to trigger the problem Update 2: armbianmonitor -u finally finished on one of the devices. It took almost 3 hours for armbianmonitor -u to complete: http://ix.io/1Pa4 Update 3: Tested an Odroid-C2 board (S905) and the problem is still there Are there any known RNG issues on the S905 and its variants? This is starting to look like a lack of available entropy for the RNG system

Hi All I started playing with Armbian recently with some old S905 / S905X based TV Box devices I had lying around. Im working with a T95 S1, an A95X and an MXQ Pro and using the images from @balbes150 yandisk repo (thanks!) Mostly everything seems to work fine except when transferring data to or from the devices (I havent tested WiFi or BT). My problem is that I cant use the ethernet to transfer any files. Network data transfers start as expected then get gradually slower and slower until after 10-15 seconds at which point data transfer stops and the device becomes unresponsive for a few minutes. Eventually I can SSH back into the device but any attempts at transferring data over the network fail as before. I have tried a USB ethernet adaptor which is normally used quite heavily on an RK3399 based device and works fine on the RK3399 armbian build but the results were the same which probably rules out a network driver issue. Anybody seen anything like this before? One other thing I noticed was that all the devices had the same ethernet MAC addresses after flashing the @balbes150 images (00:15:18:01:81:31) which was strange and obviously caused a lot of network problems initially when they are all on the same LAN subnet. Is this hardcoded in the images or is it a quirk of S905 based hardware? I get the same results on all 3 devices. I cant post an armbianmonitor -u because it hangs when uploading to xi.xo. Is there a way to get the local output and manually push it up to xi.xo? Thanks qstaq