qstaq Posted August 8, 2019 Posted August 8, 2019 I have a number of headless AML S905 / S912 1GB Ram TV boxes running a bastardised Armbian for systems monitoring. I have a 200MB zram compressed block device setup to store temporary data which gets processed locally then synced to a command server every hour because the eMMC's on these cheap boxes do not last very long under sustained write scenarios. I would like to increase the zram filesystem size to 500MB so I only have to sync daily instead of hourly but because 25% of the RAM is allocated to the GPU/VPU I dont have enough available ram All the devices I have seen seem to allocate approx 250MB for the GPU which is obviously wasted in a headless scenario. I would like to reduce this to 32MB or 64MB which would give enough system RAM left over for a 500MB zram allocation Can anyone point me in the right direction of what I need to add or modify in the DTS files? So far my digging has me here: Im guessing this is or can be accomplished through modifying the DTB's for these devices but having looked through the DTS source files for various AML S9xx and RK33xx boxes I cant find anywhere where the GPU RAM allocation is defined. Im a complete noob when it comes to device trees so I have been studying @balbes150 Amlogic_s905-kernel github repo Using a Nexbox A95X as an example I can see that the dtb image gets built from the source code of the following dts files: https://github.com/150balbes/Amlogic_s905-kernel/tree/master/arch/arm64/boot/dts/amlogic meson-gxl-s905x-nexbox-a95x.dts meson-gxl-s905x.dtsi meson-gxl.dtsi meson-gx.dtsi meson-gxl-mali.dtsi Potentially Interesting looking entries: In the meson-gxl-s905x-nexbox-a95x.dts file I found the following, but modifying it seems to just reduce total RAM available to the SoC memory@0 { device_type = "memory"; reg = <0x0 0x0 0x0 0x80000000>; }; In meson-gx.dtsi I found vdec and vpu declarations. They seem to declare 5 separate memory regions but nothing seems to match the size of what has been reserved vdec: video-decoder@c8820000 { compatible = "amlogic,gx-vdec"; reg = <0x0 0xc8820000 0x0 0x10000>, <0x0 0xc110a580 0x0 0xe4>; reg-names = "dos", "esparser"; interrupts = <GIC_SPI 44 IRQ_TYPE_EDGE_RISING>, <GIC_SPI 32 IRQ_TYPE_EDGE_RISING>; interrupt-names = "vdec", "esparser"; amlogic,ao-sysctrl = <&sysctrl_AO>; amlogic,canvas = <&canvas>; }; vpu: vpu@d0100000 { compatible = "amlogic,meson-gx-vpu"; reg = <0x0 0xd0100000 0x0 0x100000>, <0x0 0xc883c000 0x0 0x1000>, <0x0 0xc8838000 0x0 0x1000>; reg-names = "vpu", "hhi", "dmc"; interrupts = <GIC_SPI 3 IRQ_TYPE_EDGE_RISING>; #address-cells = <1>; #size-cells = <0>; amlogic,canvas = <&canvas>; /* CVBS VDAC output port */ cvbs_vdac_port: port@0 { reg = <0>; }; /* HDMI-TX output port */ hdmi_tx_port: port@1 { reg = <1>; hdmi_tx_out: endpoint { remote-endpoint = <&hdmi_tx_in>; }; }; };
qstaq Posted August 8, 2019 Author Posted August 8, 2019 A more interesting section in meson-gx.dtsi: reserved-memory { #address-cells = <2>; #size-cells = <2>; ranges; /* 16 MiB reserved for Hardware ROM Firmware */ hwrom_reserved: hwrom@0 { reg = <0x0 0x0 0x0 0x1000000>; no-map; }; /* 2 MiB reserved for ARM Trusted Firmware (BL31) */ secmon_reserved: secmon@10000000 { reg = <0x0 0x10000000 0x0 0x200000>; no-map; }; /* Alternate 3 MiB reserved for ARM Trusted Firmware (BL31) */ secmon_reserved_alt: secmon@5000000 { reg = <0x0 0x05000000 0x0 0x300000>; no-map; }; /* 32 MiB reserved for ARM Trusted Firmware (BL32) */ secmon_reserved_bl32: secmon@5300000 { reg = <0x0 0x05300000 0x0 0x2000000>; no-map; }; linux,cma { compatible = "shared-dma-pool"; reusable; size = <0x0 0x38000000>; alignment = <0x0 0x400000>; linux,cma-default; }; }; I presume this is reserving 53MB for ROM and ATF? The linux cma section seems to be declaring 896MB of shared RAM but I dont understand what the alignment declaration does and none of these numbers match up with what seems to be allocated for the GPU
hexdump Posted August 8, 2019 Posted August 8, 2019 i was looking into this a while ago for a s905w tv box and found out, that a lot of memory seems to be blocked already by legacy u-boot etc. (about 120m) and you cannot get it back by changes on linux kernel side. what helps is to chainload mainline u-boot (just build the libretech-cc u-boot from 2019.07 and use the resulting u-boot.bin): fatload mmc 0 0x01000000 u-boot.bin go 0x01000000 in legacy u-boot and then you can use regular boot.scr or extlinux to boot further from there using the chainloaded u-boot just booted. good luck and best wishes - hexdump
jock Posted August 12, 2019 Posted August 12, 2019 On 8/8/2019 at 3:32 AM, qstaq said: I presume this is reserving 53MB for ROM and ATF? The reg property is used to declare where the device have their memory-mapped region to interact with the operating system. You can imagine that devices (like the ethernet, the gpu, etc...) have one or more "windows" in the virtual address space of the CPU. Normally these windows are fixed in the address space, and the device tree declares where those windows are, letting the operating system drivers know where to access the devices. In case of reserved-memory section, I see no real drivers involved there, so I guess that the memory is already reserved by bootloaders (the amlogic proprietary bootloader, mainly) and the declarations there inform the kernel not to use those locations because they are reserved. You could remove the reserved sections, probably the kernel will show that more memory is available, but you may also expect malfunctions because some application may write in those reserved locations causing unpredictable behaviour! Also check the size of the reserved CMA block (dmesg | grep cma). It should be possible to resize it via extraargs in armbianEnv.txt or recompiling the kernel. 8 megabytes there should suffice. edit: I checked on a S905 device, the CMA can be changed only editing the device tree and should not be less than 32M, otherwise X fails to start on my configuration.
qstaq Posted August 15, 2019 Author Posted August 15, 2019 Thanks @jock I have created a new dtb with the following change: linux,cma { compatible = "shared-dma-pool"; reusable; size = <0x0 0x2000000>; alignment = <0x0 0x400000>; linux,cma-default; }; This should reduce the cma size to 32MB (0x2000000) if I understand correctly? However even with that change I still see "cma: Reserved 256 MiB" in dmesg, not 32MB as I would expect. I have also tried various different "cma=x" values on the kernel command line but the argument seems to be ignored Have I misunderstood something?
jock Posted August 15, 2019 Posted August 15, 2019 10 minutes ago, qstaq said: Thanks @jock I have created a new dtb with the following change: linux,cma { compatible = "shared-dma-pool"; reusable; size = <0x0 0x2000000>; alignment = <0x0 0x400000>; linux,cma-default; }; This should reduce the cma size to 32MB (0x2000000) if I understand correctly? However even with that change I still see "cma: Reserved 256 MiB" in dmesg, not 32MB as I would expect. I have also tried various different "cma=x" values on the kernel command line but the argument seems to be ignored Have I misunderstood something? Yes it should work that way. At least, it worked on my S905 machine. Didn't you forget to recompile the dts back to dtb, did you? I'm not sure, but the reusable flag there should allow the kernel to use the same CMA region for its own needs, so it should act totally as a shared dynamic memory. You can anyway take a look to the official documentation becuase I may miss something: https://www.kernel.org/doc/Documentation/devicetree/bindings/reserved-memory/reserved-memory.txt
qstaq Posted August 15, 2019 Author Posted August 15, 2019 Im de-compiling the existing dbt with: dtc -I dtb -O dts meson-gxl-s905x-nexbox-a95x.dtb > custom.dts then modifying custom.dts and re-compiling with: dtc -O dtb -o custom.dtb custom.dts Then changing uEnv.ini to point at custom.dts Edit: Working Now OK so setting the FDT option in uEnv.ini doesnt do anything on S905x devices, you have to change the FDT in extlinux.conf instead (balbes images)
qstaq Posted August 15, 2019 Author Posted August 15, 2019 On 8/8/2019 at 4:48 PM, hexdump said: i was looking into this a while ago for a s905w tv box and found out, that a lot of memory seems to be blocked already by legacy u-boot etc. (about 120m) and you cannot get it back by changes on linux kernel side. what helps is to chainload mainline u-boot (just build the libretech-cc u-boot from 2019.07 and use the resulting u-boot.bin): fatload mmc 0 0x01000000 u-boot.bin go 0x01000000 in legacy u-boot and then you can use regular boot.scr or extlinux to boot further from there using the chainloaded u-boot just booted. good luck and best wishes - hexdump Hi @hexdump I have built the libretech-cc u-boot and copied it to my /boot fs. How do I instruct legacy u-boot, installed on an sd card, to chainload the libretech u-boot? Thanks for the help
hexdump Posted August 16, 2019 Posted August 16, 2019 by just putting the above lines into your s905_autoboot.cmd and creating a fresh s905_autoboot from it. then it will on boot start the mainline u-boot instead of the kernel. and you need a working extlinux/extlinux.conf file then which tells the mainline what to do. good luck - hexdump
Guest Posted September 7, 2019 Posted September 7, 2019 On 8/8/2019 at 11:48 PM, hexdump said: i was looking into this a while ago for a s905w tv box and found out, that a lot of memory seems to be blocked already by legacy u-boot etc. (about 120m) and you cannot get it back by changes on linux kernel side. what helps is to chainload mainline u-boot (just build the libretech-cc u-boot from 2019.07 and use the resulting u-boot.bin): fatload mmc 0 0x01000000 u-boot.bin go 0x01000000 in legacy u-boot and then you can use regular boot.scr or extlinux to boot further from there using the chainloaded u-boot just booted. good luck and best wishes - hexdump I'm trying to get this to work but the board will always reboot itself with this error: gxl_p230_v1#fatload usb 0 0x01000000 u-boot.bin reading u-boot.bin 851968 bytes read in 43 ms (18.9 MiB/s) gxl_p230_v1#go 0x01000000 ## Starting application at 0x01000000 ... "Synchronous Abort" handler, esr 0x02000000 ELR: 1000000 LR: 77ee4824 x0 : 0000000000000001 x1 : 0000000073ede1b8 x2 : 0000000073ede1b8 x3 : 0000000001000000 x4 : 0000000000000020 x5 : 000000008c137742 x6 : 0000000077f41000 x7 : 000000000000000f x8 : 0000000077f5f748 x9 : 0000000000000000 x10: 00000000003e0000 x11: 0000000000010000 x12: 000000000000000d x13: 0000000000000000 x14: 0000000000000000 x15: 0000000000000000 x16: 0000000000000000 x17: 0000000000000000 x18: 0000000073ec8e28 x19: 0000000073ede1b8 x20: 0000000000000002 x21: 0000000001000000 x22: 0000000000000002 x23: 0000000077f71ef8 x24: 0000000000000000 x25: 0000000073ee4800 x26: 0000000000000000 x27: 0000000000000000 x28: 0000000000000000 x29: 0000000073ec8b80 Resetting CPU ... resetting ... @hexdump The board is a s905d TV box, and I built the u-boot binaries with this tutorial: http://wiki.loverpi.com/faq:sbc:libre-aml-s805x-howto-compile-u-boot The "Synchronous Abort" handler always appear whenever I try to boot just about anything. I'm trying to get Fedora on this box, and possibly even UEFI booting, but the stock u-boot just won't let me test anything. Any suggestions on how I should proceed?
hexdump Posted September 7, 2019 Posted September 7, 2019 @utgf - maybe try the attached u-boot.bin (after gunzipping it) - if that does not work, then maybe do a printenv on the legacy u-boot prompt - maybe it uses another address (your should find some start address in there) best wishes - hexdump u-boot.bin-libretech-cc.gz
Recommended Posts