Jump to content

CSC Armbian for RK322x TV box boards


jock

Recommended Posts

6 hours ago, jock said:

 

Hello @Seth, I inspected your device tree and it looks like you have a led which is connected to a GPIO pin which is different from the two existing configurations. I can arrange a device tree overlay if you're interested in getting support for the second led too.

 

@jock

yes please. and can i have a walkthrough on the differences are and how you fixed the leds. i'm currently working on reversing an antenna for 2.4GHz for use with the box, well at least the usb wifi i'm using on it. i'm planning on making one made up of more segments to increase gain as i will make this my main access point (and mqtt server) at home for solar IoT projects.

IMG_20210220_123322~3_exported_5900008750129762049.jpg

Link to comment
Share on other sites

@Seth Here it is the device tree overlay: rk322x-led-conf3.dtbo

 

Put the file in /boot/dtb/overlays directory and then enable it editing in /boot/armbianEnv.txt (just replace the existing led-conf1 with led-conf3), and then reboot.

 

By default the other led is configured to blink during eMMC access, but you can change the behaviour following the instructions of the previous post: https://forum.armbian.com/topic/12656-csc-armbian-for-rk322x-tv-boxes/?do=findComment&comment=119748

 

About the walkthrough, well I converted the device tree filesystem structure into dts with:

dtc -I fs -O dts device-tree-copy -o device-tree.dts

then looked into device-tree.dts to find out the gpio-leds compatible string, and there you will find this section:

	power-led {
                status = "disabled";
                compatible = "gpio-leds";

                red {
                        gpios = <0x75 0x08 0x00>;
                        default-state = "off";
                };

                green {
                        gpios = <0xa9 0x15 0x00>;
                        default-state = "on";
                };
        };

 

The "green" led is attached to the GPIO bank whose phandle is 0xa9 (following the phandle it will be GPIO bank #3), and the pin is 0x15 (21 in decimal, or RK_PC5 as rockchip constant from source code) - this is the default led which is attached this way on all the boxes no matter the manufacturer.

The "red" led is attached to GPIO bank #0 and the pin is 0x8 (RK_PB0 as rockchip constant).

 

This commit contains the changes to add the device tree overlay led-conf3 to armbian.

 

Link to comment
Share on other sites

@dale

since you are able to manage linux and linux command , you can enjoy yourself with a chron command too:
a simple bash script that in the morning at 8:00 ( or when is usefull for you) does a

echo default-on ( ot timer or heartbeat or cpu0 ) >>> /sys/class/leds/here_the_name_of_your_led/trigger 

 

and the at 20:00 ( or when is usefull for you) does a

echo none ( ATTENTION: none and default on " could be inverted as logical but you will realize it by yourself ;-)  ) >>>  /sys/class/leds/put_here_your_led_name/trigger

 

If you are electronic skilled, too, you can even control something with a simple transistor and a relay attached to that led ;)
 

Link to comment
Share on other sites

Hi, i tried to install to nand (Debian, Ubuntu, LE) using multitool but never boots, then i erased the nand with rkdevelop and tried to use the loader i find on the first page, but multitool stopped recognizing the nand

What i can do?

Rk3228A, Mxq Pro 4k, ssv6051p

Edited by rlukas210
forgot some info
Link to comment
Share on other sites

17 hours ago, rlukas210 said:

Hi, i tried to install to nand (Debian, Ubuntu, LE) using multitool but never boots, then i erased the nand with rkdevelop and tried to use the loader i find on the first page, but multitool stopped recognizing the nand

What i can do?

Rk3228A, Mxq Pro 4k, ssv6051p

Erasing the NAND with rkdeveloptool was not a smart move, as you noticed right now the NAND is not detected anymore.

First of all, be aware that everytime you reboot the box, unplug the power cord and detach the serial connectors from the board if you have them attached: NAND boxes are very sensitive to even the lowest current.

As long as you tried both Armbian and LibreELEC, you should specify what image you did try: forget the mainline kernel images, they won't work with NAND because there is no driver. You must use legacy 4.4 kernel images.

 

From my point of view, as long as you erased the flash (thus removed the original idbloader), you may do something smarter and use the sdcard to understand if legacy kernel Armbian/LE boots from sdcard or not.

If it is not booting, you need to attach a serial adapter and take some logs because something wrong is happening and the kernel is not able to boot.

Note that probably you will need to erase the flash again if you installed a non working idbloader to make the board boot from sdcard without interferences.

 

Instead, if Armbian or LE are booting fine from sdcard, proceeding to restoring the NAND functionality can happen.

 

As @fabiobassa said, photos and signatures of the board as useful to understand if it is a known board or similar. I have an MXQ Pro 4K tv box right here and it looks similar to yours, but the commercial name is worthless since the board inside may be of a different revision with different components.

Link to comment
Share on other sites

8 hours ago, jock said:

Erasing the NAND with rkdeveloptool was not a smart move, as you noticed right now the NAND is not detected anymore.

First of all, be aware that everytime you reboot the box, unplug the power cord and detach the serial connectors from the board if you have them attached: NAND boxes are very sensitive to even the lowest current.

As long as you tried both Armbian and LibreELEC, you should specify what image you did try: forget the mainline kernel images, they won't work with NAND because there is no driver. You must use legacy 4.4 kernel images.

 

From my point of view, as long as you erased the flash (thus removed the original idbloader), you may do something smarter and use the sdcard to understand if legacy kernel Armbian/LE boots from sdcard or not.

If it is not booting, you need to attach a serial adapter and take some logs because something wrong is happening and the kernel is not able to boot.

Note that probably you will need to erase the flash again if you installed a non working idbloader to make the board boot from sdcard without interferences.

 

Instead, if Armbian or LE are booting fine from sdcard, proceeding to restoring the NAND functionality can happen.

 

As @fabiobassa said, photos and signatures of the board as useful to understand if it is a known board or similar. I have an MXQ Pro 4K tv box right here and it looks similar to yours, but the commercial name is worthless since the board inside may be of a different revision with different components.

I got the multitool to work again (reinstall stock Android) , and It recognize nand again, but i already tried Installing armbian using multitool (stepnand, i forgot to say before), tried with rkdevelop too (with 4mb stripped as i read on older posts), but no sucess

Installing It directly to sd card work well, but my old sdcard broke and the actual is only 4gb, and nand is 8gb

 

@fabiobassa @jock I'll atach the board photos, some logs about nand i got from rkdeveloptool and the stock ROM i used before below:

https://bit.ly/3kvflrj

Link to comment
Share on other sites

14 hours ago, rlukas210 said:

I got the multitool to work again (reinstall stock Android) , and It recognize nand again, but i already tried Installing armbian using multitool (stepnand, i forgot to say before), tried with rkdevelop too (with 4mb stripped as i read on older posts), but no sucess

Installing It directly to sd card work well, but my old sdcard broke and the actual is only 4gb, and nand is 8gb

 

@fabiobassa @jock I'll atach the board photos, some logs about nand i got from rkdeveloptool and the stock ROM i used before below:

https://bit.ly/3kvflrj

I'm a bit confused.

Did you follow my advice to try testing via sdcard with empty NAND and it worked, right?

Are you sure you are using a legacy kernel 4.4 Armbian image?

 

steP-nand is the only way to install a functional Armbian on NAND; the older post about "stripping 4mb" required more actions that have to be taken before and alone won't work.

LibreELEC instead must be installed using "Burn image to flash" Multitool menu option.

 

Can't say anything more about, logs are not in the google drive directory and anyway rkdeveloptool logs are not so much interesting.

Link to comment
Share on other sites

Quote

Hardware video acceleration (fully supported via RKMPP on legacy kernel, partial support via hantro kernel driver on mainline)

Full GPU acceleration on legacy kernel, mainline kernel has lima driver compiled in but X11 is very slow - you are still free to compile and install mali kernel driver on mainline yourself.

 

Are these still valid, or do we have video and gpu acceleration on mainline now?

 

Also, any chance to get RTL8703 Wifi (from tanix tx2) working? Maybe adapting the drivers from the android firmware?

Link to comment
Share on other sites

9 hours ago, chaiz said:

 

Are these still valid, or do we have video and gpu acceleration on mainline now?

 

Also, any chance to get RTL8703 Wifi (from tanix tx2) working? Maybe adapting the drivers from the android firmware?

Hello,

video acceleration in on the way to go, mainline image already has all the bits in place for h.264 hardware video decoding.

You sill need to compile ffmpeg and Kodi from libreelec suggested sources.

Mpv was an easier subject to compile and use other than Kodi, but it does not work yet (I filed an issue on github but got no attention: https://github.com/mpv-player/mpv/issues/8522)

Nonetheless it works pretty fine.

 

GPU acceleration is already there, Lima driver is albeit generally very slow; at the moment is disabled in X11 via /etc/X11/xorg.conf (otherwise the desktop experience is way too slow).

rtl8703 should work with 8723cs driver; last time I tried with legacy kernel and it worked, but actually I don't know the current state.

 

Link to comment
Share on other sites

On 3/7/2021 at 8:05 AM, jock said:

Hello,

video acceleration in on the way to go, mainline image already has all the bits in place for h.264 hardware video decoding.

You sill need to compile ffmpeg and Kodi from libreelec suggested sources.

Mpv was an easier subject to compile and use other than Kodi, but it does not work yet (I filed an issue on github but got no attention: https://github.com/mpv-player/mpv/issues/8522)

Nonetheless it works pretty fine.

 

GPU acceleration is already there, Lima driver is albeit generally very slow; at the moment is disabled in X11 via /etc/X11/xorg.conf (otherwise the desktop experience is way too slow).

rtl8703 should work with 8723cs driver; last time I tried with legacy kernel and it worked, but actually I don't know the current state.

 

 

Thanks!

I tried installing kodi on buster mainline, but it was painfully slow. Also tried installing the media solutions for 4.4 in both buster kernels, without sucess. Is there any way to make buster legacy work with kodi well? It would be perfect for me then.

 

I tried setting up focal legacy kernel to test the mods in the OP, but when i run rk322x-config and reboot it, it doesnt load the DE and gets stuck on a black screen. Even on a fresh install.

 

Only got wireless working on legacy buster, maybe legacy focal if i can solve the above issue.

 

Basically I want to setup a media box, using radarr, sonarr, torrent, etc and still have kodi in the device for media consumption (could work as libreelec, without a DE). I'm just not finding the right combination to do it (when kodi has worked, it was slow and videos were playing very chopped - something like 1 frame every 2 seconds).

Link to comment
Share on other sites

1 hour ago, chaiz said:

 

Thanks!

I tried installing kodi on buster mainline, but it was painfully slow. Also tried installing the media solutions for 4.4 in both buster kernels, without sucess. Is there any way to make buster legacy work with kodi well? It would be perfect for me then.

The media package use Ubuntu Focal with legacy kernel: I built it and tested only with Ubuntu Focal (either Desktop or minimal should work fine, on the download page you can download desktop flavour, if you want minimal you have to compile it yourself). Debian Buster probably has some packages that are too old and thus does not work. I didn't test it so can't say for sure.

The media package won't work with mainline kernel, it uses some rockchip things that are only in legacy 4.4 kernel.

 

1 hour ago, chaiz said:

I tried setting up focal legacy kernel to test the mods in the OP, but when i run rk322x-config and reboot it, it doesnt load the DE and gets stuck on a black screen. Even on a fresh install.

 

Try to select "emmc-nand" in the rk322x-config menu - that's odd because this is the second post in a few days about this issue: https://forum.armbian.com/topic/17215-rk3228-reboot-black-screen/

 

1 hour ago, chaiz said:

Only got wireless working on legacy buster, maybe legacy focal if i can solve the above issue.

realtek drivers are quite a mess, unfortunately I don't have a board with rtl8703 here with me so can't do continous testing and support.

 

1 hour ago, chaiz said:

Basically I want to setup a media box, using radarr, sonarr, torrent, etc and still have kodi in the device for media consumption (could work as libreelec, without a DE). I'm just not finding the right combination to do it (when kodi has worked, it was slow and videos were playing very chopped - something like 1 frame every 2 seconds).

Ubuntu Focal with legacy kernel still works best. Hopefully things will be stable and upstreamed for mainline kernel + ffmpeg + kodi + mpv: it will far better in terms of interoperability, even because legacy kernel and legacy media acceleration was not intended to work with X11 and thus there are many limitations.

Link to comment
Share on other sites

On 22.1.2021 at 2:28 PM, jock said:

@Alex83 can't really say what is broken with your configuration.

ssv6051p received little to nothing changes, except for an optimization compile flag and the disable of the p2p interface which was causing reboots and crashes and slowing down the connection.

Slow boot times may be due to many things. Not able to connect is one of them, but can be a faulty NAND, a network device mount that is not waiting for timeout, etc... etc...

Try to do apt update && apt upgrade to get the latest packages up to date.

Post armbianmonitor -u link the help in troubleshooting

Hey @jock it has been a while, but now I found time again to reinstall completely. 

 

Actually after a fresh install of armbian 21.02.3 the first start worked and I was able to reactivate wifi ssv6051 again. 

 

Unfortunately there is still no possible reboot. So if device needs a reboot I need to unplug and plug in the power adapter and it's impossible for remote task... 

 

So after I did the 

 

Rk322x-config I got trouble to reboot into xserver. The second reboot went into the desktop, but after a few seconds the server crashed, before activated wifi it stayed stable in desktop... Anyway I just want a console server, but didn't find a minimal Ubuntu version, so installed the desktop version and disabled it in armbian-config. 

 

So long story, short question... 

 

If there is no wifi traffic the device looses reachability... So I thought device hung, but after I checked again I saw if I send a ping again to the ip of my requesting device there is a connection possible again. 

 

So for me I get the feeling there is some mess with energy savings or similar stuff ongoing with my rk3229 V88 mini 4k device. 

 

Maybe you can help me with reboot and unreachability from remote devices... 

 

I already changed power supply, but didn't get a better situation... 

 

I tried also a new performance test. 

 

With iperf3 I get around 32 Mbit/s up and down with wifi@72Mbit/s.

 

http://ix.io/2SRB

Edited by Alex83
Put the output of armbianmonitor - u into this post
Link to comment
Share on other sites

Hello @Alex83, your issues about wifi are strange but indeed can happen.

 

I mean, I experienced crashes and such kind of issues with my boxes (2x ssv6051p and 1x ssv6256p), but now they are quite stable after disabling the p2p interface. The wifi kernel driver also does not help because it is quite messy and low-quality, so some corner cases may cause a kernel panic or crashes.

 

To be honest, I got the reachability issues often as you describe, but I think it is due to some slowness of the interface to answer to ARP requests. If it is so, the issue can be some kind of power management and probably it is controlled by the driver in some ways I actually don't know how to configure.

 

Actually the X server crashing is hardly related to wifi, but it may indeed happen.

I can propose you different steps trying to isolate the crashing issues. Proceed to the next step in case you still have instabilities.

 

 First step: to disable the possibile wifi power saving you can run

iwconfig wlan power off

but I don't remember if the driver allows power saving toggle. Just try and see what iwconfig tells you.

 

Second step: you can set the most conservative approach selecting the cpu model as rk3228a and the flash type as emmc-nand in rk322x-config, so the CPU frequency is set to max 1.2Ghz and the NAND is left to default settings.


Third step: if you still have issues with wifi, try to substitute /boot/dtb/rk322x-box.dtb with this one:rk322x-box.dtb

 

Fourth step: as a test to understand if the problem is really wifi, disable it at all and, if you can, use an USB dongle for some time.

 

About the reboot issue, recently me and @fabiobassainvestigated - again - a lot on the NAND subject. We still don't understand why this happens, but we got some interesting new knowledge about the rockchip way to handle NAND things. After many tests, which involved NAND "formatting" from scratch, I noticed that my board started to reboot without any issue.

In the past I had to detach everything from the board to properly reboot it, including the HDMI and USB-to-TTL serial adapter. Now reboot just works. :huh:

 

I can give some some instructions on how to do so, but you need a male-to-male USB cable. An USB-to-TTL serial cable is of great help for debug (but not really necessary) and you need to be confident in case you have to follow the unbrick procedure, because we're going to replace the bootloader with a newer one, which has some risk involved. Results are not guaranteed, you can see this as an experiment.

 

edit: aside from the bootloader upgrade, it just popped in my mind that you may want to download a new recent copy of the Multitool and reinstall the system via steP-nand: some time ago I updated the trustos binary with a newer and more compatible one. This may (or may not) "fix" the NAND reboot, but if you feel confident you may give a try.

Link to comment
Share on other sites

13 hours ago, jock said:

Hello @Alex83, your issues about wifi are strange but indeed can happen.

 

I mean, I experienced crashes and such kind of issues with my boxes (2x ssv6051p and 1x ssv6256p), but now they are quite stable after disabling the p2p interface. The wifi kernel driver also does not help because it is quite messy and low-quality, so some corner cases may cause a kernel panic or crashes.

 

To be honest, I got the reachability issues often as you describe, but I think it is due to some slowness of the interface to answer to ARP requests. If it is so, the issue can be some kind of power management and probably it is controlled by the driver in some ways I actually don't know how to configure.

 

Actually the X server crashing is hardly related to wifi, but it may indeed happen.

I can propose you different steps trying to isolate the crashing issues. Proceed to the next step in case you still have instabilities.

 

 First step: to disable the possibile wifi power saving you can run


iwconfig wlan power off

but I don't remember if the driver allows power saving toggle. Just try and see what iwconfig tells you.

 

Second step: you can set the most conservative approach selecting the cpu model as rk3228a and the flash type as emmc-nand in rk322x-config, so the CPU frequency is set to max 1.2Ghz and the NAND is left to default settings.


Third step: if you still have issues with wifi, try to substitute /boot/dtb/rk322x-box.dtb with this one:rk322x-box.dtb

 

Fourth step: as a test to understand if the problem is really wifi, disable it at all and, if you can, use an USB dongle for some time.

 

About the reboot issue, recently me and @fabiobassainvestigated - again - a lot on the NAND subject. We still don't understand why this happens, but we got some interesting new knowledge about the rockchip way to handle NAND things. After many tests, which involved NAND "formatting" from scratch, I noticed that my board started to reboot without any issue.

In the past I had to detach everything from the board to properly reboot it, including the HDMI and USB-to-TTL serial adapter. Now reboot just works. :huh:

 

I can give some some instructions on how to do so, but you need a male-to-male USB cable. An USB-to-TTL serial cable is of great help for debug (but not really necessary) and you need to be confident in case you have to follow the unbrick procedure, because we're going to replace the bootloader with a newer one, which has some risk involved. Results are not guaranteed, you can see this as an experiment.

 

edit: aside from the bootloader upgrade, it just popped in my mind that you may want to download a new recent copy of the Multitool and reinstall the system via steP-nand: some time ago I updated the trustos binary with a newer and more compatible one. This may (or may not) "fix" the NAND reboot, but if you feel confident you may give a try.

Hey Jock, 

 

Thanks again for your great response and of course to you and @fabiobassato your great and wonderful work to get finnaly a use case for this trash boxes!

 

I'm really thankful for this. 

 

I'll try your suggestions within the next couple of days, but just wanna add some more information. 

 

I have an permanent ping within 2 devices to keep the wifi busy, but now I noticed that from my smartphone there is no constant ping connection and then I only get a connection again if I send from V88 mini 4k a ping to my internal smartphone address... 

 

So it seems like it could also be some ip-layer or firewall topic. I haven't installed any ufw or other firewall. So I guess there should be no firewall working at the moment... 

 

Also I tried again a

 

Iperf3 localhost performance test. 

 

The result is ~630 Mbit/s. 

 

Compared to an orange Pi PC it has 2,5 Gbit/s. 

 

Both devices have 4 cores with around 1,4GHz.  Also if you look into htop I see 4 cores working on both devices, but in rk3229 there is just one time 1x1,4 GHz core in yellow shown as there are 4x1,4 GHz in Orange PI PC. 

 

So I think performance on rk3229 seems still pretty poor... 

 

 

Link to comment
Share on other sites

@Alex83
 

Quote

So I think performance on rk3229 seems still pretty poor.

 

Overall performances of rk322x are definetly good Alex, me and @jock have Nas, volumio, even openmediavault running on this trash box from several months without reboot or crashes.
Mine is doing even a hard routings because I host more tha 100 vpn's on this poor 3228A, the littlest child of family with just 1 gb ram.

But If you go on wifi yes, the driver is so crappy and any references to code non existent so no way to improve it.
This is and this will remain.
 

Quote

Compared to an orange Pi PC it has 2,5 Gbit/s. 


In one of my first answer to you I already said that you are comparing apples and pears but you continue to do comparing :-(
Raspberry and all the derivatives such as  orange pi banana bi and etc etc  has an official support of thousands of people and tons of opensource code.

Here at the moment we are speaking of perhaps 10 or few more  people actively spending days ( and night sometimes ) to bring improvements on rk322x but it comes a stop when NO  code source is accessible for further investigations.

I understand for you is important wifi but you have had so many answers about WIFI  that honestly I find useless continue to speak of performances on wifi.
And try consider that wifi on rk 3318 is EVEN WORST !!!! So god bless ssv6051 performances on 3229 !!!!!!!!

If wifi is mandatory for you, stay on *pi family

 

Link to comment
Share on other sites

18 hours ago, Alex83 said:

I have an permanent ping within 2 devices to keep the wifi busy, but now I noticed that from my smartphone there is no constant ping connection and then I only get a connection again if I send from V88 mini 4k a ping to my internal smartphone address... 

 

So it seems like it could also be some ip-layer or firewall topic. I haven't installed any ufw or other firewall. So I guess there should be no firewall working at the moment... 

 

I think neither ubuntu nor debian installs some firewall rules by default. I have debian on my NAS and I don't either have iptables installed, so I guess no firewall rules are installed by default on my box.

 

18 hours ago, Alex83 said:

Also I tried again a

 

Iperf3 localhost performance test. 

 

The result is ~630 Mbit/s. 

 

This is the result of a straightforward iperf3 run on my NAS box:

paolo@nas:~$ iperf3 -c localhost -t 60 -i 5
Connecting to host localhost, port 5201
[  5] local 127.0.0.1 port 43690 connected to 127.0.0.1 port 5201
^Y[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-5.00   sec   964 MBytes  1.62 Gbits/sec    0    767 KBytes       
[  5]   5.00-10.00  sec   971 MBytes  1.63 Gbits/sec    0   1.69 MBytes       
[  5]  10.00-15.01  sec   852 MBytes  1.43 Gbits/sec    0   2.56 MBytes       
[  5]  15.01-20.00  sec   935 MBytes  1.57 Gbits/sec    0   2.56 MBytes       
[  5]  20.00-25.00  sec   966 MBytes  1.62 Gbits/sec    0   2.56 MBytes       
[  5]  25.00-30.00  sec   962 MBytes  1.62 Gbits/sec    0   2.56 MBytes       
[  5]  30.00-35.00  sec   966 MBytes  1.62 Gbits/sec    0   2.56 MBytes       
[  5]  35.00-40.00  sec   968 MBytes  1.62 Gbits/sec    0   2.56 MBytes       
[  5]  40.00-45.01  sec   966 MBytes  1.62 Gbits/sec    0   2.56 MBytes       
[  5]  45.01-50.00  sec   939 MBytes  1.58 Gbits/sec    0   2.56 MBytes       
[  5]  50.00-55.00  sec   964 MBytes  1.62 Gbits/sec    0   2.56 MBytes       
[  5]  55.00-60.00  sec   960 MBytes  1.61 Gbits/sec    0   2.56 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec  11.1 GBytes  1.60 Gbits/sec    0             sender
[  5]   0.00-60.01  sec  11.1 GBytes  1.60 Gbits/sec                  receiver

I got a much better result of 1.6 gbit/s and two cores were at 100%, one core was hogged by iperf3 client instance and the other one by iperf3 server instance. Can't see any way to use more threads with iperf3 :unsure:

 

The fact is, as I think already stated previously, that you are just stressing the local network stack, so mostly you are moving chunks of data from a memory location into another memory location, with the added overhead of linux network stack code. That's all, it does not reflect the real performance of the hardware because no network hardware is in use.

 

Moreover, with such kind of benchmark mostly you are benchmarking the SoC memory subsystem and its throughput.

Boxes come in two memory flavours mostly:

  • DDR2 in 32 bit configurations
  • DDR3 in 16 bit configurations

DDR2 chips usually are 333 Mhz (rarely 400 Mhz) parts, and DDR3 memory chips usually are 667 Mhz (often 800) parts.

Both are initialized at 330 Mhz during the very first boot stage, so in this condition (both at 330Mhz) the bandwidth of the DDR2 memories is twice the DDR3 memories (note: I don't consider the access timings here for simplicity)

 

Despite being installed with half the bus, DDR3 memories support runtime reclocking via DDR Memory Controller (DMC) kernel driver: they usually can be reclocked up to 800 Mhz, depending on the load. The DMC is disabled in armbian for a good reason: memory chips have different access timings and memories have different configurations, so having a one-for-all configuration and guarantee system stability is hard if ever possible.

 

The good news is that the bootloader (actually a piece of it called ddrbin) that is shipped with armbian is configured to boot DDR2 at 330 Mhz and DDR3 at 660 Mhz, so both kind of memories are mostly on par with bandwidth. The ddrbin is a proprietary rockchip 10 kbytes closed-source blob (bad thing), but is able to autodetect the memory timings (called training) for optimal performance (good thing).

 

Here comes your situation: since you have a NAND flash, the bootloader part is not accessible to the linux kernel rknand driver. For this and other reasons, the bootloader on NAND can't be updated using the multitool or via generic linux, so you are stuck with the original bootloader that initializes the memories at ~330 Mhz (some ddrbin are configured even as low as 300 Mhz).

 

Since you are ultimately benchmarking your memory, the number of 670 mbit/s is a quite good approximation of your memory bandwidth. If I do some rough calculations, supposing your bootloader initializes your DDR3 memories at 300 Mhz, based upon my benchmark and yours, I get this number:

 

1600 (mbit/s, my benchmark result) * 0.5 (16bit DDR3 vs 32bit DDR2 bus ratio) * 0.9 (your 300Mhz DDR3 / my 330Mhz DDR2 ratio) = 727 mbit/s

 

which is roughly the number you get from your benchmark (still I'm not considering memory timings, neither CPU frequency, which may explain the little lower benchmark figure you get). Now this number is just some guessing; it may be totally invented number, since I don't really know your hardware.

 

About the comparison against Orange Pi PC, that's a different platform with different memories, a different memory controller (this is very important!) and probably a totally different kernel configuration, so comparison is very hard. I guess it has DDR3 800 Mhz chips to, but I have no time to check mine now. Yet if there is room for improvement (like the 4-core usage vs. 2-core usage) I'll dip my nose into ;)

 

I hope this lengthy post clarified some dark corners about your iperf3 benchmark.

 

Link to comment
Share on other sites

On 14.3.2021 at 11:56 PM, jock said:

Hello @Alex83, your issues about wifi are strange but indeed can happen.

 

I mean, I experienced crashes and such kind of issues with my boxes (2x ssv6051p and 1x ssv6256p), but now they are quite stable after disabling the p2p interface. The wifi kernel driver also does not help because it is quite messy and low-quality, so some corner cases may cause a kernel panic or crashes.

 

To be honest, I got the reachability issues often as you describe, but I think it is due to some slowness of the interface to answer to ARP requests. If it is so, the issue can be some kind of power management and probably it is controlled by the driver in some ways I actually don't know how to configure.

 

Actually the X server crashing is hardly related to wifi, but it may indeed happen.

I can propose you different steps trying to isolate the crashing issues. Proceed to the next step in case you still have instabilities.

 

 First step: to disable the possibile wifi power saving you can run



iwconfig wlan power off

but I don't remember if the driver allows power saving toggle. Just try and see what iwconfig tells you.

 

Second step: you can set the most conservative approach selecting the cpu model as rk3228a and the flash type as emmc-nand in rk322x-config, so the CPU frequency is set to max 1.2Ghz and the NAND is left to default settings.


Third step: if you still have issues with wifi, try to substitute /boot/dtb/rk322x-box.dtb with this one:rk322x-box.dtb

 

Fourth step: as a test to understand if the problem is really wifi, disable it at all and, if you can, use an USB dongle for some time.

 

About the reboot issue, recently me and @fabiobassainvestigated - again - a lot on the NAND subject. We still don't understand why this happens, but we got some interesting new knowledge about the rockchip way to handle NAND things. After many tests, which involved NAND "formatting" from scratch, I noticed that my board started to reboot without any issue.

In the past I had to detach everything from the board to properly reboot it, including the HDMI and USB-to-TTL serial adapter. Now reboot just works. :huh:

 

I can give some some instructions on how to do so, but you need a male-to-male USB cable. An USB-to-TTL serial cable is of great help for debug (but not really necessary) and you need to be confident in case you have to follow the unbrick procedure, because we're going to replace the bootloader with a newer one, which has some risk involved. Results are not guaranteed, you can see this as an experiment.

 

edit: aside from the bootloader upgrade, it just popped in my mind that you may want to download a new recent copy of the Multitool and reinstall the system via steP-nand: some time ago I updated the trustos binary with a newer and more compatible one. This may (or may not) "fix" the NAND reboot, but if you feel confident you may give a try.

Hey Jock, 

 

Thanks again for your great response and of course to you and @fabiobassato your great and wonderful work to get finnaly a use case for this trash boxes!

 

I'm really thankful for this. 

 

I'll try your suggestions within the next couple of days, but just wanna add some more information. 

 

I have an permanent ping within 2 devices to keep the wifi busy, but now I noticed that from my smartphone there is no constant ping connection and then I only get a connection again if I send from V88 mini 4k a ping to my internal smartphone address... 

 

So it seems like it could also be some ip-layer or firewall topic. I haven't installed any ufw or other firewall. So I guess there should be no firewall working at the moment... 

 

Also I tried again a

 

Iperf3 localhost performance test. 

 

The result is ~630 Mbit/s. 

 

Compared to an orange Pi PC it has 2,5 Gbit/s. 

 

Both devices have 4 cores with around 1,4GHz.  Also if you look into htop I see 4 cores working on both devices, but in rk3229 there is just one time 1x1,4 GHz core in yellow shown as there are 4x1,4 GHz in Orange PI PC. 

 

So I think performance on rk3229 seems still pretty poor... 

 

Hey guys, thanks again for your great improvements and very helpful answers.

 

@fabiobassasorry, I don't wanna bother you and yes you comment this wifi issues already. It was not meant of me to complain about your very good development! Thanks very much again to give us any possibility to use armbian on these devices.

 

@jock yes, I would love to brick my device again and recover it in mask mode. ;)

 

I have that male to male cable and windows and Ubuntu machines... 

 

You just need to be a bit detailed about backing up stuff and recovery. I did an recovery with multitool on SD before I flashed the first time armbian into nand

 

Maybe the ram timing issues are also some reasons why my ip stack is so crappie... 

 

Unfortunately the device looses it's connection after a while and I also saw some kernel panic on my TV (connected via hdmi)... I also thought maybe the power supply is to weak or something like that... 

 

But anyway to double ram speed sounds pretty interesting and maybe I can use an USB wifi. Most importantly for me is to be able to reboot the device... That's really annoying if that's  not possible. 

 

So maybe you can give me some instructions on how to get a better bootloader. My hardware is also shown on my armbianmonitor - u output link in my post above. 

 

I think it has ddr3 ram in my device. 

 

Should I post some hardware pictures of my board? 

Link to comment
Share on other sites

On 3/16/2021 at 11:07 PM, Alex83 said:

You just need to be a bit detailed about backing up stuff and recovery.

Just use the multitool to backup your current installation if you want to. We are going to erase everything on the NAND so if you want to keep the content just do a quick backup.

I suggest you to install from scratch, but if you spent time to do some configurations you don't want to lose again, the multitool will make your life easier.

 

What you need:

  • from the ilmich/rkflashtool you need to clone/download the repository and compile the binary using the instructions there.

 

The procedure (if you are already in maskrom mode, go directly to step 8):

  1. Do the backup using the Multitool
  2. Do "Erase Flash" using the Multitool
  3. Unplug the power cord, detach all unnecessary things: no network, no hdmi, no sdcard, no power cord and no USB things; if serial adapter is attached, keep only ground and TX wires (stock bootloader uses 1.5mbps speed)
  4. Connect the USB male-to-male cable to the computer and then to the USB OTG port of the box
  5. The box should turn on automatically, you should see a device with ID 2207:320b running lsusb command
  6. Erase the flash bootloader: invoke the command rkflashtool e 0 8192 and wait a few seconds
  7. Now we have to put the the board in maskrom mode: unplug the USB cable, wait a few seconds and replug the USB cable. If you don't see anything on serial adapter and the device is listed in lsusb, you are in maskrom mode! As an alternative to unplug/replug, you can also run rkflashtool b 3, but it is preferred to do a power cycle.
  8. If you have a serial adapter attached, depending on the loader, you may need to set the speed to 1500000bps or 115200bps to see the box output.
  9. Upload the loader to the board: rkflashtool l bin/rk322x_loader_v1.10.238_256.bin
  10. Update the loader on the board: rkflashtool a bin/rk322x_loader_v1.10.238_256.bin
  11. Unplug the USB cable
  12. Done!

 

If you are at step 9 and rkflashtool is stuck at "info: send ddrbin vendor code", do a power cycle and use the alternative 1T bootloader: bin/rk322x_loader_v1.10.238_256_1t.bin

You can now restore the backup using the multitool (or do a new installation).

 

Note: I also attached to this post a couple of known working bootloaders in case the one I suggested above does not work and you need to restore back the functionality of your board.

Use them if the one above does not work.

 

 

RK322XMiniLoaderAll_V2.51_spectek_en_ddr2_rd_odt_180703.bin RK322XMiniLoaderAll_V2.47_spectek_en_ddr2_rd_odt_171127.bin

Link to comment
Share on other sites

 
root@ThinkPad-T510 /usr/local/bin > chmod +x rkdeveloptool 
root@ThinkPad-T510 /usr/local/bin > chmod 755 rkdeveloptool 
root@ThinkPad-T510 /usr/local/bin > sudo rkdeveloptool ef
Erasing flash complete.
root@ThinkPad-T510 /usr/local/bin > sudo rkdeveloptool rd 3
Reset Device OK.
root@ThinkPad-T510 /usr/local/bin > cd /home/Downloads/
root@ThinkPad-T510 /home/Downloads > chmod +x /usr/local/bin/upgrade_tool
root@ThinkPad-T510 /home/Downloads > chmod 755 /usr/local/bin/upgrade_tool
root@ThinkPad-T510 /home/Downloads > sudo /usr/local/bin/upgrade_tool ul rk322x_loader_v1.10.256.bin 
Not found config.ini
Program Data in /usr/local/bin
Loading loader...
Support Type:RK322A	Loader ver:2.56	Loader Time:2021-03-13 16:06:57
Upgrade loader ok.
root@ThinkPad-T510 /home/Downloads > chmod 755 rkdeveloptool 
root@ThinkPad-T510 /home/Downloads > chmod +x rkdeveloptool 
root@ThinkPad-T510 /home/Downloads > sudo /usr/local/bin/rkdeveloptool rd
Reset Device OK.
root@ThinkPad-T510 /home/Downloads > sudo /usr/local/bin/rkdeveloptool ef
Erasing flash complete.

Hi Jock,

 

I did it. My Board is a Chiptryp XT-MX4MN-V13 from 2017-97-07 

 

Acutally I am not sure if it's an RK3228a, because of the "Support Type:RK322A" in output of the upgrade_tool. I thought it should be an RK3229.

Link to comment
Share on other sites

18 minutes ago, Alex83 said:


 


root@ThinkPad-T510 /usr/local/bin > chmod +x rkdeveloptool 
root@ThinkPad-T510 /usr/local/bin > chmod 755 rkdeveloptool 
root@ThinkPad-T510 /usr/local/bin > sudo rkdeveloptool ef
Erasing flash complete.
root@ThinkPad-T510 /usr/local/bin > sudo rkdeveloptool rd 3
Reset Device OK.
root@ThinkPad-T510 /usr/local/bin > cd /home/Downloads/
root@ThinkPad-T510 /home/Downloads > chmod +x /usr/local/bin/upgrade_tool
root@ThinkPad-T510 /home/Downloads > chmod 755 /usr/local/bin/upgrade_tool
root@ThinkPad-T510 /home/Downloads > sudo /usr/local/bin/upgrade_tool ul rk322x_loader_v1.10.256.bin 
Not found config.ini
Program Data in /usr/local/bin
Loading loader...
Support Type:RK322A	Loader ver:2.56	Loader Time:2021-03-13 16:06:57
Upgrade loader ok.
root@ThinkPad-T510 /home/Downloads > chmod 755 rkdeveloptool 
root@ThinkPad-T510 /home/Downloads > chmod +x rkdeveloptool 
root@ThinkPad-T510 /home/Downloads > sudo /usr/local/bin/rkdeveloptool rd
Reset Device OK.
root@ThinkPad-T510 /home/Downloads > sudo /usr/local/bin/rkdeveloptool ef
Erasing flash complete.

Hi Jock,

 

I did it. My Board is a Chiptryp XT-MX4MN-V13 from 2017-97-07 

 

Acutally I am not sure if it's an RK3228a, because of the "Support Type:RK322A" in output of the upgrade_tool. I thought it should be an RK3229.

Hello @Alex83, you did it and did it work? Do your board still works? Did you run the benchmarks? Is it rebooting or you still have issues?

 

If you read carefully, the "Support Type" is RK322A and not RK3228A, but that's just a "meaningless" label found in the loader binary: the loader is exactly the same for rk3328a, rk3228b and rk3229 and should work on all of them without distinction.

Link to comment
Share on other sites

Hi Jock, thanks for your very quick reply! The board still boots with multitool, but unfortunately it doesn't recognize the NAND anymore. It says something like there is no eMMC storage recognized. I guess the NANA-Driver is missing or something similar... So I can't do a restore anymore... But there is still a booting into multitool. Also the option for install into NAND now missing in Multitool. I also loaded onto a second SD newest of your multitool version. It has the same behavoir... What do you suggest? Should I send you some pictures of the board? On the RAM Chips there is the Information:

 

SEC 146 HYH9

K4B2G0446D

 

and on NAND:

 

7QG22

NW814

 

and logo seems like an M with surrounded circle

 

I think it should be SAMSUNG DDR3 RAM, but unfortunately i can't find connectors for serial. I have adapter and so on, but can't find pads or the best point to connect via serial.

 

Edit: @jock Do you think I need a "config.ini" like it said in CLI "Not found config.ini"

Edited by Alex83
Link to comment
Share on other sites

14 minutes ago, Alex83 said:

Hi Jock, thanks for your very quick reply! The board still boots with multitool, but unfortunately it doesn't recognize the NAND anymore. It says something like there is no eMMC storage recognized. I guess the NANA-Driver is missing or something similar... So I can't do a restore anymore... But there is still a booting into multitool. Also the option for install into NAND now missing in Multitool. I also loaded onto a second SD newest of your multitool version. It has the same behavoir... What do you suggest? Should I send you some pictures of the board? On the RAM Chips there is the Information:

 

SEC 146 HYH9

K4B2G0446D

 

and on NAND:

 

7QG22

NW814

 

and logo seems like an M with surrounded circle

 

I think it should be SAMSUNG DDR3 RAM, but unfortunately i can't find connectors for serial. I have adapter and so on, but can't find pads or the best point to connect via serial.

Ah, that's very disappointing :(

The NAND is a Micron MT29F64G08CBCGBJ4-5M:G and it is identical to the one I have on my board, but on my board it just works fine.

 

It would be interesting to have at least the output of dmesg to see what the rknand kernel driver says about, usually when the NAND is not recognized it says that it requires to be formatted.

You don't have a serial adapter, do you? Serial output would be awesome at this point. (edit: just read the note about the missing serial... yeah Chiptrip boxes often have the serial pins hidden somewhere, and it is not easy to spot them, don't mind...).

 

I would suggest you to repeat the procedure again with the newest loader, then if still does not work, try again with the loader v2.51 and finally with the loader v2.47.

 

edit: the config.ini message is a bit suspicious, I never got it during my tests

Link to comment
Share on other sites

ok, I will repeat and send you a feedback in a few minuts. I have serial Adapter, but I can't find the connectors on the board! I would really love to give you a dmesg output to continue your great development improvements! - But is my guess correctly that if NAND can't get recognised again the device is trash or can I still use my old Android-Backup and reflash it in Mask ROM Mode?

 

When I reconnect the board to my PC this is the output:

 

[ 4788.722591] usb 1-1.1: new high-speed USB device number 5 using ehci-pci
[ 4788.831239] usb 1-1.1: New USB device found, idVendor=2207, idProduct=320b, bcdDevice= 1.00
[ 4788.831248] usb 1-1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 5114.936216] usb 1-1.1: USB disconnect, device number 5
[ 5115.229255] usb 1-1.1: new high-speed USB device number 6 using ehci-pci
[ 5115.341606] usb 1-1.1: New USB device found, idVendor=2207, idProduct=320b, bcdDevice= 1.00
[ 5115.341611] usb 1-1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 5233.720230] usb 1-1.1: USB disconnect, device number 6
[ 5235.782011] usb 1-1.1: new high-speed USB device number 7 using ehci-pci
[ 5235.890832] usb 1-1.1: New USB device found, idVendor=2207, idProduct=320b, bcdDevice= 1.00
[ 5235.890836] usb 1-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 5235.890837] usb 1-1.1: Product: USB-MSC
[ 5235.890838] usb 1-1.1: Manufacturer: RockChip
[ 5235.890839] usb 1-1.1: SerialNumber: rockchip
[ 5433.199976] usb 1-1.1: USB disconnect, device number 7
[ 5435.424812] usb 1-1.1: new high-speed USB device number 8 using ehci-pci
[ 5435.537960] usb 1-1.1: New USB device found, idVendor=2207, idProduct=320b, bcdDevice= 1.00
[ 5435.537963] usb 1-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 5435.537965] usb 1-1.1: Product: USB-MSC
[ 5435.537966] usb 1-1.1: Manufacturer: RockChip
[ 5435.537967] usb 1-1.1: SerialNumber: rockchip
[ 5589.455455] mmc0: new high speed SD card at address 1234
[ 5589.455947] mmcblk0: mmc0:1234 SA02G 1.83 GiB 
[ 5589.475283]  mmcblk0: p1 p2
[ 5589.475291] mmcblk0: p1 size 4069376 extends beyond EOD, truncated
[ 6823.457926] mmc0: card 1234 removed
[ 6827.632405] usb 1-1.1: USB disconnect, device number 8
[ 7161.190702] perf: interrupt took too long (8087 > 7976), lowering kernel.perf_event_max_sample_rate to 24500
[ 8052.060956] usb 1-1.1: new high-speed USB device number 9 using ehci-pci
[ 8052.173680] usb 1-1.1: New USB device found, idVendor=2207, idProduct=320b, bcdDevice= 1.00
[ 8052.173684] usb 1-1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0


root@ThinkPad-T510 /home/Downloads > lsusb
Bus 002 Device 003: ID 17ef:6019 Lenovo 
Bus 002 Device 002: ID 8087:0020 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 009: ID 2207:320b Fuzhou Rockchip Electronics Company RK3228/RK3229 in Mask ROM mode
Bus 001 Device 002: ID 8087:0020 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
root@ThinkPad-T510 /home/Downloads > 

 

Edit:

 

root@alex-ThinkPad-T510 /home/alex/Downloads > rkdeveloptool ef
Starting to erase flash...
Getting flash info from device failed!


 

Edited by Alex83
it seems like of course I can't erase flash when it's not recognised anymore
Link to comment
Share on other sites

@Alex83 Don't worry, your board won't become trash, you can always use the Android tool for windows to restore NAND functionality as you did in the past.

 

Tell me if things are working better with the v2.51 and v2.47 loaders, in the meantime I try to craft a new bootloader combining them with the 660 Mhz ddrbin so you could try them to get the best out the hardware.

Please have patience ^_^ (as you see, that's very messy getting things working with these boards...)

Link to comment
Share on other sites

Hey Jock! Of course I'll try to be patient and have this little "development exiting feeling" right now, but it seems like my board is stuck in MASK ROM Mode. Unfortunately I don't know what I should do right now...

 

root@alex-ThinkPad-T510 /home/alex/Downloads > lsusb
Bus 002 Device 003: ID 17ef:6019 Lenovo 
Bus 002 Device 002: ID 8087:0020 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 009: ID 2207:320b Fuzhou Rockchip Electronics Company RK3228/RK3229 in Mask ROM mode
Bus 001 Device 002: ID 8087:0020 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
root@alex-ThinkPad-T510 /home/alex/Downloads > rkdeveloptool ef
Starting to erase flash...
Getting flash info from device failed!
root@alex-ThinkPad-T510 /home/alex/Downloads > rkdeveloptool rd 3
Reset Device failed!
root@alex-ThinkPad-T510 /home/alex/Downloads > rkdeveloptool rd
Reset Device failed!
root@alex-ThinkPad-T510 /home/alex/Downloads > 

 

As you mentioned is rkdeveloptool rd the command to put back device into "loader mode", but it seems like I am stuck and can't flash neither v2.51 nor v2.47 in my understanding atm.

 

Edit:

 

I noticed maybe my fault:

 

root@alex-ThinkPad-T510 /home/alex/Downloads > sudo /usr/local/bin/upgrade_tool ul rk322x_loader_v1.10.256.bin 
Not found config.ini
Program Data in /usr/local/bin
Loading loader...
Support Type:RK322A	Loader ver:2.56	Loader Time:2021-03-13 16:06:57
Upgrade loader ok.

The rk322x_loader_v1.10.256.bin was not in /usr/local/bin. It was as you see in /home/alex/Downloads. It this maybe the mistake I did? - Unfortunately I don't know if I need to search for my other laptop, booting windows and don't know which Android-tool version I need to use because I remember it's depending on uboot-loader version...

 

So maybe you can assist me some howto when you have new ideas... ;) - Thanks anyway for your great development and help!

Edited by Alex83
Maybe it was my mistake with loader location
Link to comment
Share on other sites

22 minutes ago, Alex83 said:

Hey Jock! Of course I'll try to be patient and have this little "development exiting feeling" right now, but it seems like my board is stuck in MASK ROM Mode. Unfortunately I don't know what I should do right now...

 





root@alex-ThinkPad-T510 /home/alex/Downloads > lsusb
Bus 002 Device 003: ID 17ef:6019 Lenovo 
Bus 002 Device 002: ID 8087:0020 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 009: ID 2207:320b Fuzhou Rockchip Electronics Company RK3228/RK3229 in Mask ROM mode
Bus 001 Device 002: ID 8087:0020 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
root@alex-ThinkPad-T510 /home/alex/Downloads > rkdeveloptool ef
Starting to erase flash...
Getting flash info from device failed!
root@alex-ThinkPad-T510 /home/alex/Downloads > rkdeveloptool rd 3
Reset Device failed!
root@alex-ThinkPad-T510 /home/alex/Downloads > rkdeveloptool rd
Reset Device failed!
root@alex-ThinkPad-T510 /home/alex/Downloads > 

 

As you mentioned is rkdeveloptool rd the command to put back device into "loader mode", but it seems like I am stuck and can't flash neither v2.51 nor v2.47 in my understanding atm.

 

Don't trust the lsusb telling you the box is in maskrom mode... It always says maskrom mode even if you are in loader mode :wacko:

Try instead to turn the box on and do rkdeveloptool rd 3 (without rkdeveloptool ef first), then run upgrade_tool

 

edit: keep in mind to detach everything from the box when you do these operations! No HDMI, no sdcard, no network, no power plug, no USB devices, just the male to male cable

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...

Important Information

Terms of Use - Privacy Policy - Guidelines