vlad59 reacted to TonyMac32 in New official Raspberry Pi 3 Ubuntu 18.04.1 LTS (Bionic Beaver) Beta
Honestly that's the only Pi I use for anything. It is more stable than the 3 for video in my opinion.
vlad59 reacted to Igor in Next LTS kernel 4.19.y Allwinner A10, A20, A64, H2+, H3, H5, H6 debugging party
You already helped a LOT!
Indeed! Let's fix this Pinebook's troubles and we might slowly switch 4.19.y to nightly NEXT building ...
vlad59 got a reaction from guidol in Next LTS kernel 4.19.y Allwinner A10, A20, A64, H2+, H3, H5, H6 debugging party
Added report for Banana Pi with Dev image. Everything seems to work fine (did not test sata for now). The kworker CPU problem (use search for more information) is still there but removing the module still works (I never managed to stress the CPU enough to go above 55°C so I think removing the module won't cause any problem)
vlad59 reacted to TonyMac32 in SD communication electrical considerations
*** This is educational material only concerning current limits and their effects on things like rise and fall times, and how they can theoretically impact SD cards. The correlation with reality is purely theoretical and has not been empirically proven, the concept is correct but any specific values may be/ probably are off by some factor***
A lot of boards from various vendors inevitably have some difficulties with electrical signalling. You see it in delay values, phase correction in software, etc. The general prevailing theory is almost always "get it close and fix it in software". But, the software team is not always clued in, or the physical reality is simply not able to be fixed that way. Today I'll go into SD cards and the need for proper signal paths, limited capacitance along the way, and proper drive levels at the SoC.
This came to my attention while reviewing failures to boot on RK3328 devices, and while this particular issue may still be independent (no root cause confirmed as yet), I have a suspicion it is at least a contributing factor.
GPIO's on SoC's have, typically, current limiting on each pin or bank of pins. For Rockchip, this limit defaults to 4mA, with 8 and 12 being available as optional settings. This includes the SD card bus, clock, output, etc. These current limits are exactly that, limits. If the interface only needs 1 mA, it will only pull 1 mA, regardless of the GPIO drive setting. if it needs more, however, it will only get what the drive setting allows.
SD card interface standards have 3 voltage signalling modes, SBC's only make use of 1 or 2 of them. Many boards supply only 3.3V to the card, limiting to frequencies of 25 or 50 MHz. Those that support UHS-I modes allow 1.8V signalling, and the range of speeds there include the original 25 and 50 MHz, but add higher speeds as well (100 and 208). The higher speeds absolutely require the lower voltage, though the 25 and 50 Mhz speeds benefit from it as well due to improvements in rise/fall time with the smaller voltage transitions.
The SD standard requires a total line capacitance of 40 pF or less on each line, including the internals of the card. I will simply assume 40 pF for the simulations, as it's likely few if any of these boards are "optimal" in that regard.
I'll be using LTSpice as an aid to show the effects of the current limitations, measuring voltage across the capacitance. In reality there would be a complex impedance with the resistance I'm not modelling and the capacitance distributed throughout the system, which would cause some slightly different behavior. This is mostly an educational introduction to the scary universe of high-frequency switching, so I'll skip the really gory details for the sake of readability. In this circuit the LimiterDiode does exactly that, it limits current.
With a 4 mA limit into a 40 pF load at 3.3 Volts, 50 MHz, the signal is unusable. Top in blue is the current sourced by the supply, stuck at the 4 mA limit at all times, below in red is the desired waveform, in green the result.
The board will crash if it attempts this. Now, you might say, what if the board is extremely well made and the capacitance is lower?
Well, it looks like this (assuming 20 pF, which may not really be possible):
Still, no hope of operating properly. Dropping to 25 MHz has roughly the same effect. As I said, this is worst case, so don't jump down my throat about your board with the limits working at 4mA. I am also assuming 4 mA source and sink limits, it may not be symmetrical, in which case the shape goes from triangle to shark fin. Cutting the voltage almost in half by going to UHS-I signalling levels has the same effect (ratio wise vs the V_High value) as cutting the capacitance down.
On the ASUS tinker board this was discovered at some point, and the current limits increased to 8 mA. I have not tried a card with no UHS support at 50 MHz (at least to my knowledge), the results at 3.3V/50MHz still look bad in my oversimplification, much like cutting voltage in half or capacitance or frequency, again, they all play into the same charge rate/time constant
At 25 MHz, however, 8 mA is far more reasonable:
Now to the big reason for 1.8 V signalling:
50 MHz, 1.8V, 40 pF, 8 mA drive.
Assumptions: source/sink values both controlled. It's possible this is not the case. If there is no "sink" limit, or if you assume 20 mA sink limit:
3.3V signalling, 50 MHz, 40 pF load, 8 mA source, 20 mA sink:
Now, as a final bit, assume there were no current limits of any kind, and let's measure what the system would need to supply at 50 MHz 3.3V:
So 120 mA or so to create the exact waveform, assuming some sort of simulated resistance somewhere in the system (in reality there would be a measurable resistance and therefor a current lower than 120, but also a longer rise and fall time) Needless to say, a lot more than 4.
This is mostly important for boards that do not implement the 1.8V signalling modes, but do have aggressive current limiting on the SD card I/O's. 50 MHz is a bad place to live at 3.3 volts given a mechanical connector with wide flat terminals, and routing that doesn't always get as much care as maybe it deserves. RPI, a board with these issues, seems able to source/sink 16 mA, most likely accounting for it's ability (when not starving itself for power, cooking itself, etc) to be "overclocked" in highspeed 3.3V mode. Rockchip can only push 12 mA, I haven't read up on Amlogic yet, so you can't expect to have the same performance if you're not willing to add the necessary support for 1.8V signalling.
vlad59 reacted to TonyMac32 in Learning from DietPi!
I completely agree. I think, if a desire to be more minimalist exists, that we create those images specifically labelled as such, for devices that fit the case (OPi IoT, nanopi NEO, Duo, etc). Otherwise I don't see the value.
My honest reaction to the discussion is that it fits perfectly with the "What is Armbian?" discussion. I don't think playing games to save a few megabytes fits something we really care about, it isn't really our problem if an extreme subset of users wants to pull the 512 MB SD card out of their 10 year old Sandisk Sansa MP3 player and want to put Linux on it...
There are 2 main deliverables that I see: A build system for people to make their flavor of Debian for their needs, and our pre-packaged images. IMHO, if we want to make it more "lean", this can be an option available in the build system deliverable, and if we want to make questionably valuable tiny images, then we can do that in parallel with the current desktop and server images.
vlad59 reacted to sgjava in Learning from DietPi!
As a dev I vote to keep the dev packages in. Sure is nice to have the build tools (even git client) installed even if things like libtool and pkg-config are missing from the base. If you are going for a minimal install I understand stripping all this stuff out, but I'm not sure how important it is to have a 700K vs 1.6 G image in today's terms. As @tkaiser points out most people have moved on beyond 4G SD cards. I've been using 32G for years and the price/performance is good for what I need it for.
I've have some home made security cameras that write/read/delete 100s of movies and images a day 24/7 for years without failure. I realize there may be more intensive usages scenarios (more write intensive), but at the current pricing levels I'll toss the SD out every few years if I have to. As with all things the lowest common denominator or race to the bottom isn't always the best strategy. Stability and standardization are more important to me then a little extra RAM, a little more SD life, etc. Not that these are not important, but the bigger picture I think is more important to focus on.
vlad59 reacted to Igor in Problems after last kernel update
We are trying hard not to break anything since dealing with users is our direct cost. You are asking for my personal time for free, remember this. If you would find a serial console, we might be able to provide you hints for recovery ... some generic you can also find here and you will not waste your time in first place.
Armbian is the one if not the only one that provides such updating in a first place. On most others, this is the only way.
vlad59 reacted to tkaiser in ODROID N1 -- not a review (yet)
UPDATE: You'll find a preliminary performance overview at the end of the thread. Click here.
This is NOT an ODROID N1 review since it's way too early for this and the following will focus on just a very small amount of use cases the board might be used for: server stuff and everything that focuses on network, IO and internal limitations. If you want the hype instead better join Hardkernel's vendor community over there: https://forum.odroid.com/viewforum.php?f=148
All numbers you find below are PRELIMINARY since it's way too early to benchmark this board. This is just the try to get some baseline numbers to better understand for which use cases the device might be appropriate, where to look further into and which settings might need improvements.
Background info first
ODROID N1 is based on the Rockchip RK3399 SoC so we know already a lot since RK3399 isn't really new (see Chromebooks, countless TV boxes with this chip and dev boards like Firefly RK3399, ROCK960 and a lot of others... and there will be a lot more devices coming in 2018 like another board from China soon with a M.2 key M slot exposing all PCIe lanes).
What we already know is that the SoC is one of Rockchip's 'open source SoCs' so software support is already pretty good and the chip vendor itself actively upstreams software support. We also know RK3399 is not the greatest choice for compiling code (use case bottlenecked by memory bandwidth and only 2 fast cores combined with 4 slow ones, for this use case 4 x A15 or A17 cores perform much better), that ARMv8 crypto extensions are supported (see few posts below), that the SoC performs nicely with Android and 'Desktop Linux' stuff (think about GPU and VPU acceleration). We also know that this SoC has 2 USB3 ports and implements PCIe 2.1 with a four lane interface. But so far we don't know how the internal bottlenecks look like so let's focus on this now.
The PCIe 2.1 x4 interface is said to support both Gen1 and Gen2 link speeds (2.5 vs. 5GT/s) but there was recently a change in RK3399 datasheet (downgrade from Gen2 to Gen1) and some mainline kernel patch descriptions seem to indicate that RK3399 is not always able to train for Gen2 link speeds. On ODROID N1 there's a x1 PCIe link used configured as either Gen1 or Gen2 to which a dual-port SATA adapter is connected. The Asmedia ASM1061 was the obvious choice since while being a somewhat old design (AFAIK from 2010) it's cheap and 'fast enough' at least when combined with one or even two HDD.
Since the PCIe implementation on this early N1 dev samples is fixed and limited we need to choose other RK3399 devices to get a clue about PCIe limitations (RockPro64, ROCK960 or the yet not announced other board from China). So let's focus on SATA and USB3 instead. While SATA on 'development boards' isn't nothing new, it's often done with (sometimes really crappy) USB2 SATA bridges, recently sometimes with good USB3 SATA bridges (see ODROID HC1/HC2, Cloudmedia Transformer or Swiftboard) and sometimes it's even 'true' SATA:
Allwinner A10/A20/R40/V40 (many SBC) AM572x Sitara (eg. BeagleBoard-X15 with 1 x eSATA and 1 x SATA on Expansion header) Marvell Armada 38x (Clearfog Base, Clearfog Pro, Helios4) Marvell Armada 37x0 (EspressoBin) NXP i.MX6 (Cubox-i, the various Hummingboard, versions, same with Wandboard and so on)
All the above SoC families do 'native SATA' (the SoC itself implements SATA protocols and connectivity) but performance differs a lot with 'Allwinner SATA' being the worst and only the Marvell implementations performing as expected (+500 MB/s sequential and also very high random IO performance which is what you're looking after when using SSDs). As Armbian user you already know: this stuff is documented in detail, just read through this and that.
RK3399 is not SATA capable and we're talking here about PCIe attached SATA which has 2 disadvantages: slightly bottlenecking performance while increasing overall consumption. N1's SATA implementation and how it's 'advertised' (rootfs on SATA) pose another challenge but this is something for a later post (the sh*tshow known from 'SD cards' the last years now arriving at a different product category called 'SSD').
Benchmarking storage performance is challenging and most 'reviews' done on SBCs use inappropriate tools (see this nice bonnie/bonnie++ example), inappropriate settings (see all those dd and hdparm numbers testing partially filesystems buffers and caches and not storage) or focus only on irrelevant stuff (eg. sequential performance in 'worst case testing mode' only looking at one direction).
Some USB3 tests first
All SSDs I use for the test are powered externally and not by N1 since I ran more than one time in situations with board powered SSDs that performance dropped a lot when some sorts of underpowering occured. The 2 USB3 enclosures above are powered by a separate 5V rail and the SATA attached SSDs by the dual-voltage PSU behind. As expected USB3 storage can use the much faster UAS protocol (we know this from RK3328 devices like ROCK64 already which uses same XHCI controller and most probably nearly identical kernel) and also performance numbers match (with large block and file sizes we get close to 400 MB/s).
We chose iozone for the simple reason to be able to compare with previous numbers but a more thorough benchmark would need some fio testing with different test sets. But it's only about getting a baseline now. Tests done with Hardkernel's Debian Stretch image with some tweaks applied. The image relies on Rockchip's 4.4 BSP kernel (4.4.112) with some Hardkernel tweaks and I adjusted the following: First set both cpufreq governors to performance to be not affected by potentially wrong/weird cpufreq scaling behaviour. Then do static IRQ distribution for USB3 and PCIe on cpu1, cpu2 and cpu3 (all little cores but while checking CPU utilization none of the cores was fully saturated so A53@1.5GHz is fine):
echo 2 >/proc/irq/226/smp_affinity echo 4 >/proc/irq/227/smp_affinity echo 8 >/proc/irq/228/smp_affinity To avoid CPU core collissions the benchmark task itself has been sent to one of the two A72 cores:
taskset -c 5 iozone -e -I -a -s 100M -r 1k -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2 Unfortunately currently I've only crappy SSDs lying around (all cheap consumer SSDs: Samsung EVO 840 and 750, a Samsung PM851 and a Intel 540). So we need to take the results with a grain of salt since those SSDs suck especially with continuous write tests (sequential write performance drops down a lot after a short period of time).
First test is to determine whether USB3 ports behave differently (AFAIK one of the two could also be configured as an OTG port and with some SBC I've seen serious performance drops in such a mode). But nope, they perform identical:
EVO840 behind JMS567 (UAS active) on lower USB3 port (xhci-hcd:usb7, IRQ 228): random random kB reclen write rewrite read reread read write 102400 1 6200 6569 7523 7512 4897 6584 102400 4 23065 25349 34612 34813 23978 25231 102400 16 78836 87689 105249 106777 78658 88240 102400 512 302757 314163 292206 300964 292599 321848 102400 1024 338803 346394 327101 339218 329792 351382 102400 16384 357991 376834 371308 384247 383501 377039 EVO840 behind JMS567 (UAS active) on upper USB3 port (xhci-hcd:usb5, IRQ 227): random random kB reclen write rewrite read reread read write 102400 1 6195 6545 7383 7383 4816 6518 102400 4 23191 25114 34370 34716 23580 25199 102400 16 78727 86695 104957 106634 76359 87610 102400 512 307469 315243 293077 302678 293442 321779 102400 1024 335772 336833 326940 339128 330298 350271 102400 16384 366465 376863 371193 384503 383297 379898 Now attaching an EVO750 (not that fast) that performs pretty identical behind the XHCI host controller and the JMS567 controller inside the enclosure:
EVO750 behind JMS567 (UAS active) on lower USB3 port (xhci-hcd:usb7, IRQ 228): random random kB reclen write rewrite read reread read write 102400 1 6200 6569 7523 7512 4897 6584 102400 4 23065 25349 34612 34813 23978 25231 102400 16 78836 87689 105249 106777 78658 88240 102400 512 302757 314163 292206 300964 292599 321848 102400 1024 338803 346394 327101 339218 329792 351382 102400 16384 357991 376834 371308 384247 383501 377039 (so USB3 is the bottleneck here, especially with random IO an EVO840 is much much faster than an EVO750 but here they perform identical due to the massive USB protocol overhead)
Let's try both USB3 ports at the same time
First quick try was a BTRFS RAID-0 made with 'mkfs.btrfs -f -m raid0 -d raid0 /dev/sda1 /dev/sdb1'. Please note that BTRFS is not the best choice here since all (over)writes with blocksizes lower than btrfs' internal blocksize (4K default) are way slower compared to non CoW filesystems:
random random kB reclen write rewrite read reread read write 102400 1 2659 1680 189424 621860 435196 1663 102400 4 21943 18762 24206 24034 18107 17505 102400 16 41983 46379 62235 60665 52517 42925 102400 512 180106 170002 143494 149187 138185 180238 102400 1024 170757 185623 159296 156870 156869 179560 102400 16384 231366 247201 340649 351774 353245 231721 That's BS numbers, let's forget about them. Now trying the same with mdraid/ext4 configuring a RAID 0 and putting an ext4 on it and... N1 simply powered down when executing mkfs.ext4. Adding 'coherent_pool=2M' to bootargs seems to do the job (and I created the mdraid0 in between with both SSDs connected through SATA)
random random kB reclen write rewrite read reread read write 102400 4 25133 29444 38340 38490 23403 27947 102400 16 85036 97638 113992 114834 79505 95274 102400 512 306492 314124 295266 305411 289393 322493 102400 1024 344588 343012 322018 332545 316320 357040 102400 16384 384689 392707 371415 384741 388054 388908 Seems we're talking here already about one real bottleneck? We see nice improvements with small blocksizes which is an indication that RAID0 is doing its job. But with larger blocksizes we're not able to exceed the 400MB/s barrier so it seems both USB3 ports have to share bandwidth (comparable to the situation on ODROID XU4 where the two USB3 receptacles are connected to an internal USB3 hub which is connected to one USB3 port of the Exynos SoC)
Edit: @Xalius used these results to look into RK3399 TRM (technical reference manual). Quoting ROCK64 IRC:
vlad59 got a reaction from Igor in Lime2 mainline kernel with Debian 9 (stretch) becomes unresponsive (forced reboot required)
Upgraded and it's working fine, I'll get back to you if there are problem
vlad59 reacted to martinayotte in Debian Builds for Orange Pi Win in the pipeline?
As I said in many other threads, USB-TTL-Serial USB dongle as described at http://linux-sunxi.org/UART is a MUST for anyone who plays with any SoC boards !!!
Those are afordable, usually around $0.99 on eBay, you should keep several of the in inventory if you have several SoC boards ...
vlad59 got a reaction from xeros in 5.35/5.36 bug / questions collection
I also noticed that a year ago when I switched to mainline on my Banana, IIRC I talked to tkaiser about it and it was a normal side effect. I did not try to change anything about as my banana pi stay mostly idle and only use their CPU when doing compression and that does not happen very often.
The max freq is defined in the device tree so you can change it on userland
vlad59 got a reaction from 8thphloor in 5.35/5.36 bug / questions collection
So I was beat but I can also confirm that a new install is working fine.
I'm updating another banana pi that was previously running 5.36 and report back.
About the kworker, the problem was diagnosed on linux-sunxi mailing list : https://groups.google.com/forum/#!searchin/linux-sunxi/kworker|sort:date/linux-sunxi/bVEsjQRNVSo/nTa3rpOyAQAJ
If you really want to remove this load you can unload the two modules but I don't know if you'll still have thermal protection on your A20. I'm tempted to try it as it does not run too hot.
vlad59 reacted to tkaiser in Where can download working image?
You're absolutely right. The few hundred people a day downloading these broken and totally untested images (check statistics yourself: https://dl.armbian.com/_download-stats/) are all fooled since these images can't work (broken, untested, 'armbian style'). And only you are brave enough to publicly inform those lazy Armbians about this severe problem. Last image has been released back in June so those lazy Armbians already fooled thousands of poor users with providing broken and untested images. It's almost a miracle that you're the first to tell the truth! Keep on rocking!