Jump to content

Wifi Performance Benchmark test


Recommended Posts

Posted

I have seen many Wifi Benchmarks for speed only and not for reliability.

I stopped doing wifi benchmarks for speed since i get very different values for the same board a few hours later,.

For real life projects reliability and availability is what matters when you depends solely in wifi network.

I would strongly suggest @tkaiser to include "reliability" in his tests, maybe a new thread.

As for example  you can try iperf with -t 1000, -t 2000, -t 3000 and -t 10000 (if it allows?).

Try this with your board and see what happens...

Posted
  On 9/23/2017 at 1:07 PM, @lex said:

I stopped doing wifi benchmarks for speed since i get very different values for the same board a few hours later

Expand  

 

Same with me. I already knew that 'Wi-Fi benchmarking' in overcrowded 2.4 GHz band is just plain stupid but did it to verify results and being able to send new users who believe in miracles to 

 

 

IMO the most important lesson to learn when thinking about 'Wi-Fi performance' is to avoid 2.4GHz band if you live where other people also live and also to avoid single antenna stuff. So it starts to get interesting when switching to 5 GHz and at least 2x2 MIMO. Antennas also matter. Since yesterday on most next platforms an USB attached RTL8812AU dongle seems to be the best choice if you're thinking about wireless 'performance'.

 

What you were talking about (reliability and availability) seems not related to performance at all to me. But I would also add 'distance' to this since related (when a BPi M2 Berry for example with its crappy onboard aerial already loses connection to the AP 10m away then both reliability and availability are directly affected)

Posted

8812au ... and also anything that has Ralink RT5572 or RT3572. Those are not AC but are at least dual band up to 2T2R with good Linux support. I am working on to setup proper testing environment. The current problem is the other side. My router is just 10cm away and I will need to set up another AP. 2.4 band performance is pretty irrelevant in an urban area but our apartment condo has a lot of steel inside walls and it actually works quite well. Except Xradio :)

 

I have 13 wireless adaptors in a dedicated USB3.0 hub and two more are on the way. The plan is to make some scripts and update the database compatibility/kernel properties. All this setup is early WIP.

20170923_191817.jpg

Posted
  On 9/23/2017 at 5:06 PM, tkaiser said:

Antennas also matter

Expand  

I did expect more from a 12dBi Omni wifi antenna, but it's just a piece of long wire (a hard wire) surrounded by ABS, i dropped it and the wire came out, it is really heavy. For a 3 m range the gain was minimum but could make a difference at > 15 Meters.

By 'reliability' i mean you could keep transferring  a 50 GB at higher speed possible without disconnecting from your AP, one can say it depends on the AP (cheap DLink for example is a  trash), that could be a problem. But let's assume you have a good AP/Wifi router. The -t 10000 would simulate/create such a behavior (roughly).  5 GHz maybe far better but it is not for everyone.

 

  On 9/23/2017 at 5:35 PM, Igor said:

I have 13 wireless adaptors in a dedicated USB3.0 hub and two more are on the way

Expand  

  Nice setup!

 

It would be interesting to hear the results of -t 3000 (and > 3000).

Posted
  On 9/23/2017 at 6:48 PM, @lex said:

I did expect more from a 12dBi Omni wifi antenna, but it's just a piece of long wire (a hard wire) surrounded by ABS, i dropped it and the wire came out, it is really heavy. For a 3 m range the gain was minimum but could make a difference at > 15 Meters.

Expand  

 

Of course, type of antenna starts to matter if distances get larger or 'line of sight' is not given. Just as an extreme example: Almost half a km with an ESP8266 and appropriate antennas: 

 

 

I did some testing since yesterday and got a nice 'performance to stability' correlation but it's not a causation and I really think we should not talk about performance when we're looking for stability.

 

I've 2 router/AP at home (Fritzboxes from provider, one since I 'needed' ISDN and the other one since the ISDN capable was too slow to saturate a 100 Mbits/sec connection as router -- so the first Fritzbox terminates DSL, does telephony and acts as DSL modem for the 2nd Fritzbox). I remembered hearing some rumours about the one that I use just as a modem being unstable and decided to test.

 

'Iperf -s' on the box running I had no trouble to let an 'iperf -c -t 3000' run from my MacBook either from neighbours or my own flat (difference: due to distance throughput is ~3MB/s at my neighbours location and +10MB/s in my flat). When starting to continually download archives from the Internet this worked quite well in my neighbour's location but I got kicked out of Wi-Fi after less than 10 minutes with MacBook near to the AP. The whole box somehow locked up and based on touching the surface I would assume it's an overheating issue if the AP has also to play the 'Internet access router' role with NAT and some very basic firewall stuff. If the 2nd Fritzbox played AP/router utilizing the first one only as DSL modem I could download stuff with ~10 MB/s for hours (stopped after 2 hours).

 

In other words: With a 'good performance' setup (wireless client close to AP acting also as a router) I can trigger an instability issue but the problem lives inside the AP/router box and not the client (and it's not an issue as long as the traffic just bridges LAN/WLAN but the router/NAT engine in this thingie locks up once Internet is accessed at high throughput speeds).

 

IMO we should differentiate between performance, stability/reliabity and distance when we explore Wi-Fi further (taking also 'environment' into account and also the role both sides of the connection play)

Posted

Indeed... antennas do matter... you can boost the range 2x or more easily by changing from the standard 'wire' type (as the most basic antenna is nothing more than a wire of the an appropriate length so that it resonates at the correct frequency), to parabolic dish or yagi type. The straight wire type that you would find in most router 'rubber ducky' antenna is omnidirectional, whereas the parabolic dish or yagi type are directional - hence the increase in transmit/receive range - because the signal is focused. 

 

I'd also be curious since Igor is doing that setup how much of an impact the proximity of all those wifi transceivers to one another has on their signal... do some of them - especially the ones with high gain antenna instead of PCB antenna - knock the signal strength and throughput of some of the others out? There is such a thing as being too close - known in the RF world as over-saturation or swamping... where the signal is so overwhelmingly strong that it makes the other side go deaf, and hence packets start to get dropped due to said deafness. In other words, closer is NOT always better! ;)

Posted

 

Testing AP is running on Cubox with RT5572 (up to A mode) and it's about 7-8m away at my balcony. 2.4 or 5Ghz. Tests will not produce scientific values so the question is - is it worth to maintain such database? Is enough to store average bandwidth test which would last few minutes at least?

 

iperf3 -c 172.24.1.1 -t 100

 

Alfa AWUS052NH 5Ghz:

  Reveal hidden contents
 

 

Some generic cheap 8812AU, 5Ghz, similar to this one:

  Reveal hidden contents
 

 

cubox-ap.jpg

 

I also added those who work at the bottom of the download page. Only sunxi-next ... WIP.

Posted
  On 9/30/2017 at 7:41 AM, Igor said:

Testing AP is running on Cubox with RT5572 (up to A mode) and it's about 10m away at my balcony. 2.4 or 5Ghz. Tests will not produce scientific values so the question is - is it worth to maintain such database?

Expand  

 

IMO no, especially not with this AP (being the bottleneck already for a 8812AU). We should maintain a database with recommended (since tested) external adapters and better try to educate users as we tried to do it in the past always.

 

Helping users being able to interpret numbers they find here and there and understanding the importance of some other numbers, eg. that a 2T2R antenna setup compared to 1T1R will make a huge difference in a crowded area since with a single antenna setup there's almost nothing at a 15m distance while with a two antenna setup everything's still fine.

 

Same with 2.4GHz (potentially overcrowded band but better to bridge long distances) vs. 5GHz (potentially higher performance but not over long distances)

Posted

Wow.  Also, realizing it's been too many years since I spoke German...  Is this saying they got that throughput with the included antenna?  Did they have a satellite dish on the AP?  :D

Posted
  On 9/30/2017 at 2:19 PM, TonyMac32 said:

Is this saying they got that throughput with the included antenna?  Did they have a satellite dish on the AP?

Expand  

 

They used the small Xunlong antenna that's part of the package (just check the link to heise.de above for the picture -- I would say one can read PCB rev 1.4 there) and no idea about their Wi-Fi test setup.

Posted

3.4.113 Xenial, Opi zero XR with stock antenna, AP at the same spot, but on crowded 2.4Ghz band

  Reveal hidden contents
 

 

Xradio with 4.13.4

  Reveal hidden contents
 


RT5370 USB 

  Reveal hidden contents


Atheros 9271 USB

  Reveal hidden contents
 


RTL8188CUS nano USB

  Reveal hidden contents
 


8812AU with kernel 4.13.4 and on 2.4G

  Reveal hidden contents
 


After all, this comparison tells something ...
 

Posted
  On 9/30/2017 at 4:37 PM, Igor said:

After all, this comparison tells something ...

Expand  

 

But what exactly? I really don't understand why iperf numbers with same hardware (OPi Zero -- which PCB rev?) and 3.4.113 vs. 4.13.4 differs by 4 times. Wrt the Atheros 9271 the 'antenna diversity' note might be important. And do you also have a RTL8192 with 2T2R config available to test against the RTL8188 (since the latter being the crippled single antenna sibling of the former)?

 

BTW: When I did some Wi-Fi 'performance' tests here in my area (in my flat I was able to 'collect' 140 different wireless networks on my Laptop when scanning for 48 hours) the time of day was way more important than anything else. After midnight I got magnitudes better results compared to the time between 6PM and midnight (when obviously numerous neighbours around used streaming services and trashed the performance). Also a simple channel switch could make a difference of +5 times different performance. My personal conclusion: measuring Wi-Fi with single antenna setups in 2.4GHz band in crowded areas is stupidly fooling yourself.

Posted
  On 9/30/2017 at 5:47 PM, tkaiser said:

But what exactly?

Expand  


My area is also very crowded - I haven't done the actual measurement but I would say I should be somewhere around your numbers. What does this test tell? That XRadio in typical condition simply sux but it's usable for sending small chunks of data on close distance. On both kernels.

Don't have any 8192 2T2R around but 8812AU is 2T2R and its results were pretty decent. Of course, numbers should be different early in the morning ...

Posted

Yeah, I agree that XR819 could possibly be defined as 'worst case scenario' when focussing on most probably totally irrelevant iperf bandwidth measurements -- at least for the use cases these devices are bought for by people not trying to misuse them. But still... 

 

I've the same RT5572 thing as you, I've a different 8812AU dongle and a little 2x2 MIMO RTL8192 dongle (I would've never bought since myself being biased and thinking 'no real antennas --> no buy'). And when testing in totally overcrowded 2.4GHz band against a 802.11n AP (emphasis: crowded2.4GHz and 802.11n) I came to the conclusion to simply stop collecting numbers without meaning since the only obvious testing results were

  • if you've at least 802.11n available and can make use of MIMO then try to USE IT (802.11b/g don't support MIMO). In crowded areas 1T1R vs 2T2R can make the difference between 'too slow for anything' and 'works just fine'
  • with short distances and in a setup with 2 walls in between and some reflections provocated funny stuff happened (the small USB dongle performing better than the 8812AU thingie with huge antennas)
  • over longer distances numbers changed and antennas became more important

And then (as usual) I had to realize that 'benchmarking gone wrong' happened since repeating the same set of tests few hours later or when telling the AP to switch the channel from 'crowded' to 'reported as not so crowded' I got totally different numbers often in the other direction than expected.

 

My personal learning was: MIMO is important, antennas are important if distance increases, switching from 2.4GHz to 5GHz is important when you live where other people live since for whatever reasons 2.4GHz is overcrowded while 5GHz is still mostly fine. And all collected benchmark numbers in such a situation are BS anyway.

 

Unfortunately most people out there prefer data over information, love numbers even if wrong, love charts even if misleading and prefer easy answers over complex ones even if the easy answers are wrong. :)

 

I think instead of providing another set of numbers without meaning we (as Armbian project) should focus on what's important: educating our users to make the right decisions.

Posted

A small addition to the topic.

 

https://github.com/armbian/build/commit/bd0b0354b5013e9f7629520bc7f841303247255a

Perhaps we just got a new best cheap adaptor, Comfast 915AC which runs very nice with this driver. I have 1W AP at 2.4G and within two walls and a lot of stations around making noise ... At the testing spot, my phone transfer rate is 20-40down/10-30up (depend on which time of a day I do the test) ... while running AP with this adaptor I get at 5Ghz AC mode more or less constant transfer rate 70/70Mbps. Note that it is 1T/1T only.

 

There is a variant of 8811au with BT radio, which it would be nice to test and a big 4T4R.

  • Igor pinned this topic
Posted
  On 10/30/2017 at 8:20 AM, Igor said:

Perhaps we just got a new best cheap adaptor, Comfast 915AC which runs very nice with this driver.

Expand  

 

@igor, try to search "rtl8811au" on aliexpress.com. There are much more 5G dongles, not compfast 915ac only. Price with delivery ~9usd.

 

It would be nice to have those dongles work out of box on low budget boards. Now if one wish 5G wifi with minimal price, choice is very poor:

 

- Khadas Vim1 2G/16G (has built in dual band wifi, 2G/8G version still 2.4G only)

- [anything] with rtl8811au dongle (rtl8812au are twice more expensive and kill low board price) - requires pass building quest

Posted
  On 11/22/2017 at 4:50 AM, vitaly said:

It would be nice to have those dongles work out of box on low budget boards. Now if one wish 5G wifi with minimal price, choice is very poor:

Expand  


I have two 8811au around and they both work with our driver. At least this one is apparently fake: https://www.aliexpress.com/item/5-2-4Ghz-600Mbps-Wireless-Dual-Band-USB-WiFi-Dongle-Wireless-LAN-Adapter-802-11ac-a/32816342812.html and I have the other one Zapo which claims it also has BT but there is no sign of it. I prefer buying on Ebay since its possible to pay with Paypal. This one https://www.aliexpress.com/item/Mini-Wireless-Dual-Band-802-11ac-Portable-USB-WiFi-5Ghz-2-4Ghz-433Mpbs-Adapter-RTL8811AU-for/32748248429.html could be only other alternative in this price range/size.


And this driver is currently ported to 4.9 and 4.13 and it might be difficult/pointless to port it on older kernels.

Posted

I prefer ali because it allows pay without paypal, and a bit more cheap for my country.

 

https://www.aliexpress.com/af/rtl8811au.html - here i see 3 different cases with internal antenna, and 2 with external antenna. All around 10$. Not possible to find much cheaper than 9$ (if not fake).

 

Point is, that "rtl8811au" can be better search key, because price/sellers can change, and you can easily select the best one in actual moment.

 

Posted
  On 11/22/2017 at 6:04 AM, Igor said:
Expand  

 

Well, only the title is wrong but the fine-print reveals all the ugly details: Realtek 8188CU, 150Mbps, 802.11n -- won't prevent the 'buy cheap, buy twice' target audience to choose it though :) 

 

  On 11/22/2017 at 4:50 AM, vitaly said:

rtl8811au dongle (rtl8812au are twice more expensive and kill low board price

Expand  

 

Ever thought about why access points have more than one antenna? Ever thought about why those devices with really good Wi-Fi performance (better high-end laptops for example) have 2 or 3 internal antennas? Since MIMO is a basic requirement for 802.11n and 802.11ac performance (more 'spatial streams' in parallel).

Posted
  On 11/22/2017 at 6:26 AM, tkaiser said:

Ever thought about why access points have more than one antenna?

Expand  


Yes, it is better to go for 8812AU in any case ... it is slightly bigger, costs a few bucks more but performances are ... doubled. Single 5G AC stream is much better than anything on 2.4 ... so it some cases 1T1R makes sense BTW. I even got 8814AU for 17$ which lowest regular price is around 25$.

Posted
  On 11/22/2017 at 6:26 AM, tkaiser said:

Ever thought about why access points have more than one antenna?

Expand  

 

I understand :) . But in real world there many cases when you just need to leave 2.4g. Because it's unuseable at all in big cities, even if you have many antennas.

 

For example, i need octoprint device for 3d printer. Can't use 2.4g. 8811AU will be more optimal - less power requirements, more cheap. And still faster than any possible 2.4g device at my location.

 

Posted
  On 11/22/2017 at 6:04 AM, Igor said:

I prefer buying on Ebay

Expand  

 

I prefer buying somewhere where I can read honest reviews, where products are rated by customers (and not bots) and where a 'no questions asked' return/refund policy exists. Unfortunately that's Amazon. I also prefer my country's amazon.de site and when I search there for 'rtl8812au' 2nd link is already my 'manufacturer' of choice: CSL (of course they're not a manufacturer but sell the same Chinese stuff as on Ali or eBay but with certificates that are not faked and products that are not fakes but conformant to specifications/advertisements)

 

  On 11/22/2017 at 6:29 AM, Igor said:

I even got 8814AU for 17$

Expand  

 

Hmm... what's the point of using a chipset with support for 4 real antennas to make a product with zero real antennas? :) 

 

The problem remains the same as with all other 'performance metrics'. People buy numbers like (faked) chipsets and this 600 Mpbs / 1300 Mbps or even 1750 Mbps marketing BS (with single antenna it's not 600 but only 433 Mbps since with el cheapo stuff you can't use 2.4GHz and 5GHz in parallel, then this is just the PHY rate and nothing real and especially with crappy antennas and one or two walls in between performance is already ruined).

 

That said it's great that you integrated a suitable driver in 4.x kernels for those better RealTek dongles now. Tested recently on Orange Pi Zero Plus with my RTL8812AU and got almost as good numbers than my MacBook over short distances, when testing from my neighbour's flat the MacBook won easily (3 antennas). And when I repeated the test this time powering the whole setup not through Xunlong's 3A PSU with barrel plug but Micro USB this time performance was a little bit lower though I don't understand exactly why.

 

Anyway: those better dongles are prone to underpowering which is just another item we need to have on our list of 'support issues' that will pop up once people will combine 'cheapest board possible' with 'cheapest 802.11ac dongle possible' and then blame Armbian for wireless instabilities while's just the usual underpowering mess we deal all day long with already.

Posted
  On 11/22/2017 at 6:49 AM, vitaly said:

But in real world there many cases when you just need to leave 2.4g. Because it's unuseable at all in big cities, even if you have many antennas.

Expand  

 

Well, same situation here (especially in the evening 2.4GHz close to unusable) but when I tried to do some real measurements I was surprised that 2x2 MIMO made the real difference even in overcrowded 2.4 GHz band (compare the RTL8192CU numbers with the 1T1R stuff): https://forum.armbian.com/topic/3739-wi-fi-performance-and-known-issues-on-sbc/?do=findComment&comment=27158

 

Posted

I have 3x3 MIMO router + macbook/macminies behind 1 & 2 walls. Those should be 2x2 MIMO i guess.

 

Skype is totally unusable at 2.4g. And no any problems at 5g. Since my internet connection is limited with 100M, i did not measured top wifi performance - not needed, stability is more important for me.

Posted
  On 11/22/2017 at 6:52 AM, tkaiser said:

Anyway: those better dongles are prone to underpowering which is just another item we need to have on our list of 'support issues' that will pop up once people will combine 'cheapest board possible' with 'cheapest 802.11ac dongle possible' and then blame Armbian for wireless instabilities while's just the usual underpowering mess we deal all day long with already.

Expand  
 


8811 consumes up to 250mA while 8814 go up to 600mA. AP mode, single 2T2R AC client, close proximity running iperf test. When someone would attach this directly to some old BananaPi board she will shut down almost instantly at wifi power up :) Other will have less obvious troubles, while 8811 should be fine on most boards that does not use microUSB powering :)

Posted

I'd say if anything the evolution of WiFi should tell us that bigger != better and more power != better reception and longer support-time != better wifi.

 

I've experimented with that topic A LOT with many different devices and routers - expensive and cheap ones.

 

Almost all RTL8811au and RTL8812au dongles just work well  (there are obviously some black sheeps that overheat, and some that have a random MAC-Adresses assigned, but they're in a minority and can be dealed with) - even if the drivers have hiccups and are all over the place. The biggest issue is a lack of official support (especially regarding upstream support) requiring all kind of user-patches - who would wonder.  I found the driver-variant from aircrack-ng to be the most stable (I think that's the one Hardkernel also uses): https://github.com/aircrack-ng/rtl8812au. Realtek is at it again, telling us their own drivers to be a code-mess rewriting them ... supplying ... tadaaaa ... nothing as an replacement in the meanwhile ...

 

https://github.com/torvalds/linux/tree/master/drivers/net/wireless/realtek/rtlwifi

 

A year later ... we're still not there. Oh well! At least the RTL8812ae PCIe variant is already supported now. Hopefully it will get upstream support soon (tm).

 

Btw. If you're seeking small, inexpensive 5GHZ AC-WAPs (or Routers) on the other hand I can only recommend the Xiaomi Wifi 3G 2018 (with GBe, not 2017!) with Padavan Firmware ...

 

They're worth their money twofold over every other expensive Consumer-Routers I've owned before (Several Asus ACXXAU, TP-Link, Netgear Variants ...).

 

The thing is they don't even offer MU-MIMO, but that's fine - the bottlenecks for a lot of the consumer routers above will be their horrendus power-design, airflow and software.

 

They're overcramped with components that increase the heat and decrease performance, lifetime and reliability.

 

If you really need MU-MIMO then get something like an UniFi UAP-AC-PRO (usually for a flat with 100 feets that's simply a waste of money).

 

I'd say give it a few months / years and we might have upstream support (hopefully).

 

-----

 

TLDR: RTL8811au and RTL8812au are the choice to go for SBCs if you need Wireless nowadays.

 

You'll still benefit from 2 Antennas without MU-MIMO because one is being used to send and one to recieve. A single antenna can just do one thing at once. 2x MU-MIMO only means 2 users can simultaniously send/recieve on 2 Antennas at once.

Posted
  On 9/26/2018 at 12:53 PM, iamwithstupid said:

I found the driver-variant from aircrack-ng to be the most stable (I think that's the one Hardkernel also uses)

Expand  


We use that for years now in all modern kernels, Hardkernel uses something else.

Posted

With WiFi in general - it's really hard to get consistent results on a benchmark basis unless it is very controlled environment...

 

Things that affect WiFi in order...

 

1) The neighbors and other devices on your own SSID - WiFi is a shared medium, and contention based - if you're in a crowded neighborhood, no matter anything else, that's a factor overall in performance and consistency, as it's traffic dependent. If you must test, do it between 12AM and 4AM, that's the general network quiet hour unless you have a neighbor like me that can't sleep and runs netflix over wifi...

 

2) Client RF front-end and antenna - some boards have SoP's that have the chipset, but still the antenna is a bit of deal (actually a lot)

 

3) Client chipset and driver - Realtek is common, and it has it's plus/minus, Ralink-Mediatek is generally good, Broadcom's fmac is safe, but limited sometimes, and QCA's ath9k usb is the boss, but old-school and hard to find these days - sometimes get them on the shenzen marketplace, but full featured without patches, and fully supported by kernel drivers, including monitor mode which is useful for some... the XR chipset that is on some SBC's is a bit of an unknown for me at the moment.

 

BTW - for IoT - most of the ESP8266 and ESP32 boards are quite predictable, if not excellent, but they're consistent...

 

3a) Drivers are important - some are better even in the same chip, and the loose floorboards that linux-wireless provides, sometimes works, sometimes not as they're not always in sync with kernel there.

 

All being equal - let's say we have Board A vs. Board B...

 

4) Board Layout - with the SoP mentioned above, there's also self-generated noise from the board itself - some boards are going to be better than others there - same goes with the USB/SD adapters as well...

 

5) The AP - least concern - but the AP does come into play as performance over distance is generally a 10LogR function, with R being range at a high level - there are advantages for newer AP's, but generally the client chip/driver/config is dominant.

 

Get the right chips/rf/driver/ap - it's a win, obviously - a two stream 802.11ac USB adapter can do wonders if the driver is supported - likewise, a single stream wifi NIC on a noisy board is going to be a challenge no matter what the environment is.

 

Benchmarking on a single run is going to be a problem - see item (1) above, everything else becomes somewhat relative...

 

Best device for checking wifi - an old Android phone or iPod Touch - I've got an older Samsung Galaxy S4 (ex-Cricket/Leapwireless, pre ATT) with a dead SIM - nice part is the S4 supports 11ac in 5GHz, so it's a good sounder for WiFi and Google's play store has good apps to scan there - the currrent iPod touch is also good, it's not a great performer, but good enough to test for Single Board comps or Set-Top Boxes for RF characteristics...

 

 

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...

Important Information

Terms of Use - Privacy Policy - Guidelines