[Suggestion for Devs] Use apt-fast for downloading packages

[Ahmet Emin Koçal]

I don't have a fiber/fast internet connection, and I get around 200kb/s download speed when I use normal apt or apt-get.

 

But apt-fast accelerates download speeds significantly. I go up to 1MB/s using it (I set max connections option to 16 while installing it, you'll know what I mean when you try to install it using apt-add-repository, of course, if you don't know). So I suggest you to use apt-fast in "armbian-config" or any system utility. This suggestion goes to the developers. So that we (users with no good internet connection) can update the system or install the desktop environment a lot faster and easier.

 

[WarHawk_AVG]

Quote

apt-fast is a shellscript wrapper for apt-get and aptitude that can drastically improve apt download times by downloading packages in parallel, with multiple connections per package.

Nice

https://github.com/ilikenwf/apt-fast

[Tido]

7 hours ago, Ahmet Emin Koçal said:

I get around 200kb/s download

I have 40MBit/s = 5 Megabyte/s best effort from my provider and I have never had the situation that this was not maxed out per package download. That said, I am surprised you get only 200kb/s.

Does this also happen if you update your Linux Desktop/Laptop computer ?

 

[tkaiser]

When doing apt on SBC usually the limiting factor is random IO performance of the SD card in question. Apt uses tons of sync calls to ensure stuff on 'disk' is commited correctly (so package handling doesn't break) and with 'average SD cards' this is what's responsible for poor install and update performance.

 

Testing this more or less by accident since a few days since I got some crappy 4 GB cards back from a customer (replaced with good and amazingly fast SanDisk A1 Ultra) and want to destroy them by using them (to see how they fail).

 

An 'apt upgrade' on an image that is a bit outdated takes ages compared to doing the same stuff on a fast A1 card or eMMC (my download connection is 100 Mbits/sec so this is not the bottleneck).

 

Adding to this: Armbian's philosophy is a bit different to other distros. We try to educate our users about such stuff like SD card characteristics so they can either buy better products or learn that sequential writes is something entirely different than random writes. In other words: if you run off an average SD card and then want to 'install the desktop environment' this will take ages due to your SD card and might even wear the SD card out multiple times faster comparing to sequentially burning an Armbian desktop image to the same card in the first place (we provide desktop images for a reason  )

 

TL;DR: I don't think the problem is limited download speeds when talking about slow apt processing.

[Ahmet Emin Koçal]

3 hours ago, Tido said:

I have 40MBit/s = 5 Megabyte/s best effort from my provider and I have never had the situation that this was not maxed out per package download. That said, I am surprised you get only 200kb/s.

Does this also happen if you update your Linux Desktop/Laptop computer ?

 

Yeah, it is about the given max speed per connection (ISP Choices this). And 40Mbit connection has to be a fiber connection, as mine is not a fiber (normal copper cables), they are limited. Apt-fast does 16 connections for one file and this accelerates the download speed.  This is a very common issue among people who don't have a fiber connection. 

 

3 hours ago, tkaiser said:

When doing apt on SBC usually the limiting factor is random IO performance of the SD card in question. Apt uses tons of sync calls to ensure stuff on 'disk' is commited correctly (so package handling doesn't break) and with 'average SD cards' this is what's responsible for poor install and update performance.

 

Testing this more or less by accident since a few days since I got some crappy 4 GB cards back from a customer (replaced with good and amazingly fast SanDisk A1 Ultra) and want to destroy them by using them (to see how they fail).

 

An 'apt upgrade' on an image that is a bit outdated tages ages compared to doing the same stuff on a fast A1 card or eMMC (my download connection is 100 Mbits/sec so this is not the bottleneck).

 

Adding to this Armbian's philosophy is a bit different to other distros. We try to educate our users about such stuff like SD card characteristics so they can either buy better products or learn that sequential writes is something entirely different than random writes. In other words: if you run off an average SD card then want to 'install the desktop environment' this will take ages due to your SD card and might even wear the SD card out multiple times faster comparing to sequentially burning an Armbian desktop image to the same card in the first place (we provide desktop images for a reason  )

 

TL;DR: I don't think the problem is limited download speeds when talking about slow apt processing.

Yeah, sir, it is not, when talking about slow apt processing BUT this is not the situation. 

The situation is, using multiple connections while downloading a file increases speed for the people who don't have a fiber connection. It is the same whether I use a n Intel I5-3230M SSD laptop, or an orange pi plus 2.  I can give you a teamviewer id, if you want to see how does it works (I use it on my opi+2)

 

And If you think that it may break the sd card of users, you can just make an option on armbian-config. Or by testing the sd card beforehand, let the program choice if the sd card is OK for it or not.

(BTW, I don't use an sd card, I use the onboard Emmc which is pretty fast)

[tkaiser]

3 minutes ago, Ahmet Emin Koçal said:

If you think that it may break the sd card of users

 

Armbian provides desktop images to allow users to directly download a larger OS image instead of installing this manually since the reality out there is users not being aware what's important with SD cards. They look at totally irrelevant sequential transfer speeds and buy crappy cards by looking at 'reputable brands' that are none. Again: https://forum.armbian.com/topic/954-sd-card-performance/

 

 

User reality out there is that crappy flash storage lowers performance of every apt operation (eMMC users usually aren't affected until their eMMC dies and they need to switch to SD card or replace the board). Installing tons of packages on those crappy cards might result in them dying (same with dist-upgrades and such stuff).

 

We can't do that much against this problem other than still trying to educate users about what's important with their rootfs (RANDOM IO performance and choosing good cards). Unless this problem is resolved it's pretty pointless to think about improving download speeds for large package installation orgies since the average user suffers from a different problem.

 

Armbian lacks developers and the TODO list is longer as you might imagine. So I bet the only chance that something will change here is you starting to develop an easy way to let apt-fast do the work in a transparent way and then submit a PR after testing all distro variants Armbian currently supports (no idea whether it can be used as alias, won't look into since irrelevant)

[talraash]

Maybe, first step - try other repository mirror for upgrade you download speed?) As far as i know apt-fast still limited by you connection, it just download files from different mirror, so if you have some troubles for connection to one mirror... It's like p2p if some peer can give you traffic with max you channel speed, you can't take benefits with more connection ... And yes, i don't have fiber and some mirrors give me full speed, some are much lower. This is the question of the route to a specific mirror.

[Tido]

9 hours ago, Ahmet Emin Koçal said:

And 40Mbit connection has to be a fiber connection

This is correct, but also with ADSL updating the Laptop I get the maximum from the provider for example 10Mbits == 1,2Megabyte/s

 

9 hours ago, tkaiser said:

the TODO list

Do we have such an offical list ?

 

[tkaiser]

40 minutes ago, Tido said:

Do we have such an offical list ?

 

Do we have a project goal devs agree on?

[Igor]

11 hours ago, tkaiser said:

Do we have a project goal devs agree on?

 

Perhaps in 2019/2020?

 

On 8/2/2018 at 12:53 AM, Ahmet Emin Koçal said:

But apt-fast accelerates download speeds significantly. I go up to 1MB/s using it (I set max connections option to 16 while installing it, you'll know what I mean when you try to install it using apt-add-repository, of course, if you don't know). So I suggest you to use apt-fast in "armbian-config" or any system utility. This suggestion goes to the developers.

1

Updates (development, testing, development, testing, ...) are already a luxury which you don't pay for. And you want optimizations of this luxury? Don't waste time. Do it:
 

https://docs.armbian.com/Process_Contribute/

 

BTW. We have only one download server + backup.

[Ahmet Emin Koçal]

2 hours ago, Igor said:

Updates (development, testing, development, testing, ...) are already a luxury which you don't pay for. And you want optimizations of this luxury? Don't waste time. Do it:
 

https://docs.armbian.com/Process_Contribute/

 

BTW. We have only one download server + backup.

 

I think I'll do it  

I inspected the code a little bit, and I see that it uses "debconf-apt-progress" to show what it's doing. I wonder if apt-fast supports it. If it does, there will be no problems changing apt-get commands to apt-fast.  And probably it doesn't as it uses aria2c command to download packages. But I'll give it a chance.

 

21 hours ago, talraash said:

it just download files from different mirror, so if you have some troubles for connection to one mirror...

It does not download files from a different mirror. It does use the same server, but with multiple connections at the same time. This allows us to use connection speed at its limit. Actually, it uses aria2 to download packages. This is how it does it.

[talraash]

24 minutes ago, Ahmet Emin Koçal said:

It does not download files from a different mirror. It does use the same server, but with multiple connections at the same time. This allows us to use connection speed at its limit. Actually, it uses aria2 to download packages. This is how it does it.

amm.. but the  author of script says that it use "multiples mirror for increase download speed" and in apt-fast.conf you can add multiples mirror.  In any case for most users it pointless and more it may increase disk and cpu usage with no benefits. 

[Ahmet Emin Koçal]

14 minutes ago, talraash said:

amm.. but the scrip author says that it use and in apt-fast.conf you can add multiples mirror. 

There may be an option for that, but it is not the main aim.  

 

15 minutes ago, talraash said:

In any case for most users it pointless and more it may increase disk and cpu usage with no benefits. 

I don't see downloading packages 5 times faster as a waste of time. 

[talraash]

Just now, Ahmet Emin Koçal said:

I don't see downloading packages 5 times faster as a waste of time. 

Such a large increase in speed is primarily related to the network settings of you and / or your provider. In most cases, for most users, the apt allows to download at the maximum available network speed ...

[Ahmet Emin Koçal]

Just now, talraash said:

Such a large increase in speed is primarily related to the network settings of you and / or your provider. In most cases, for most users, the apt allows to download at the maximum available network speed ...

2

Yes, it is. And this is why that tool is created. It depends on the country you live in, and believe me there are very many people suffering from such problems.  So many that some people made tools like these (aria2, apt-fast, or even "internet download manager" on windows). These tools aren't created for few people.

[talraash]

21 minutes ago, Ahmet Emin Koçal said:

Yes, it is. And this is why that tool is created. It depends on the country you live in, and believe me there are very many people suffering from such problems.  So many that some people made tools like these (aria2, apt-fast, or even "internet download manager" on windows). These tools aren't created for few people.

How many people with this problem use armbian? And more how many linux users are experiencing the same problem? How many users solve this problem by changing the mirror? In your case, this is a fight against symptoms, not problem. It is very likely that you RTT to the server is large(or another of the network problems solved by standard means), so TCP's slow start is "slow", and then the small window size limits the speed. This can be tweaked by standard means.

[tkaiser]

2 hours ago, Ahmet Emin Koçal said:

It does use the same server, but with multiple connections at the same time

 

If this increases speed I would strongly suggest looking into @talraash's hint wrt round-trip times (do a ping test to the apt server in question and so on)