My apt search has become super slow recently

[linuxfan 0]

Doing apt search took just a few seconds before, but since some days, probably after an apt update && apt upgrade, it has become so slow that a search takes many minutes (20+ minutes) instead of seconds. I don't know why, and I don't know how to fix this. Do you know what to do?

 

The slow part of the search is where it says "Full Text Search". It reaches 50% quickly, and then goes slow.

 

If it matters: ODROID-C2, Buster

[Werner 324]

armbianmonitor -u

[matteo 3]

I'm having the same problem with a fresh install of Buster server on my ODROID-N2, in case it's related.

 

http://ix.io/2nbL

[Werner 324]

Tried to clear the apt cache and recreate it?

[matteo 3]

I just tried (apt clean, apt update), but no change. Thank you for the suggestion all the same.

[xwiggen 6]

 

Might check if 'ps -ef' or iotop has a process [ext4lazyinit],  this borks your system while cleaning inode tables after resize.

 

[matteo 3]

Thank you for the suggestion, xwiggen, and sorry for the delayed response. I just checked in both places, but no trace of ext4lazyinit. I noticed that an apt search causes one of the cores to ramp up to 100% for the duration of the search. There doesn't appear to be any disk activity while searching, and RAM usage only increases by ~25 MB.

[Werner 324]

Did you check if the "wa" rises significantly while searching? This would mean the bottleneck is the sd card.

[matteo 3]

I'm running off of a 64GB emmc, but I just checked and the wa stays at 0.0.

 

I found that searching with aptitude works fine (aptitude search package), so that's a good workaround.

[guidol 269]

7 minutes ago, matteo said:

I found that searching with aptitude works fine (aptitude search package), so that's a good workaround.

is that "apt-cache search" or how did you search in the past? which command did you use?

[matteo 3]

Normally I'd search with apt, but that's performing poorly on my device ( e.g. 12+ minutes per search):

apt search mariadb

 

Searches with aptitude work fine, both with the ncurses interface, and from the command line:

aptitude search mariadb

 

I just tried and apt-cache search works fine as well:

apt-cache search mariadb

 

[linuxfan 0]

This must be a software bug inside Armbian, considering it worked well before an update and that your fresh install has the same bug.

[matteo 3]

I was able to fix my slow apt searches by removing /etc/apt/apt.conf.d/02-armbian-compress-indexes as described in this thread (forum.odroid.com). This didn't immediately fix the problem, but it started working normally after I removed all of the .lz4 files from /var/lib/apt/lists/ and ran apt update. I confirmed that all files ending with "_Packages.lz4" are now stored in uncompressed form,  and my searches are nearly instantaneous.

 

Aside from lost disk space, are there any downsides to this approach?

[Werner 324]

4 minutes ago, matteo said:

I was able to fix my slow apt searches by removing /etc/apt/apt.conf.d/02-armbian-compress-indexes as described in this thread (forum.odroid.com). This didn't immediately fix the problem, but it started working normally after I removed all of the .lz4 files from /var/lib/apt/lists/ and ran apt update. I confirmed that all files ending with "_Packages.lz4" are now stored in uncompressed form,  and my searches are nearly instantaneous.

 

Aside from lost disk space, are there any downsides to this approach?

Just curious: How much is the difference in space?

[matteo 3]

It looks like the files are approximately three times larger now (roughly 35 MB vs 93).

 

I noticed that /etc/apt/apt.conf.d/02-armbian-compress-indexes specifies that files should be gzipped:

Acquire::GzipIndexes "true";
Acquire::CompressionTypes::Order:: "gz";

while mine were compressed with lz4. Not sure if this could explain the poor performance, but I thought it was curious. For what it's worth, they're compressed with .gz on my x86_64 desktop.

 

Edit: scratch that - the package files on my desktop are uncompressed, but there are some other files in that directory compressed with gz.

Edited June 1, 2020 by matteo
correcting erroneous information

[Werner 324]

Not sure why this has been implemented. Maybe in the early days to save valuable disk space but today maybe it could be sacrificed for the sake of performance?

[guidol 269]

from my personal feeling the time-usage of apt has increased since (armbian) buster.

An apt update after a fresh install was much faster before buster (with stretch, jessie, wheezy)

[matteo 3]

3 hours ago, Werner said:

Not sure why this has been implemented. Maybe in the early days to save valuable disk space but today maybe it could be sacrificed for the sake of performance?

I'm not sure either, though I checked and the files are uncompressed on my old cubox i4 pro running Stretch server.  Whatever the case, I prefer searching with apt, so the trade off is certainly worth it for me.

 

3 hours ago, guidol said:

from my personal feeling the time-usage of apt has increased since (armbian) buster.

An apt update after a fresh install was much faster before buster (with stretch, jessie, wheezy)

This is difficult for me to judge, as I went from running Stretch on a cubox via microsd, to Buster on an n2 via emmc. This new system is so much faster that I'm afraid I can't draw any comparisons.

[linuxfan 0]

Many thanks @matteo for solving the problem! The compressed .lz4 files were the cause.

Edited June 2, 2020 by linuxfan
I made a mistake

[matteo 3]

You're welcome, linuxfan, and thank you for confirming that it works on your setup as well. Big thanks to Igor for mentioning the compressed indexes over on the odroid forum!

 

I did some poking around, and it looks like the decision to compress the indexes was made upstream:

Quote

I'd like us to try out using LZ4 compressed index files in
/var/lib/apt/lists for the next APT release series, starting
in October, after the release of Ubuntu 17.10 "artful".

This is done by swapping the default for Acquire::gzipIndexes
from false to true.

On my system, this compresses /var/lib/apt/lists
from 1.3 GB down to 241 MB, which is a lot of space.

https://groups.google.com/forum/#!topic/linux.debian.devel/f_rlg3-cw2Y

 

I ran a test and found that a Tritium H5 running Bionic has no problem searching compressed indexes with apt. Maybe this issue is peculiar to Amlogic SBCs? Or perhaps a Debian vs Ubuntu issue?

 

[Tido 186]

19 minutes ago, matteo said:

has no problem searching compressed indexes

https://forum.armbian.com/topic/8315-daily-tech-related-news-diet/?do=findComment&comment=87391

 

and the one comment above

 

[matteo 3]

5 hours ago, Tido said:

https://forum.armbian.com/topic/8315-daily-tech-related-news-diet/?do=findComment&comment=87391

 

and the one comment above

 

Very interesting, and thank you for sharing. Do you have any insight into why the Tritium H5 would be so much faster than the ODroid N2 (9 seconds vs 750), despite both including the NEON extension? Not that it matters, I'm happy to use uncompressed indexes.

[linuxfan 0]

The bug has returned recently! APT searches are super slow once again. Can the Armbian team fix this?

[Werner 324]

Nobody took care about that yet.

https://armbian.atlassian.net/browse/AR-296?jql=text ~ "indexes"

[gymnodemi 1]

Hi everyone, 

Interestingly enough I had one board have this problem while a fresh new install on another identical board that has been around much longer did not have the issue. 

 

At first deleted the armbian repo lz4 files first and ran apt update. That didn't seem to solve the issue.

 

I was able to find that the culprit was this file:

/var/lib/apt/lists/deb.debian.org_debian_dists_buster_main_binary-arm64_Packages.lz4

 

The freshly installed board board did not have the compressed version. I deleted that file and ran apt update and it was recreated as a lz4 file (and took a good while running cnf-update-db). 

 

I ended up simply unlz4'ing the file and now my full text searches are back to normal speed. I re-ran apt update and it didn't pull down a compressed version. 

 

Since these are the debian based packages I thought it had to do with the compress index but the two boards had identical /etc/apt/apt.conf.d/ entries. The only difference was updates that happened in between when I ran the build script - so I'm wondering if once you have a lz4 version apt continues getting that version instead of the uncompressed one.

 

If the symptoms come back I will update this thread. 

[linuxfan 0]

@gymnodemi I have done what you posted, and (thank you!) it did help for a while. But now the error has come back! I don't understand why the error keeps coming back. Can we somehow reach out to the Armbian team? Maybe they don't even know about this bug?

[Igor 2117]

8 minutes ago, linuxfan said:

Can we somehow reach out to the Armbian team?

... which has endless resources to cover your R&D ideas, fix bugs in public software and cover 99% of the costs by doing that.
 

17 minutes ago, linuxfan said:

Maybe they don't even know about this bug?

Your support contract:

https://github.com/armbian/build#support

 

Without your help, things will probably not move anywhere:

https://armbian.atlassian.net/browse/AR-296

https://github.com/armbian/build/tree/master/packages/bsp/common/etc/apt/apt.conf.d
https://docs.armbian.com/Process_Contribute/

https://forum.armbian.com/forum/54-help-wanted/