My apt search has become super slow recently

linuxfan · May 16, 2020

Doing apt search took just a few seconds before, but since some days, probably after an apt update && apt upgrade, it has become so slow that a search takes many minutes (20+ minutes) instead of seconds. I don't know why, and I don't know how to fix this. Do you know what to do?

The slow part of the search is where it says "Full Text Search". It reaches 50% quickly, and then goes slow.

If it matters: ODROID-C2, Buster

Werner · May 16, 2020

armbianmonitor -u

matteo · May 24, 2020

I'm having the same problem with a fresh install of Buster server on my ODROID-N2, in case it's related.

http://ix.io/2nbL

Werner · May 24, 2020

Tried to clear the apt cache and recreate it?

matteo · May 25, 2020

I just tried (apt clean, apt update), but no change. Thank you for the suggestion all the same.

xwiggen · May 25, 2020

Might check if 'ps -ef' or iotop has a process [ext4lazyinit], this borks your system while cleaning inode tables after resize.

matteo · May 28, 2020

Thank you for the suggestion, xwiggen, and sorry for the delayed response. I just checked in both places, but no trace of ext4lazyinit. I noticed that an apt search causes one of the cores to ramp up to 100% for the duration of the search. There doesn't appear to be any disk activity while searching, and RAM usage only increases by ~25 MB.

Werner · May 29, 2020

Did you check if the "wa" rises significantly while searching? This would mean the bottleneck is the sd card.

matteo · May 29, 2020

I'm running off of a 64GB emmc, but I just checked and the wa stays at 0.0.

I found that searching with aptitude works fine (aptitude search package), so that's a good workaround.

guidol · May 29, 2020

7 minutes ago, matteo said:

I found that searching with aptitude works fine (aptitude search package), so that's a good workaround.

is that "apt-cache search" or how did you search in the past? which command did you use?

matteo · May 29, 2020

Normally I'd search with apt, but that's performing poorly on my device ( e.g. 12+ minutes per search):

apt search mariadb

Searches with aptitude work fine, both with the ncurses interface, and from the command line:

aptitude search mariadb

I just tried and apt-cache search works fine as well:

apt-cache search mariadb

linuxfan · May 30, 2020

This must be a software bug inside Armbian, considering it worked well before an update and that your fresh install has the same bug.

matteo · June 1, 2020

I was able to fix my slow apt searches by removing /etc/apt/apt.conf.d/02-armbian-compress-indexes as described in this thread (forum.odroid.com). This didn't immediately fix the problem, but it started working normally after I removed all of the .lz4 files from /var/lib/apt/lists/ and ran apt update. I confirmed that all files ending with "_Packages.lz4" are now stored in uncompressed form, and my searches are nearly instantaneous.

Aside from lost disk space, are there any downsides to this approach?

Werner · June 1, 2020

4 minutes ago, matteo said:

I was able to fix my slow apt searches by removing /etc/apt/apt.conf.d/02-armbian-compress-indexes as described in this thread (forum.odroid.com). This didn't immediately fix the problem, but it started working normally after I removed all of the .lz4 files from /var/lib/apt/lists/ and ran apt update. I confirmed that all files ending with "_Packages.lz4" are now stored in uncompressed form, and my searches are nearly instantaneous.

Aside from lost disk space, are there any downsides to this approach?

Just curious: How much is the difference in space?

matteo · June 1, 2020

It looks like the files are approximately three times larger now (roughly 35 MB vs 93).

I noticed that /etc/apt/apt.conf.d/02-armbian-compress-indexes specifies that files should be gzipped:

Acquire::GzipIndexes "true";
Acquire::CompressionTypes::Order:: "gz";

while mine were compressed with lz4. Not sure if this could explain the poor performance, but I thought it was curious. For what it's worth, they're compressed with .gz on my x86_64 desktop.

Edit: scratch that - the package files on my desktop are uncompressed, but there are some other files in that directory compressed with gz.

Edited June 1, 2020 by matteo
correcting erroneous information

Werner · June 1, 2020

Not sure why this has been implemented. Maybe in the early days to save valuable disk space but today maybe it could be sacrificed for the sake of performance?

guidol · June 1, 2020

from my personal feeling the time-usage of apt has increased since (armbian) buster.

An apt update after a fresh install was much faster before buster (with stretch, jessie, wheezy)

matteo · June 1, 2020

3 hours ago, Werner said:

Not sure why this has been implemented. Maybe in the early days to save valuable disk space but today maybe it could be sacrificed for the sake of performance?

I'm not sure either, though I checked and the files are uncompressed on my old cubox i4 pro running Stretch server. Whatever the case, I prefer searching with apt, so the trade off is certainly worth it for me.

3 hours ago, guidol said:

from my personal feeling the time-usage of apt has increased since (armbian) buster.

An apt update after a fresh install was much faster before buster (with stretch, jessie, wheezy)

This is difficult for me to judge, as I went from running Stretch on a cubox via microsd, to Buster on an n2 via emmc. This new system is so much faster that I'm afraid I can't draw any comparisons.

linuxfan · June 2, 2020

Many thanks @matteo for solving the problem! The compressed .lz4 files were the cause.

Edited June 2, 2020 by linuxfan
I made a mistake

matteo · June 2, 2020

You're welcome, linuxfan, and thank you for confirming that it works on your setup as well. Big thanks to Igor for mentioning the compressed indexes over on the odroid forum!

I did some poking around, and it looks like the decision to compress the indexes was made upstream:

Quote

I'd like us to try out using LZ4 compressed index files in
/var/lib/apt/lists for the next APT release series, starting
in October, after the release of Ubuntu 17.10 "artful".

This is done by swapping the default for Acquire::gzipIndexes
from false to true.

On my system, this compresses /var/lib/apt/lists
from 1.3 GB down to 241 MB, which is a lot of space.

https://groups.google.com/forum/#!topic/linux.debian.devel/f_rlg3-cw2Y

I ran a test and found that a Tritium H5 running Bionic has no problem searching compressed indexes with apt. Maybe this issue is peculiar to Amlogic SBCs? Or perhaps a Debian vs Ubuntu issue?

Tido · June 2, 2020

19 minutes ago, matteo said:

has no problem searching compressed indexes

https://forum.armbian.com/topic/8315-daily-tech-related-news-diet/?do=findComment&comment=87391

and the one comment above

matteo · June 3, 2020

5 hours ago, Tido said:

https://forum.armbian.com/topic/8315-daily-tech-related-news-diet/?do=findComment&comment=87391

and the one comment above

Very interesting, and thank you for sharing. Do you have any insight into why the Tritium H5 would be so much faster than the ODroid N2 (9 seconds vs 750), despite both including the NEON extension? Not that it matters, I'm happy to use uncompressed indexes.

linuxfan · August 13, 2020

The bug has returned recently! APT searches are super slow once again. Can the Armbian team fix this?

Werner · August 13, 2020

Nobody took care about that yet.

https://armbian.atlassian.net/browse/AR-296?jql=text ~ "indexes"

gymnodemi · August 24, 2020

Hi everyone,

Interestingly enough I had one board have this problem while a fresh new install on another identical board that has been around much longer did not have the issue.

At first deleted the armbian repo lz4 files first and ran apt update. That didn't seem to solve the issue.

I was able to find that the culprit was this file:

/var/lib/apt/lists/deb.debian.org_debian_dists_buster_main_binary-arm64_Packages.lz4

The freshly installed board board did not have the compressed version. I deleted that file and ran apt update and it was recreated as a lz4 file (and took a good while running cnf-update-db).

I ended up simply unlz4'ing the file and now my full text searches are back to normal speed. I re-ran apt update and it didn't pull down a compressed version.

Since these are the debian based packages I thought it had to do with the compress index but the two boards had identical /etc/apt/apt.conf.d/ entries. The only difference was updates that happened in between when I ran the build script - so I'm wondering if once you have a lz4 version apt continues getting that version instead of the uncompressed one.

If the symptoms come back I will update this thread.

linuxfan · January 11, 2021

@gymnodemi I have done what you posted, and (thank you!) it did help for a while. But now the error has come back! I don't understand why the error keeps coming back. Can we somehow reach out to the Armbian team? Maybe they don't even know about this bug?

Igor · January 11, 2021

8 minutes ago, linuxfan said:

Can we somehow reach out to the Armbian team?

... which has endless resources to cover your R&D ideas, fix bugs in public software and cover 99% of the costs by doing that.

17 minutes ago, linuxfan said:

Maybe they don't even know about this bug?

Your support contract:

https://github.com/armbian/build#support

Without your help, things will probably not move anywhere:

https://armbian.atlassian.net/browse/AR-296

https://github.com/armbian/build/tree/master/packages/bsp/common/etc/apt/apt.conf.d
https://docs.armbian.com/Process_Contribute/

https://forum.armbian.com/forum/54-help-wanted/

mrjpaxton · February 9, 2021

I am also suffering from this issue on my Helios64.

Let's also look at this from a more scientific and engineering perspective with what info we gathered so far...

The big problem is why the problem keeps returning for people. Something might be trying to force LZ4 compression on both the apt lists when we are specifically trying to specify gzip instead in the `/etc/apt/apt.conf.d/02-armbian-compress-indexes` file. It might be related to that Ubuntu patch mentioned here: https://forum.armbian.com/topic/14064-my-apt-search-has-become-super-slow-recently/?do=findComment&comment=102968

Or it could be something else.

It is a completely different method of compression from gzip compression. It's slower, but gives a higher compression ratio. Seems like most 32-bit or 64-bit ARM CPUs don't optimize for LZ4 well, so Armbian could either try something lighter like gzip, or leave it uncompressed and clean it once in a while, right?

By the way, has anyone also disabled their ZRAM and RAM logging? Mine have been disabled. I can't remember if that affects `/var` or `/tmp`. But just for the record, mine is disabled.

Okay, so with that in mind I am still working on my own solution to this problem. I think we should really stop using LZ4 (just as a test case), and try out gzip, then uncompressed if gzip doesn't work.

I'm going to at least check and see if I get some `*.gz` files to populate in my `/var/lib/apt/lists` directory, then see if that works. If no gzip files show up, then we know something is trying to force lz4, and that's a bigger issue on our hands. If both gzip and uncompressed don't work "quickly enough", then that's also a big issue.

Edited February 9, 2021 by mrjpaxton

mrjpaxton · February 9, 2021

Okay, okay, listen to this. This is very important.

So I tested by clearing out the directory with `rm -r /var/lib/apt/lists/` and rebooting.

Then I ran `apt update` and noticed that the files in `/var/lib/apt/lists/partial/` were, indeed, *.gz files.

Well this was a good start, I thought.

However, when the apt update command completed though, everything was moved out of the partial directory as LZ4 files.

What the heck is going on???

So now I need to figure out if the files have been renamed, or have been recompressed because that's an important difference. I'm a bit skeptical at using the `file` command because it only uses the MIME types of files. But using `file` did report it as LZ4.

This was odd to me because `apt update` still works quickly. Is the problem here that LZ4 can compress quickly on ARM, but uncompress very slowly?

So I ran another test. I had to install rename `apt install rename`, which could rename these files as *.gz instead with the built-in `rename` command, like so...

rename 's/.lz4$/.gz/' *.lz4

I thought that maybe I could trick APT/dpkg to handle these files specifically as *.gz since I thought there was no way that after `apt update` it would compress so quickly into LZ4. I was really wondering...

When I tried that though, I got these errors:

E: Encountered a section with no Package: header
E: Problem with MergeList /var/lib/apt/lists/deb.debian.org_debian_dists_buster_main_binary-arm64_Packages.gz
E: The package lists or status file could not be parsed or opened.
E: Encountered a section with no Package: header
E: Problem with MergeList /var/lib/apt/lists/deb.debian.org_debian_dists_buster_main_binary-arm64_Packages.gz
E: The package lists or status file could not be parsed or opened.

I ran `file` again, just to make absolutely sure, and I got the same result as last time:

It says: "LZ4 compressed data (v1.4+)"

Okay, so what the heck is causing this? It really should be gzip compression.

I'll keep clearing `/var/lib/apt/lists` try to see if there are any apt.conf rules that can explicitly disable LZ4, because that's stupid. I'm sorry but... wow. What kind of idiot makes a patch to force compression rules like that?

I'm also going to look in /etc/apt/apt.conf.d/ and report back....

mrjpaxton · February 9, 2021

Okay, after looking in `/etc/apt/apt.conf.d/` I noticed there is a higher priority file called 50apt-file.conf, which might be the culprit here. And here's why...

First of all, it's a higher priority than 02-armbian-compress-indexes so that means it is going to run after or override those settings, secondly, I noticed a line that says: KeepCompressed "true";

I've never seen this kind of line before, and nothing about this is mentioned in the manpage for `man apt.conf` so this had me really curious....

I looked up that exact option parameter, and ran into this page: https://www.apt-browse.org/browse/ubuntu/xenial/main/amd64/appstream/0.9.4-1/file/etc/apt/apt.conf.d/50appstream

This Webpage also mentions a like that says: KeepCompressedAs "gz";

These lines are not located anywhere in the `50apt-file.conf` shipped with Armbian.

My thoughts now go in two directions: Is the default compression for `50apt-file.conf` LZ4, or is it gzip?

Okay, so I'll add those `KeepCompressedAs "gz";` lines to the end of each stanza in that file, run my test again, reboot, and see what happens....

WOW, okay. Something really interesting and strange happened now. I have both *.gz and *.lz4 files, except for `security.debian.org`

It really doesn't make sense to me. Why are LZ4 files still being created? Hmm... I'll run the "apt-search speed test" now anyways....

Yep, still slow! Ugh... it just seems like nothing is working, and I'm running out of options here. Time to look at apt.conf some more....

Sign In

My apt search has become super slow recently

Recommended Posts

linuxfan

Werner

matteo

Werner

matteo

xwiggen

matteo

Werner

matteo

guidol

matteo

linuxfan

matteo

Werner

matteo

Werner

guidol

matteo

linuxfan

matteo

Tido

matteo

linuxfan

Werner

gymnodemi

linuxfan

Igor

mrjpaxton

mrjpaxton

mrjpaxton

Join the conversation

Forums

My Activity Streams

Download

Store

Important Information