Jump to content

My apt search has become super slow recently


linuxfan
 Share

Recommended Posts

Doing apt search took just a few seconds before, but since some days, probably after an apt update && apt upgrade, it has become so slow that a search takes many minutes (20+ minutes) instead of seconds. I don't know why, and I don't know how to fix this. Do you know what to do?

 

The slow part of the search is where it says "Full Text Search". It reaches 50% quickly, and then goes slow.

 

If it matters: ODROID-C2, Buster

Link to comment
Share on other sites

When discussing a problem make sure to provide full logs!

Thank you for the suggestion, xwiggen, and sorry for the delayed response. I just checked in both places, but no trace of ext4lazyinit. I noticed that an apt search causes one of the cores to ramp up to 100% for the duration of the search. There doesn't appear to be any disk activity while searching, and RAM usage only increases by ~25 MB.

Link to comment
Share on other sites

7 minutes ago, matteo said:

I found that searching with aptitude works fine (aptitude search package), so that's a good workaround.

is that "apt-cache search" or how did you search in the past? which command did you use?

Link to comment
Share on other sites

Normally I'd search with apt, but that's performing poorly on my device ( e.g. 12+ minutes per search):

apt search mariadb

 

Searches with aptitude work fine, both with the ncurses interface, and from the command line:

aptitude search mariadb

 

I just tried and apt-cache search works fine as well:

apt-cache search mariadb

 

Link to comment
Share on other sites

I was able to fix my slow apt searches by removing /etc/apt/apt.conf.d/02-armbian-compress-indexes as described in this thread (forum.odroid.com). This didn't immediately fix the problem, but it started working normally after I removed all of the .lz4 files from /var/lib/apt/lists/ and ran apt update. I confirmed that all files ending with "_Packages.lz4" are now stored in uncompressed form,  and my searches are nearly instantaneous.

 

Aside from lost disk space, are there any downsides to this approach?

Link to comment
Share on other sites

4 minutes ago, matteo said:

I was able to fix my slow apt searches by removing /etc/apt/apt.conf.d/02-armbian-compress-indexes as described in this thread (forum.odroid.com). This didn't immediately fix the problem, but it started working normally after I removed all of the .lz4 files from /var/lib/apt/lists/ and ran apt update. I confirmed that all files ending with "_Packages.lz4" are now stored in uncompressed form,  and my searches are nearly instantaneous.

 

Aside from lost disk space, are there any downsides to this approach?

Just curious: How much is the difference in space?

Link to comment
Share on other sites

It looks like the files are approximately three times larger now (roughly 35 MB vs 93).

 

I noticed that /etc/apt/apt.conf.d/02-armbian-compress-indexes specifies that files should be gzipped:

Acquire::GzipIndexes "true";
Acquire::CompressionTypes::Order:: "gz";

while mine were compressed with lz4. Not sure if this could explain the poor performance, but I thought it was curious. For what it's worth, they're compressed with .gz on my x86_64 desktop.

 

Edit: scratch that - the package files on my desktop are uncompressed, but there are some other files in that directory compressed with gz.

Edited by matteo
correcting erroneous information
Link to comment
Share on other sites

3 hours ago, Werner said:

Not sure why this has been implemented. Maybe in the early days to save valuable disk space but today maybe it could be sacrificed for the sake of performance?

I'm not sure either, though I checked and the files are uncompressed on my old cubox i4 pro running Stretch server.  Whatever the case, I prefer searching with apt, so the trade off is certainly worth it for me.

 

3 hours ago, guidol said:

from my personal feeling the time-usage of apt has increased since (armbian) buster.

An apt update after a fresh install was much faster before buster (with stretch, jessie, wheezy)

This is difficult for me to judge, as I went from running Stretch on a cubox via microsd, to Buster on an n2 via emmc. This new system is so much faster that I'm afraid I can't draw any comparisons.

Link to comment
Share on other sites

You're welcome, linuxfan, and thank you for confirming that it works on your setup as well. Big thanks to Igor for mentioning the compressed indexes over on the odroid forum!

 

I did some poking around, and it looks like the decision to compress the indexes was made upstream:

Quote

I'd like us to try out using LZ4 compressed index files in
/var/lib/apt/lists for the next APT release series, starting
in October, after the release of Ubuntu 17.10 "artful".

This is done by swapping the default for Acquire::gzipIndexes
from false to true.

On my system, this compresses /var/lib/apt/lists
from 1.3 GB down to 241 MB, which is a lot of space.

https://groups.google.com/forum/#!topic/linux.debian.devel/f_rlg3-cw2Y

 

I ran a test and found that a Tritium H5 running Bionic has no problem searching compressed indexes with apt. Maybe this issue is peculiar to Amlogic SBCs? Or perhaps a Debian vs Ubuntu issue?

 

Link to comment
Share on other sites

5 hours ago, Tido said:

Very interesting, and thank you for sharing. Do you have any insight into why the Tritium H5 would be so much faster than the ODroid N2 (9 seconds vs 750), despite both including the NEON extension? Not that it matters, I'm happy to use uncompressed indexes.

Link to comment
Share on other sites

Hi everyone, 

Interestingly enough I had one board have this problem while a fresh new install on another identical board that has been around much longer did not have the issue. 

 

At first deleted the armbian repo lz4 files first and ran apt update. That didn't seem to solve the issue.

 

I was able to find that the culprit was this file:

/var/lib/apt/lists/deb.debian.org_debian_dists_buster_main_binary-arm64_Packages.lz4

 

The freshly installed board board did not have the compressed version. I deleted that file and ran apt update and it was recreated as a lz4 file (and took a good while running cnf-update-db). 

 

I ended up simply unlz4'ing the file and now my full text searches are back to normal speed. I re-ran apt update and it didn't pull down a compressed version. 

 

Since these are the debian based packages I thought it had to do with the compress index but the two boards had identical /etc/apt/apt.conf.d/ entries. The only difference was updates that happened in between when I ran the build script - so I'm wondering if once you have a lz4 version apt continues getting that version instead of the uncompressed one.

 

If the symptoms come back I will update this thread. 

Link to comment
Share on other sites

8 minutes ago, linuxfan said:

Can we somehow reach out to the Armbian team?


... which has endless resources to cover your R&D ideas, fix bugs in public software and cover 99% of the costs by doing that.
 

17 minutes ago, linuxfan said:

Maybe they don't even know about this bug?


Your support contract:

https://github.com/armbian/build#support

 

Without your help, things will probably not move anywhere:

https://armbian.atlassian.net/browse/AR-296

https://github.com/armbian/build/tree/master/packages/bsp/common/etc/apt/apt.conf.d
https://docs.armbian.com/Process_Contribute/

https://forum.armbian.com/forum/54-help-wanted/

Link to comment
Share on other sites

I am also suffering from this issue on my Helios64.

 

Let's also look at this from a more scientific and engineering perspective with what info we gathered so far...

 

The big problem is why the problem keeps returning for people. Something might be trying to force LZ4 compression on both the apt lists when we are specifically trying to specify gzip instead in the `/etc/apt/apt.conf.d/02-armbian-compress-indexes` file. It might be related to that Ubuntu patch mentioned here: https://forum.armbian.com/topic/14064-my-apt-search-has-become-super-slow-recently/?do=findComment&comment=102968

 

Or it could be something else.

 

 

It is a completely different method of compression from gzip compression. It's slower, but gives a higher compression ratio. Seems like most 32-bit or 64-bit ARM CPUs don't optimize for LZ4 well, so Armbian could either try something lighter like gzip, or leave it uncompressed and clean it once in a while, right?

 

By the way, has anyone also disabled their ZRAM and RAM logging? Mine have been disabled. I can't remember if that affects `/var` or `/tmp`. But just for the record, mine is disabled.

 

Okay, so with that in mind I am still working on my own solution to this problem. I think we should really stop using LZ4 (just as a test case), and try out gzip, then uncompressed if gzip doesn't work.

 

I'm going to at least check and see if I get some `*.gz` files to populate in my `/var/lib/apt/lists` directory, then see if that works. If no gzip files show up, then we know something is trying to force lz4, and that's a bigger issue on our hands. If both gzip and uncompressed don't work "quickly enough", then that's also a big issue.

 

 

Edited by mrjpaxton
Link to comment
Share on other sites

Okay, okay, listen to this. This is very important.

 

So I tested by clearing out the directory with `rm -r /var/lib/apt/lists/` and rebooting.

Then I ran `apt update` and noticed that the files in `/var/lib/apt/lists/partial/` were, indeed, *.gz files.

 

Well this was a good start, I thought.

 

However, when the apt update command completed though, everything was moved out of the partial directory as LZ4 files.

 

What the heck is going on???

 

So now I need to figure out if the files have been renamed, or have been recompressed because that's an important difference. I'm a bit skeptical at using the `file` command because it only uses the MIME types of files. But using `file` did report it as LZ4.

 

This was odd to me because `apt update` still works quickly. Is the problem here that LZ4 can compress quickly on ARM, but uncompress very slowly?

 

So I ran another test. I had to install rename `apt install rename`, which could rename these files as *.gz instead with the built-in `rename` command, like so...

 

rename 's/.lz4$/.gz/' *.lz4

 

I thought that maybe I could trick APT/dpkg to handle these files specifically as *.gz since I thought there was no way that after `apt update` it would compress so quickly into LZ4. I was really wondering...

 

When I tried that though, I got these errors:

E: Encountered a section with no Package: header
E: Problem with MergeList /var/lib/apt/lists/deb.debian.org_debian_dists_buster_main_binary-arm64_Packages.gz
E: The package lists or status file could not be parsed or opened.
E: Encountered a section with no Package: header
E: Problem with MergeList /var/lib/apt/lists/deb.debian.org_debian_dists_buster_main_binary-arm64_Packages.gz
E: The package lists or status file could not be parsed or opened.

 

I ran `file` again, just to make absolutely sure, and I got the same result as last time:

 

It says: "LZ4 compressed data (v1.4+)"

 

Okay, so what the heck is causing this? It really should be gzip compression.

 

I'll keep clearing `/var/lib/apt/lists` try to see if there are any apt.conf rules that can explicitly disable LZ4, because that's stupid. I'm sorry but... wow. What kind of idiot makes a patch to force compression rules like that?

 

I'm also going to look in /etc/apt/apt.conf.d/ and report back....

Link to comment
Share on other sites

Okay, after looking in `/etc/apt/apt.conf.d/` I noticed there is a higher priority file called 50apt-file.conf, which might be the culprit here. And here's why...

 

First of all, it's a higher priority than 02-armbian-compress-indexes so that means it is going to run after or override those settings, secondly, I noticed a line that says: KeepCompressed "true";

 

I've never seen this kind of line before, and nothing about this is mentioned in the manpage for `man apt.conf` so this had me really curious....

 

I looked up that exact option parameter, and ran into this page: https://www.apt-browse.org/browse/ubuntu/xenial/main/amd64/appstream/0.9.4-1/file/etc/apt/apt.conf.d/50appstream

 

This Webpage also mentions a like that says: KeepCompressedAs "gz";

 

These lines are not located anywhere in the `50apt-file.conf` shipped with Armbian.

 

My thoughts now go in two directions: Is the default compression for `50apt-file.conf` LZ4, or is it gzip?

 

Okay, so I'll add those `KeepCompressedAs "gz";` lines to the end of each stanza in that file, run my test again, reboot, and see what happens....

 

WOW, okay. Something really interesting and strange happened now. I have both *.gz and *.lz4 files, except for `security.debian.org`

 

It really doesn't make sense to me. Why are LZ4 files still being created? Hmm... I'll run the "apt-search speed test" now anyways....

 

Yep, still slow! Ugh... it just seems like nothing is working, and I'm running out of options here. Time to look at apt.conf some more....

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
 Share

×
×
  • Create New...