linuxfan

I've been dealing with this problem for a while now, and after seeing this thread, I decided to dive in a little deeper.
With LZ4-compressed lists, running an apt search on my Rock64 currently takes at least 20 minutes. As other users have reported, it appears to be bottlenecked by CPU, and is not saturating the storage I/O. Needless to say, this is not a usable speed.
 
Based on what I've found, I believe with substantial confidence that there is no performance problem with LZ4. This should not be too much of a surprise, because speed, especially decompression speed, is LZ4's primary objective; it is in fact considerably faster than LZO in that regard, which means that it's dramatically faster than DEFLATE (gzip). I cannot imagine any conceivable case where switching from LZ4 to DEFLATE would result in decompression performance improvements, and indeed it does not seem to do so here.
 
In any case, you can trivially see this for yourself by using the command-line lz4 utility to decompress one of the compressed list files in /var/lib/apt/lists. On my Rock64, decompressing the largest of these (/var/lib/apt/lists/apt.armbian.com_dists_buster_main_binary-arm64_Packages.lz4, which weighs about 1.8 MB) takes around a tenth of a second. For some perspective, my test run of an apt search took just over 2700 seconds, so if it did spend all or most of this time doing LZ4 decompression, then it could easly have decompressed every list in the directory over a thousand times each. I am not an expert on the intricacies of file compressors, but I find it to be highly unlikely that LZ4 is responsible for the slowness we're seeing here.
 
I had a look at the strace output for the search operation. I'm new to reading straces, but I think I can confirm what a previous poster said, which is that it's accessing the same files over and over again with small reads, often many times in the same place. It is unclear to me what function this serves, but this appears to be what takes up the majority of the runtime. If you would like to see for yourself, I've made the full strace output available at https://fluora.net/strace_apt-search_armbian.7z. Be warned that it's around 260 megabytes when uncompressed, but since it's extremely repetitive, it compresses efficiently, and the 7z archive is only about 96 kB. In addition, you may compare this to the strace of a sane apt-search operation that I took on an x86 laptop running Debian Bullseye with gzip-compressed package lists, available at https://fluora.net/strace_apt-search_debian86.txt (116 kB, not compressed). This isn't much more than a guess, but I wonder if apt is decompressing the entire list with each one of these repeated read operations - this would explain not only the slowness, but also why switching to uncompressed lists seems to help (since it allows the storage cache to render the repeated accesses cheap).
 
From what I've found here, I strongly suspect that something is wrong with the version of apt distributed by Armbian. I don't know enough about the code to know where to start looking, but this is the point where I would hope that someone familiar with Armbian's modifications to apt could help out.

i have reviewed the strace output helpfully provided by Fluora and most of it is taken up by repeatedly opening, reading, and closing the same index file over 26 thousand times. specifically, the file is opened, one chunk is read, then it is closed, and the process repeats. eventually it gets to reading 2 chunks on each iteration, and then 3, and so on
 
that seems like the problem! there's no reason to be repeatedly opening and reading a file from the beginning tens of thousands of times just for one search. and this does sound like an apt bug -- somehow it is deciding to just spin the same file read for 20 minutes, rather than just reading it once

Oh, sure enough. I see this now.
I had a look at the strace dumps again, and it looks like this is in fact an upstream bug affecting all apt versions I tested.
I didn't think this was the case at first because Armbian is the only one where the bug inflates the processing time enough to make apt search actually unusable - on my Raspberry Pis running Raspbian and laptop running Debian, searching with compressed indexes only takes a few seconds, even despite the many repeated access operations. I remain perplexed as to why only Armbian seems to take around two orders of magnitude longer to complete the operation (presumably due to a similarly higher number of extraneous read operations).
 
In any case, the solution in the meantime is just to disable compression, either by commenting out the lines in /etc/apt/apt.conf.d/02-armbian-compress-indexes or by removing that file, then clearing and re-fetching the index files in /var/lib/apt/lists/. When operating on uncompressed lists, apt seems to be much better behaved, and I don't tihnk I see any signs of the vast numbers of repeated file accesses that I see when the lists are compressed. As I already mentioned, and contrary to what has been said previously in this thread, switching to gzip will not be effective. Disabling list compression entirely seems to be the only way to dodge this bug at the moment. Fortunately, it does appear to work properly; when operating on uncompressed lists, apt doesn't seem to perform the extraneous reads at all (and you're not just being saved by your disk cache).
 
I still can't quite figure out why this seems to be affecting Armbian so much more severely than any of the other Debian-derived distros I currently run. The bug clearly happens on my other systems, but only makes the search operation take around 10 seconds at most, whereas on Armbian, searches on compressed lists take upwards of 20 minutes, and don't go faster than about 16 seconds even with uncompressed ilsts. It could be that there are just more lists to search through under my Armbian install than on my others, but I don't think the differences are enough to account for that, even considering that the bug causes the time it takes to load a file to scale with the third power of the file size.
 
Anyway, a bug report clearly needs to be opened on apt upstream, and Armbian's configuration tweak to enable compression of index files should probably be removed or disabled, at least until the upstream bug is fixed.

@linuxfan the package syslinux for buster is only referenced for the platforms i386 & amd64:
https://packages.debian.org/buster/syslinux
because this are Boot-Loaders für DOS FAT und NTFS which are mainly not used on ARM-Systems!?