linuxfan 0 Posted May 16, 2020 Share Posted May 16, 2020 Doing apt search took just a few seconds before, but since some days, probably after an apt update && apt upgrade, it has become so slow that a search takes many minutes (20+ minutes) instead of seconds. I don't know why, and I don't know how to fix this. Do you know what to do? The slow part of the search is where it says "Full Text Search". It reaches 50% quickly, and then goes slow. If it matters: ODROID-C2, Buster 0 Quote Link to post Share on other sites
Werner 324 Posted May 16, 2020 Share Posted May 16, 2020 armbianmonitor -u 0 Quote Link to post Share on other sites
matteo 3 Posted May 24, 2020 Share Posted May 24, 2020 I'm having the same problem with a fresh install of Buster server on my ODROID-N2, in case it's related. http://ix.io/2nbL 0 Quote Link to post Share on other sites
Werner 324 Posted May 24, 2020 Share Posted May 24, 2020 Tried to clear the apt cache and recreate it? 0 Quote Link to post Share on other sites
matteo 3 Posted May 25, 2020 Share Posted May 25, 2020 I just tried (apt clean, apt update), but no change. Thank you for the suggestion all the same. 0 Quote Link to post Share on other sites
xwiggen 6 Posted May 25, 2020 Share Posted May 25, 2020 Might check if 'ps -ef' or iotop has a process [ext4lazyinit], this borks your system while cleaning inode tables after resize. 0 Quote Link to post Share on other sites
matteo 3 Posted May 28, 2020 Share Posted May 28, 2020 Thank you for the suggestion, xwiggen, and sorry for the delayed response. I just checked in both places, but no trace of ext4lazyinit. I noticed that an apt search causes one of the cores to ramp up to 100% for the duration of the search. There doesn't appear to be any disk activity while searching, and RAM usage only increases by ~25 MB. 0 Quote Link to post Share on other sites
Werner 324 Posted May 29, 2020 Share Posted May 29, 2020 Did you check if the "wa" rises significantly while searching? This would mean the bottleneck is the sd card. 0 Quote Link to post Share on other sites
matteo 3 Posted May 29, 2020 Share Posted May 29, 2020 I'm running off of a 64GB emmc, but I just checked and the wa stays at 0.0. I found that searching with aptitude works fine (aptitude search package), so that's a good workaround. 0 Quote Link to post Share on other sites
guidol 269 Posted May 29, 2020 Share Posted May 29, 2020 7 minutes ago, matteo said: I found that searching with aptitude works fine (aptitude search package), so that's a good workaround. is that "apt-cache search" or how did you search in the past? which command did you use? 0 Quote Link to post Share on other sites
matteo 3 Posted May 29, 2020 Share Posted May 29, 2020 Normally I'd search with apt, but that's performing poorly on my device ( e.g. 12+ minutes per search): apt search mariadb Searches with aptitude work fine, both with the ncurses interface, and from the command line: aptitude search mariadb I just tried and apt-cache search works fine as well: apt-cache search mariadb 0 Quote Link to post Share on other sites
linuxfan 0 Posted May 30, 2020 Author Share Posted May 30, 2020 This must be a software bug inside Armbian, considering it worked well before an update and that your fresh install has the same bug. 0 Quote Link to post Share on other sites
matteo 3 Posted June 1, 2020 Share Posted June 1, 2020 I was able to fix my slow apt searches by removing /etc/apt/apt.conf.d/02-armbian-compress-indexes as described in this thread (forum.odroid.com). This didn't immediately fix the problem, but it started working normally after I removed all of the .lz4 files from /var/lib/apt/lists/ and ran apt update. I confirmed that all files ending with "_Packages.lz4" are now stored in uncompressed form, and my searches are nearly instantaneous. Aside from lost disk space, are there any downsides to this approach? 0 Quote Link to post Share on other sites
Werner 324 Posted June 1, 2020 Share Posted June 1, 2020 4 minutes ago, matteo said: I was able to fix my slow apt searches by removing /etc/apt/apt.conf.d/02-armbian-compress-indexes as described in this thread (forum.odroid.com). This didn't immediately fix the problem, but it started working normally after I removed all of the .lz4 files from /var/lib/apt/lists/ and ran apt update. I confirmed that all files ending with "_Packages.lz4" are now stored in uncompressed form, and my searches are nearly instantaneous. Aside from lost disk space, are there any downsides to this approach? Just curious: How much is the difference in space? 0 Quote Link to post Share on other sites
matteo 3 Posted June 1, 2020 Share Posted June 1, 2020 (edited) It looks like the files are approximately three times larger now (roughly 35 MB vs 93). I noticed that /etc/apt/apt.conf.d/02-armbian-compress-indexes specifies that files should be gzipped: Acquire::GzipIndexes "true"; Acquire::CompressionTypes::Order:: "gz"; while mine were compressed with lz4. Not sure if this could explain the poor performance, but I thought it was curious. For what it's worth, they're compressed with .gz on my x86_64 desktop. Edit: scratch that - the package files on my desktop are uncompressed, but there are some other files in that directory compressed with gz. Edited June 1, 2020 by matteo correcting erroneous information 0 Quote Link to post Share on other sites
Werner 324 Posted June 1, 2020 Share Posted June 1, 2020 Not sure why this has been implemented. Maybe in the early days to save valuable disk space but today maybe it could be sacrificed for the sake of performance? 0 Quote Link to post Share on other sites
guidol 269 Posted June 1, 2020 Share Posted June 1, 2020 from my personal feeling the time-usage of apt has increased since (armbian) buster. An apt update after a fresh install was much faster before buster (with stretch, jessie, wheezy) 0 Quote Link to post Share on other sites
matteo 3 Posted June 1, 2020 Share Posted June 1, 2020 3 hours ago, Werner said: Not sure why this has been implemented. Maybe in the early days to save valuable disk space but today maybe it could be sacrificed for the sake of performance? I'm not sure either, though I checked and the files are uncompressed on my old cubox i4 pro running Stretch server. Whatever the case, I prefer searching with apt, so the trade off is certainly worth it for me. 3 hours ago, guidol said: from my personal feeling the time-usage of apt has increased since (armbian) buster. An apt update after a fresh install was much faster before buster (with stretch, jessie, wheezy) This is difficult for me to judge, as I went from running Stretch on a cubox via microsd, to Buster on an n2 via emmc. This new system is so much faster that I'm afraid I can't draw any comparisons. 0 Quote Link to post Share on other sites
linuxfan 0 Posted June 2, 2020 Author Share Posted June 2, 2020 (edited) Many thanks @matteo for solving the problem! The compressed .lz4 files were the cause. Edited June 2, 2020 by linuxfan I made a mistake 0 Quote Link to post Share on other sites
matteo 3 Posted June 2, 2020 Share Posted June 2, 2020 You're welcome, linuxfan, and thank you for confirming that it works on your setup as well. Big thanks to Igor for mentioning the compressed indexes over on the odroid forum! I did some poking around, and it looks like the decision to compress the indexes was made upstream: Quote I'd like us to try out using LZ4 compressed index files in /var/lib/apt/lists for the next APT release series, starting in October, after the release of Ubuntu 17.10 "artful". This is done by swapping the default for Acquire::gzipIndexes from false to true. On my system, this compresses /var/lib/apt/lists from 1.3 GB down to 241 MB, which is a lot of space. https://groups.google.com/forum/#!topic/linux.debian.devel/f_rlg3-cw2Y I ran a test and found that a Tritium H5 running Bionic has no problem searching compressed indexes with apt. Maybe this issue is peculiar to Amlogic SBCs? Or perhaps a Debian vs Ubuntu issue? 0 Quote Link to post Share on other sites
Tido 186 Posted June 2, 2020 Share Posted June 2, 2020 19 minutes ago, matteo said: has no problem searching compressed indexes https://forum.armbian.com/topic/8315-daily-tech-related-news-diet/?do=findComment&comment=87391 and the one comment above 0 Quote Link to post Share on other sites
matteo 3 Posted June 3, 2020 Share Posted June 3, 2020 5 hours ago, Tido said: https://forum.armbian.com/topic/8315-daily-tech-related-news-diet/?do=findComment&comment=87391 and the one comment above Very interesting, and thank you for sharing. Do you have any insight into why the Tritium H5 would be so much faster than the ODroid N2 (9 seconds vs 750), despite both including the NEON extension? Not that it matters, I'm happy to use uncompressed indexes. 0 Quote Link to post Share on other sites
linuxfan 0 Posted August 13, 2020 Author Share Posted August 13, 2020 The bug has returned recently! APT searches are super slow once again. Can the Armbian team fix this? 0 Quote Link to post Share on other sites
Werner 324 Posted August 13, 2020 Share Posted August 13, 2020 Nobody took care about that yet. https://armbian.atlassian.net/browse/AR-296?jql=text ~ "indexes" 0 Quote Link to post Share on other sites
gymnodemi 1 Posted August 24, 2020 Share Posted August 24, 2020 Hi everyone, Interestingly enough I had one board have this problem while a fresh new install on another identical board that has been around much longer did not have the issue. At first deleted the armbian repo lz4 files first and ran apt update. That didn't seem to solve the issue. I was able to find that the culprit was this file: /var/lib/apt/lists/deb.debian.org_debian_dists_buster_main_binary-arm64_Packages.lz4 The freshly installed board board did not have the compressed version. I deleted that file and ran apt update and it was recreated as a lz4 file (and took a good while running cnf-update-db). I ended up simply unlz4'ing the file and now my full text searches are back to normal speed. I re-ran apt update and it didn't pull down a compressed version. Since these are the debian based packages I thought it had to do with the compress index but the two boards had identical /etc/apt/apt.conf.d/ entries. The only difference was updates that happened in between when I ran the build script - so I'm wondering if once you have a lz4 version apt continues getting that version instead of the uncompressed one. If the symptoms come back I will update this thread. 0 Quote Link to post Share on other sites
linuxfan 0 Posted January 11 Author Share Posted January 11 @gymnodemi I have done what you posted, and (thank you!) it did help for a while. But now the error has come back! I don't understand why the error keeps coming back. Can we somehow reach out to the Armbian team? Maybe they don't even know about this bug? 0 Quote Link to post Share on other sites
Igor 2117 Posted January 11 Share Posted January 11 8 minutes ago, linuxfan said: Can we somehow reach out to the Armbian team? ... which has endless resources to cover your R&D ideas, fix bugs in public software and cover 99% of the costs by doing that. 17 minutes ago, linuxfan said: Maybe they don't even know about this bug? Your support contract: https://github.com/armbian/build#support Without your help, things will probably not move anywhere: https://armbian.atlassian.net/browse/AR-296 https://github.com/armbian/build/tree/master/packages/bsp/common/etc/apt/apt.conf.d https://docs.armbian.com/Process_Contribute/ https://forum.armbian.com/forum/54-help-wanted/ 0 Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.