Problem with apt list

[RockBian 6]

Today I updated and upgraded my system, and after a reboot 'apt list' did no longer work. It shows 'Listing... done', and then uses 100% CPU on a single core, without producing anything. According to strace it's looping on reading a file:

openat(AT_FDCWD, "/var/lib/apt/lists/deb.debian.org_debian_dists_buster_main_binary-arm64_Packages.lz4", O_RDONLY) = 28
read(28, "\4\"M\30@@\300\206z\0\0\370\377\31Package: 0ad-data\n"..., 65536) = 65536
read(28, "ta$E\17p\2+\377\21c376d3f140540d9424753d"..., 65536) = 65536
read(28, "a5180e3e2d3f503ddcb5dd7b93\201\2\2\275\0\17"..., 65536) = 65536
read(28, "\2\3700071880bc23623a4ea3664b773a1b6"..., 65536) = 65536
read(28, "\f1'da?\2Jdan_\207\1\v=\2f215438\272\t\365\0213e4c"..., 65536) = 65536
read(28, "a6f6005b836871a9cc032472bbece3b7"..., 65536) = 65536
read(28, "3783a1460d749c6f887843fdd95ec981"..., 65536) = 65536
read(28, "\4\0032 - \16\1\0022\32\t?\27\177foreign\270\0041\370\21f1e69"..., 65536) = 65536
read(28, "dR\203\1\6\4\6T\37\3\324\0\",\ng_zplaying\2759\vt\203\1\233"..., 65536) = 65536
read(28, "\20u://ftp.\24'_nu/bc\2\3\0\363\20b8da7e3f11"..., 65536) = 65536
close(28)                               = 0
openat(AT_FDCWD, "/var/lib/apt/lists/deb.debian.org_debian_dists_buster_main_binary-arm64_Packages.lz4", O_RDONLY) = 28
read(28, "\4\"M\30@@\300\206z\0\0\370\377\31Package: 0ad-data\n"..., 65536) = 65536
read(28, "ta$E\17p\2+\377\21c376d3f140540d9424753d"..., 65536) = 65536
read(28, "a5180e3e2d3f503ddcb5dd7b93\201\2\2\275\0\17"..., 65536) = 65536
read(28, "\2\3700071880bc23623a4ea3664b773a1b6"..., 65536) = 65536
read(28, "\f1'da?\2Jdan_\207\1\v=\2f215438\272\t\365\0213e4c"..., 65536) = 65536
read(28, "a6f6005b836871a9cc032472bbece3b7"..., 65536) = 65536
read(28, "3783a1460d749c6f887843fdd95ec981"..., 65536) = 65536
read(28, "\4\0032 - \16\1\0022\32\t?\27\177foreign\270\0041\370\21f1e69"..., 65536) = 65536
read(28, "dR\203\1\6\4\6T\37\3\324\0\",\ng_zplaying\2759\vt\203\1\233"..., 65536) = 65536
read(28, "\20u://ftp.\24'_nu/bc\2\3\0\363\20b8da7e3f11"..., 65536) = 65536
close(28)                               = 0
openat(AT_FDCWD, "/var/lib/apt/lists/deb.debian.org_debian_dists_buster_main_binary-arm64_Packages.lz4", O_RDONLY) = 28
read(28, "\4\"M\30@@\300\206z\0\0\370\377\31Package: 0ad-data\n"..., 65536) = 65536
read(28, "ta$E\17p\2+\377\21c376d3f140540d9424753d"..., 65536) = 65536
read(28, "a5180e3e2d3f503ddcb5dd7b93\201\2\2\275\0\17"..., 65536) = 65536
read(28, "\2\3700071880bc23623a4ea3664b773a1b6"..., 65536) = 65536
read(28, "\f1'da?\2Jdan_\207\1\v=\2f215438\272\t\365\0213e4c"..., 65536) = 65536
read(28, "a6f6005b836871a9cc032472bbece3b7"..., 65536) = 65536
read(28, "3783a1460d749c6f887843fdd95ec981"..., 65536) = 65536
read(28, "\4\0032 - \16\1\0022\32\t?\27\177foreign\270\0041\370\21f1e69"..., 65536) = 65536
read(28, "dR\203\1\6\4\6T\37\3\324\0\",\ng_zplaying\2759\vt\203\1\233"..., 65536) = 65536
read(28, "\20u://ftp.\24'_nu/bc\2\3\0\363\20b8da7e3f11"..., 65536) = 65536
close(28)                               = 0

This repeats over and over. I checked the file /var/lib/apt/lists/deb.debian.org_debian_dists_buster_main_binary-arm64_Packages.lz4, and it seems valid, as in, lz4 decompresses it without complaints. When I put the decompressed /var/lib/apt/lists/deb.debian.org_debian_dists_buster_main_binary-arm64_Packages in the same directory, 'apt list' works.

 

What is going on? It might be size related. The file is the biggest *_Packages.lz4 file in that directory. There are bigger lz4 files, but according to strace they are not accessed. The timestamp of deb.debian.org_debian_dists_buster_main_binary-arm64_Packages.lz4 is Dec 5 11:03, so I don't think it grew, at last update.

 

The upgraded files are:

Start-Date: 2021-01-17  14:07:30
Commandline: apt upgrade
Requested-By: rockbian (1000)
Install: javascript-common:arm64 (11, automatic), php-symfony-yaml:arm64 (3.4.22+dfsg-2+deb10u1, automatic), libjs-codemirror:arm64 (5.43.0-1+deb10u1, automatic), libjs-popper.js:arm64 (1.14.6+ds2-1, automatic), libjs-jquery-mousewheel:arm64 (1:3.1.13-2, automatic), php-symfony-filesystem:arm64 (3.4.22+dfsg-2+deb10u1, automatic), libjs-jquery-timepicker:arm64 (1.2-1, automatic), php-symfony-dependency-injection:arm64 (3.4.22+dfsg-2+deb10u1, automatic), libjs-jquery-ui:arm64 (1.12.1+dfsg-5, automatic), libjs-bootstrap4:arm64 (4.3.1+dfsg2-1, automatic), php-symfony-config:arm64 (3.4.22+dfsg-2+deb10u1, automatic), php-twig-i18n-extension:arm64 (3.0.0-2~bpo10+1, automatic)
Upgrade: nodejs:arm64 (10.21.0~dfsg-1~deb10u1, 10.23.1~dfsg-1~deb10u1), python-apt-common:arm64 (1.8.4.2, 1.8.4.3), phpmyadmin:arm64 (4:4.9.7+dfsg1-1~bpo10+1, 4:5.0.4+dfsg2-1~bpo10+1), php-google-recaptcha:arm64 (1.2.4-1~bpo10+1, 1.2.4-3~bpo10+1), linux-buster-root-current-helios64:arm64 (20.11.4, 20.11.6), armbian-config:arm64 (20.11.3, 20.11.6), libnode64:arm64 (10.21.0~dfsg-1~deb10u1, 10.23.1~dfsg-1~deb10u1), libp11-kit0:arm64 (0.23.15-2, 0.23.15-2+deb10u1), php-mariadb-mysql-kbs:arm64 (1.2.11-1~bpo10+1, 1.2.12-1~bpo10+1), python3-apt:arm64 (1.8.4.2, 1.8.4.3), tzdata:arm64 (2020d-0+deb10u1, 2020e-0+deb10u1), linux-u-boot-helios64-current:arm64 (20.11.4, 20.11.6)
End-Date: 2021-01-17  14:08:10

 

[Igor 2458]

9 hours ago, RockBian said:

What is going on?

Probably this https://armbian.atlassian.net/browse/AR-296 but someone needs to investigate and fix.

[ShadowDance 38]

If you want to speed it up, you can quick-fix it by doing the following:

 

sudo rm /etc/apt/apt.conf.d/02-armbian-compress-indexes
sudo rm -rf /var/lib/apt/lists/*
sudo apt update

 

If you don't want to remove 02-armbian-compress-indexes you can edit it and change true to false. Do note that these will, unfortunately, be overwritten/replaced by future updates to the linux-buster-root-current-helios64 package.

[RockBian 6]

The problem is not that it's slow, but it doesn't work at all. Or did you mean speed up the solution?

[ShadowDance 38]

@RockBian my suggestion could fix your issue since it gets rid of the gzip compression. The caveat, of course, being that the files will no longer be compressed. It might be possible to edit 02-armbian-compress-indexes and changing the compression to lz4 (if that works) but I haven't tested it.

[jpegxguy 4]

Just for anyone reading this, this might be related. Someone suggested prioritizing gzip in order to escape the slowness of LZ4 on ARM platforms.
Here is the solution, which was also merged in Armbian itself: