Package list error due to filesystem corruption

valant · July 21, 2018

I was faced with a very weird erroneous behavior. The distribution and board are the same. the board is 4GB.

the oddity is in that apt-get cannot do anything complaining that it is "unable to parse package list file" /var/lib/apt/armbian.com_dists_InRelease. sorry the name of the file could be written not 100% precise, but you probably get what the file is.

You know what I found inside the file where looked into it? In the middle of it there were some bullshit network configuration related and inserted by NetworkManager (there was a comment inside - "inserted/merged by network manager")! I very doubt it is normal. Removing that didn't help, because apparently it's not enough to recover the file. I barely could imagine what the thing could happen for that infamous networkmanager to corrupt this file, how? Anyway, any ideas on how I could recover the packaging system to make it "great again"? :lol: Has anyone encountered this before?

I almost didn't anything with the board before noticing that. what I did:

set up static IP

uninstalled unattended-upgrades

resized (offline) ext4 volume where the root resides and the appropriate partition. namely shrank them. But, everything worked after that just fine. hardly that bug could be from this?

valant · July 21, 2018

here is the quote from the command line.

apt-get check
Reading package lists... Error!
E: Unable to parse package file /var/lib/apt/lists/apt.armbian.com_dists_xenial_InRelease (1)
E: The package lists or status file could not be parsed or opened.

If you know how to repair it without reflashing the card, please tell me. I need the FAT partition and resizing it all over again would be so frustrating.

Myy · July 22, 2018

What if you do :

mv /var/lib/apt/lists/apt.armbian.com_dists_xenial_InRelease{,.bak}
apt update

?

If that fails miserably, just do :

mv /var/lib/apt/lists/apt.armbian.com_dists_xenial_InRelease{.bak,}

tkaiser · July 22, 2018

19 hours ago, valant said:

I barely could imagine what the thing could happen for that infamous networkmanager to corrupt this file, how?

Instead of following stupid networkmanager bashing you might better focus on reality. What you obviously experience is filesystem corruption and quite normal with SBC (result of broken/counterfeit SD cards). In such a situation a filesystem check would be a good idea as well as some integrity checking (of packages and the entire SD card surface):

sudo armbianmonitor -v
armbianmonitor -c $HOME


root@rock64:~# armbianmonitor 
Usage: armbianmonitor [-h] [-b] [-c $path] [-d $device] [-D] [-m] [-p] [-r] [-u]

############################################################################

 Use armbianmonitor for the following tasks:

 armbianmonitor -c /path/to/test performs disk health/performance tests
 armbianmonitor -d monitors writes to $device
 armbianmonitor -D tries to upload debug disk info to improve armbianmonitor
 armbianmonitor -m provides simple CLI monitoring - scrolling output
 armbianmonitor -M provides simple CLI monitoring - fixed-line output
 armbianmonitor -n provides simple CLI network monitoring - scrolling output
 armbianmonitor -N provides simple CLI network monitoring - fixed-line output
 armbianmonitor -p tries to install cpuminer for performance measurements
 armbianmonitor -r tries to install RPi-Monitor
 armbianmonitor -u tries to upload armbian-hardware-monitor.log for support purposes
 armbianmonitor -v tries to verify installed package integrity

############################################################################

root@rock64:~# armbianmonitor -v
Starting package integrity check. This might take some time. Be patient please...

It appears you may have corrupt packages.

This is usually a symptom of filesystem corruption caused by SD cards or eMMC
dying or burning the OS image to the installation media went wrong.

The following changes from packaged state files were detected:

/usr/src/linux-headers-4.4.138-rk3328/scripts/Makefile
/usr/lib/python3.5/weakref.py

tk@rock64:~$ armbianmonitor -c $HOME
Starting to fill /dev/mmcblk0p2 with test patterns, please be patient this might take a very long time
Free space: 13.47 GB
Creating file 1.h2w ... OK!
Creating file 2.h2w ... OK!
Creating file 3.h2w ... OK!
Creating file 4.h2w ... OK!
Creating file 5.h2w ... OK!
Creating file 6.h2w ... OK!
Creating file 7.h2w ... OK!
Creating file 8.h2w ... OK!
Creating file 9.h2w ... OK!
Creating file 10.h2w ... OK!
Creating file 11.h2w ... OK!
Creating file 12.h2w ... OK!
Creating file 13.h2w ... OK!
Creating file 14.h2w ... OK!
Free space: 160.88 MB
Average writing speed: 10.10 MB/s

Now verifying the written data:
                  SECTORS      ok/corrupted/changed/overwritten
Validating file 1.h2w ... 2097152/        0/      0/      0
Validating file 2.h2w ... 2097152/        0/      0/      0
Validating file 3.h2w ... 2097152/        0/      0/      0
Validating file 4.h2w ... 2097152/        0/      0/      0
Validating file 5.h2w ... 2097152/        0/      0/      0
Validating file 6.h2w ... 2097152/        0/      0/      0
Validating file 7.h2w ... 2097152/        0/      0/      0
Validating file 8.h2w ... 2097152/        0/      0/      0
Validating file 9.h2w ... 2097152/        0/      0/      0
Validating file 10.h2w ... 2097152/        0/      0/      0
Validating file 11.h2w ... 2097152/        0/      0/      0
Validating file 12.h2w ... 2097152/        0/      0/      0
Validating file 13.h2w ... 2097152/        0/      0/      0
Validating file 14.h2w ...  561290/        0/      0/      0

  Data OK: 13.27 GB (27824266 sectors)
Data LOST: 0.00 Byte (0 sectors)
	       Corrupted: 0.00 Byte (0 sectors)
	Slightly changed: 0.00 Byte (0 sectors)
	     Overwritten: 0.00 Byte (0 sectors)
Average reading speed: 119.12 MB/s

Starting iozone tests. Be patient, this can take a very long time to complete:
	Iozone: Performance Test of File I/O
	        Version $Revision: 3.429 $
		Compiled for 64 bit mode.
		Build: linux 

	Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
	             Al Slater, Scott Rhine, Mike Wisner, Ken Goss
	             Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
	             Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
	             Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
	             Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
	             Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer,
	             Vangel Bojaxhi, Ben England, Vikentsi Lapa.

	Run began: Thu Jul 19 13:41:07 2018

	Include fsync in write timing
	O_DIRECT feature enabled
	Auto Mode
	File size set to 102400 kB
	Record Size 4 kB
	Record Size 512 kB
	Record Size 16384 kB
	Command line used: iozone -e -I -a -s 100M -r 4k -r 512k -r 16M -i 0 -i 1 -i 2
	Output is in kBytes/sec
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 1024 kBytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride                                    
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
          102400       4     2116     2801    15913    11167    10142     2507                                                          
          102400     512    44468    20183    78366    77570    77159    12012                                                          
          102400   16384    13459    13774   119536   119121   119391    12728                                                          

iozone test complete.

The results from testing /dev/mmcblk0p2 (btrfs):
  Data OK: 13.27 GB (27824266 sectors)
Data LOST: 0.00 Byte (0 sectors)
Average writing speed: 10.10 MB/s
Average reading speed: 119.12 MB/s
                                            random    random
reclen    write  rewrite    read    reread    read     write
     4     2116     2801    15913    11167    10142     2507                                                          
   512    44468    20183    78366    77570    77159    12012                                                          
 16384    13459    13774   119536   119121   119391    12728                                                          

Health summary: OK

Performance summary:
Sequential reading speed:119.12 MB/s 
 4K random reading speed: 10142 KB/s 
Sequential writing speed: 10.10 MB/s 
 4K random writing speed:  2507 KB/s 

To interpret the results above correctly or search for better storage
alternatives please refer to http://oss.digirati.com.br/f3/ and also
http://www.jeffgeerling.com/blogs/jeff-geerling/raspberry-pi-microsd-card
and http://thewirecutter.com/reviews/best-microsd-card/

valant · July 22, 2018

thanks for your responses. To Myy, yes, that sequence made it work.) well, apt-get now doesn't complain about being unable to parse that file. still it seems the card is the problem. because I did the check, tkaiser suggested. It reports 1 corrupted block (8 sectors).

well, sd cards... but it's still funny how in the middle of package list file the output from networkmanager occured.

tkaiser · July 22, 2018

44 minutes ago, valant said:

it's still funny how in the middle of package list file the output from networkmanager occured

Well, with flash storage we have an additional layer where things can go wrong.

Filesystems use allocation tables to map 'entities' (blocks) to file contents (or vice versa). If such a pointer is written wrongly then mysteriously other blocks are mapped into a file from somewhere else ('mysteriously' can be related to bit flips happening in memory, one of the many reasons to want devices with ECC RAM support and also filesystems providing data integrity like ZFS, btrfs, ReFS or even Apple's APFS the latter at least providing filesystem metadata integrity so stuff like this can't happen)
On flash storage the underlying 'entities' (pages) are fully abstracted from the view from above. There's the FTL (flash translation layer) in between dynamically mapping between representation to the host as block device and internal representation as pages. And if something here went wrong the same happens: some data (pages this time) from somewhere now mapped to somewhere else

Personally I think 'bit flip in RAM' is most probably the culprit (so no SD card to blame this time)

@chwe: IMO this subthread is worth being splitted off to common issues subforum.

Edited July 22, 2018 by chwe
done, feel free to suggest an appropriate thread name

valant · July 22, 2018

Quote

Personally I think 'bit flip in RAM' is most probably the culprit (so no SD card to blame this time)

but the check revealed corruption of a FS block (8 sectors). isn't it too often for RAM bit flops? I mean it should be a very rare event. here, we have package list file corruption, and now test discovers yet another FS corruption. This card hasn't been used extensively. Almost new. A CentOS installation for Banana PI M2 Ultra was there and everything worked. But I used it a couple of hours in total. so.

root@krypton:/home/val# armbianmonitor -v
Starting package integrity check. This might take some time. Be patient please...
dpkg: unrecoverable fatal error, aborting:
files list file for package 'unattended-upgrades' contains empty filename

It appears you may have corrupt packages.

This is usually a symptom of filesystem corruption caused by SD cards or eMMC
dying or burning the OS image to the installation media went wrong.

The following changes from packaged state files were detected:

... bunch of files ...

root@krypton:/home/val# armbianmonitor -c $HOME
Checking disks is not permitted as root or through sudo. Exiting
root@krypton:/home/val# exit
exit
val@krypton:~$ armbianmonitor -c $HOME

WARNING: It seems you're not testing the SD card but instead /dev/mmcblk1p1 (ext4)

Starting to fill /dev/mmcblk1p1 with test patterns, please be patient this might take a very long time
Free space: 6.97 GB
Creating file 1.h2w ... OK!
Creating file 2.h2w ... OK!
Creating file 3.h2w ... OK!
Creating file 4.h2w ... OK!
Creating file 5.h2w ... OK!
Creating file 6.h2w ... OK!
Creating file 7.h2w ... OK!
Free space: 103.06 MB
Average writing speed: 8.66 MB/s

Now verifying the written data:
SECTORS ok/corrupted/changed/overwritten
Validating file 1.h2w ... 2097144/ 8/ 0/ 0
Validating file 2.h2w ... 2097152/ 0/ 0/ 0
Validating file 3.h2w ... 2097152/ 0/ 0/ 0
Validating file 4.h2w ... 2097152/ 0/ 0/ 0
Validating file 5.h2w ... 2097152/ 0/ 0/ 0
Validating file 6.h2w ... 2097152/ 0/ 0/ 0
Validating file 7.h2w ... 1830544/ 0/ 0/ 0

Data OK: 6.87 GB (14413448 sectors)
Data LOST: 4.00 KB (8 sectors)
Corrupted: 4.00 KB (8 sectors)
Slightly changed: 0.00 Byte (0 sectors)
Overwritten: 0.00 Byte (0 sectors)
Average reading speed: 20.64 MB/s

Starting iozone tests. Be patient, this can take a very long time to complete:
Iozone: Performance Test of File I/O
Version $Revision: 3.429 $
Compiled for 64 bit mode.
Build: linux

Contributors: good guys.

Run began: Fri Jul 20 06:29:58 2018

Include fsync in write timing
O_DIRECT feature enabled
Auto Mode
File size set to 102400 kB
Record Size 4 kB
Record Size 512 kB
Record Size 16384 kB
Command line used: iozone -e -I -a -s 100M -r 4k -r 512k -r 16M -i 0 -i 1 -i 2
Output is in kBytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 kBytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
random random bkwd record stride
kB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
102400 4 1325 1378 4922 4929 4412 224
102400 512 9128 8563 21971 21989 21917 455
102400 16384 8868 8909 22672 22676 22676 5962

iozone test complete.

The results from testing /dev/mmcblk1p1 (ext4):
Data OK: 6.87 GB (14413448 sectors)
Data LOST: 4.00 KB (8 sectors)
Average writing speed: 8.66 MB/s
Average reading speed: 20.64 MB/s
random random
reclen write rewrite read reread read write
4 1325 1378 4922 4929 4412 224
512 9128 8563 21971 21989 21917 455
16384 8868 8909 22672 22676 22676 5962

Health summary: /dev/mmcblk1p1 failed. Replace it as soon as possible!
Data LOST: 4.00 KB (8 sectors)
Corrupted: 4.00 KB (8 sectors)
Slightly changed: 0.00 Byte (0 sectors)
Overwritten: 0.00 Byte (0 sectors)

Performance summary:
Sequential reading speed: 20.64 MB/s
4K random reading speed: 4412 KB/s
Sequential writing speed: 8.66 MB/s
4K random writing speed: 224 KB/s (way too low)

The device you tested seems to perform too slow to be used with Armbian.
This applies especially to desktop images where slow storage is responsible
for sluggish behaviour. If you want to have fun with your device do NOT use
this media to put the OS image or the user homedirs on.

tkaiser · July 22, 2018

19 minutes ago, valant said:

but the check revealed corruption of a FS block (8 sectors)

Ah, ok. You were really running the 'armbianmonitor -c' check testing for data integrity (I thought about a fsck first). Yeah, then the SD card is at fault and corrupting data.

tkaiser · July 22, 2018

30 minutes ago, valant said:

This card hasn't been used extensively. Almost new

Care to provide 'armbianmonitor -u' output to get SD card metadata? Would also be interesting if using SD Association's 'SD Formatter' can cure the card (unfortunately only available for macOS and Windows)

valant · July 22, 2018

34 minutes ago, tkaiser said:

Ah, ok. You were really running the 'armbianmonitor -c' check testing for data integrity (I thought about a fsck first). Yeah, then the SD card is at fault and corrupting data.

I was just struggling with formatting on the forum to post it not an ugly way, so it took some time (almost as much as the test did).

fsck said "everything is clean".

there was yet another fail. I inserted another SD card through USB-SD adapter into USB3 port, it's a good adapter it worked well before, and it was recognized by linux. but resizing the FS and partition went wrong at all. fdisk wrote zeros in the MBR instead of what I typed. and the USBCUV signature at the beginning of MBR. fdisk task succeeded only on the second time. fsck, resize2fs makefs.ext4 didn't behave properly. resize2fs reported it has done successfully but fsck couldn't find ext4 volume. makefs just exited in between the process without reporting success/fail. after inserting the adapter in the USB2 port it worked out. now I think what if really it's RAM is damaged in some way. would be so not cool...

Quote

Care to provide 'armbianmonitor -u' output to get SD card metadata? Would also be interesting if using SD Association's 'SD Formatter' can cure the card (unfortunately only available for macOS and Windows)

the link is http://ix.io/1i7t

the info on SD

Quote

### mmc1:0003 info:

cid: 4134325344313647304118014d0104d9

csd: 400e00325b59000073677f800a4000f1

scr: 0235800300000000

date: 04/2016

name: SD16G

type: SD preferred_erase_size: 4194304

fwrev: 0x0

hwrev: 0x3

oemid: 0x3432

manfid: 0x000041

serial: 0x4118014d

uevent: DRIVER=mmcblk MMC_TYPE=SD MMC_NAME=SD16G MODALIAS=mmc:block erase_size: 512

I'll try SD formatter later and tell. The second run of the armbianmonitor -c check was OK.

tkaiser · July 23, 2018

8 hours ago, valant said:

I inserted another SD card through USB-SD adapter into USB3 port ... after inserting the adapter in the USB2 port it worked out

Just to be clear: this was on Rock64 and you fiddled around with SD cards in an external USB card reader? Is the card reader SuperSpeed capable ('USB 3.0')?

valant · July 23, 2018

7 hours ago, tkaiser said:

Just to be clear: this was on Rock64 and you fiddled around with SD cards in an external USB card reader? Is the card reader SuperSpeed capable ('USB 3.0')?

yes, on Rock64. CardReader is USB2.

I reproduced the same error with NetworkManager generated file inserted into that poor package list file. after typing several times apt-get with different commands (purge, clean, autoclean, install, update) and dpkg --configure -a, package list file got corrupted again with exactly the same NM stuff...

every debian package system utility (apt-get, debconf-*) complained about syntax errors in Perl files. Mostly in File.pm. looking there showed something like this:


nless(XXX){}

    }

   }

  }

}

1;

^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;

with "nless" and mismatched curly braces. XXX - some C-macro looking thing, I forgot exact name of it.

Could this kind of damage be a result of a FS corruption? Are these perl library files generated dynamically somehow, or are they just unpacked from zipped archives during the installation?

Since it got quite psychedeIic, I just reinstalled Armbian on the same card and it worked well so far. And by the way the proper file File.pm even doesn't have "unless" keyword in it.

tkaiser · July 23, 2018

19 minutes ago, valant said:

Could this kind of damage be a result of a FS corruption?

Sure. That's for example one of the many symptoms of burning OS images not with Etcher when some slight data corruption happened (Etcher would've catched). Some bit flips somewhere in libraries and code jumps to wrong addresses and so on. But that's what 'armbianmonitor -v' is for (see explanation and example output above) and IIRC you also tested for this and got no corrupted packages reported?

valant · July 23, 2018

41 minutes ago, tkaiser said:

Sure. That's for example one of the many symptoms of burning OS images not with Etcher when some slight data corruption happened (Etcher would've catched). Some bit flips somewhere in libraries and code jumps to wrong addresses and so on. But that's what 'armbianmonitor -v' is for (see explanation and example output above) and IIRC you also tested for this and got no corrupted packages reported?

Yes I did, it reported corrupted packages, both times with -v, and with -c first time it detected corruption, the second time didn't. and exactly there were mostly perl related files. I showed this just removed file list, it was long.

tkaiser · July 23, 2018

15 minutes ago, valant said:

it reported corrupted packages, both times with -v

If 'armbianmonitor -v' reports corrupted packages then something went already horribly wrong and usually there's no easy way to recover from this other than starting from scratch. Seriously: if you have corrupted packages what should be the result if any other software calls such a corrupted library? It will just cause more and more harm.

valant · July 23, 2018

3 hours ago, tkaiser said:

If 'armbianmonitor -v' reports corrupted packages then something went already horribly wrong and usually there's no easy way to recover from this other than starting from scratch. Seriously: if you have corrupted packages what should be the result if any other software calls such a corrupted library? It will just cause more and more harm.

I fear it's resizing the FS went wrong. Too bad I didn't check packages before doing it. I have a question. Let's assume I shrink 14.3GB ext4 volume to 8GB doing:

resize2fs /dev/sda2 8G

and the utility tells me that the FS now is N 4KB blocks length.

Is N the number I should use for partition shrinking with fdisk (translated into number of sectors, of course)? Logically yes, but who knows.

I mean is N the whole size of the volume with metadata and data (free space included)?

tkaiser · July 23, 2018

18 minutes ago, valant said:

fdisk

Sorry, parted user here -- can't even remember what I did with fdisk last century

valant · July 23, 2018

haha. fdisk is the finishing touch with shrinking. I meant what the size does resize2fs report. apparently it should be the whole FS volume size, but I needed to ensure after all these fails.)

valant · July 25, 2018

just to summarize. the second flashing of the same card still had issues with packaging system, this time dpkg complained about "crontab" group missing (or present ) in statoverride file or something. so, I gave up and flashed armbian on another card. everything is working so far. I did full format of the offending card with the SDA formatter, and it reported no problems with it. so yeah, not very clear but most probably it was a problem with the SD card. thanks evrybody for the help.

Btw, have you noticed that attaching an HDMI monitor to the board adds to the SoC temperature ~9 C. Even if it's just a console output (no GUI). Having nothing attached except Ethernet, gives 43 C in the idle (heatsink mounted). With HDMI, it gives 52 C.

tkaiser · July 25, 2018

24 minutes ago, valant said:

have you noticed that attaching an HDMI monitor to the board adds to the SoC temperature ~9 C

As expected. Personally not experienced with any RK3328 device (since never attached displays to them) but with older Allwinner SoCs I measured a ~5°C difference with HDMI active or not.

It's just as adding a GPU to a PC: Wastes more energy and generates more heat. With SoCs it's more IP blocks active and HDMI PHY inside the SoC so more or less as expected (or an indication for a driver doing things correctly: detecting a display and if none present disabling stuff that's not needed)

Sign In

Package list error due to filesystem corruption

Recommended Posts

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Important Information