eMMC errors

alchemist · August 4, 2021

Hi,

I got write errors on the eMMC, and the filesystem becomes read-only.

Dmesg shows that:

[  304.982737] mmc2: running CQE recovery
[  304.987677] mmc2: running CQE recovery
[  304.989762] blk_update_request: I/O error, dev mmcblk2, sector 26826008 op 0x1:(WRITE) flags 0x4000 phys_seg 127 prio class 0
[  304.989791] EXT4-fs warning (device mmcblk2p1): ext4_end_bio:345: I/O error 10 writing to inode 789645 starting block 3353569)
[  305.021383] mmc2: running CQE recovery
[  305.026213] mmc2: running CQE recovery
[  305.030808] mmc2: running CQE recovery
[  305.032702] blk_update_request: I/O error, dev mmcblk2, sector 26265544 op 0x1:(WRITE) flags 0x4000 phys_seg 111 prio class 0
[  305.033382] EXT4-fs warning (device mmcblk2p1): ext4_end_bio:345: I/O error 10 writing to inode 793979 starting block 3283345)
[  305.109165] mmc2: running CQE recovery
[  305.114772] mmc2: running CQE recovery
[  305.120284] mmc2: running CQE recovery
[  305.125405] mmc2: running CQE recovery
[  305.126960] blk_update_request: I/O error, dev mmcblk2, sector 25560376 op 0x1:(WRITE) flags 0x0 phys_seg 15 prio class 0
[  305.126976] EXT4-fs warning (device mmcblk2p1): ext4_end_bio:345: I/O error 10 writing to inode 789517 starting block 3195094)
[  305.128183] mmc2: running CQE recovery
[  305.132052] mmc2: running CQE recovery
[  305.138255] mmc2: running CQE recovery
[  305.143172] mmc2: running CQE recovery
[  305.143705] blk_update_request: I/O error, dev mmcblk2, sector 25568256 op 0x1:(WRITE) flags 0x0 phys_seg 11 prio class 0
[  305.143719] EXT4-fs warning (device mmcblk2p1): ext4_end_bio:345: I/O error 10 writing to inode 789519 starting block 3196066)
[  305.146059] mmc2: running CQE recovery
[  305.150856] mmc2: running CQE recovery
[  305.153652] blk_update_request: I/O error, dev mmcblk2, sector 25566136 op 0x1:(WRITE) flags 0x0 phys_seg 14 prio class 0
[  305.153674] EXT4-fs warning (device mmcblk2p1): ext4_end_bio:345: I/O error 10 writing to inode 789524 starting block 3195813)
[  305.156107] mmc2: running CQE recovery
[  305.161761] mmc2: running CQE recovery
[  305.166281] mmc2: running CQE recovery
[  305.173900] mmc2: running CQE recovery
[  305.179678] mmc2: running CQE recovery
[  305.184108] mmc2: running CQE recovery
[  305.189973] mmc2: running CQE recovery
[  305.192654] blk_update_request: I/O error, dev mmcblk2, sector 25550264 op 0x1:(WRITE) flags 0x0 phys_seg 6 prio class 0
[  305.192674] EXT4-fs warning (device mmcblk2p1): ext4_end_bio:345: I/O error 10 writing to inode 789534 starting block 3193811)
[  305.195645] mmc2: running CQE recovery
[  305.201382] mmc2: running CQE recovery
[  305.206397] mmc2: running CQE recovery
[  305.209876] mmc2: running CQE recovery
[  305.212505] EXT4-fs warning (device mmcblk2p1): ext4_end_bio:345: I/O error 10 writing to inode 789547 starting block 3194323)
[  305.215282] mmc2: running CQE recovery
[  305.267254] mmc2: running CQE recovery
[  305.272126] mmc2: running CQE recovery
[  305.276368] mmc2: running CQE recovery
[  305.282479] mmc2: running CQE recovery
[  305.287200] mmc2: running CQE recovery
[  305.291543] mmc2: running CQE recovery
[  305.315011] JBD2: Detected IO errors while flushing file data on mmcblk2p1-8
[  305.317983] mmc2: running CQE recovery
[  305.322443] mmc2: running CQE recovery
[  305.329227] mmc2: running CQE recovery
[  305.330168] Aborting journal on device mmcblk2p1-8.
[  305.332721] mmc2: running CQE recovery
[  305.333640] EXT4-fs error (device mmcblk2p1): ext4_journal_check_start:83: Detected aborted journal
[  305.334031] EXT4-fs (mmcblk2p1): Remounting filesystem read-only
[  305.334048] EXT4-fs (mmcblk2p1): failed to convert unwritten extents to written extents -- potential data loss!  (inode 794035, error -30)
[  305.334380] EXT4-fs error (device mmcblk2p1): ext4_journal_check_start:83: Detected aborted journal
[  305.335183] EXT4-fs (mmcblk2p1): failed to convert unwritten extents to written extents -- potential data loss!  (inode 794033, error -30)
[  305.336335] EXT4-fs (mmcblk2p1): failed to convert unwritten extents to written extents -- potential data loss!  (inode 794038, error -30)
[  305.337472] EXT4-fs (mmcblk2p1): failed to convert unwritten extents to written extents -- potential data loss!  (inode 794037, error -30)
[  305.337497] EXT4-fs (mmcblk2p1): failed to convert unwritten extents to written extents -- potential data loss!  (inode 794036, error -30)
[  305.339676] EXT4-fs (mmcblk2p1): failed to convert unwritten extents to written extents -- potential data loss!  (inode 794039, error -30)

I did a fsck, and rebooted, but it gives the same kind of errors after that,

It seems I cannot write on eMMC anymore.

I use eMMC to store the boot script + a rescue subsystem (I boot it by renaming /boot.rescue to /boot). It is formatted as a ext4 partition.

What can I do to get back to a sane state ?

Kind regards,

Xavier Miller.

alchemist · August 4, 2021

FYI, I am running a custom kernel based on chosen armbian patches. Seems I need to add some more...

Re-patching and re-compiling until I get the kernel running (some arbmian patches don't compile on 5.10.55)

EDIT: reverted to an older kernel (5.10.34) and eMMC is operational :-)

ScottP · September 22, 2021

I get similar errors from any 5.13 or 5.14 kernel on nanopct4 which also has rk3399. Works OK on 5.10

prahal · January 19, 2022

On 9/22/2021 at 3:24 PM, ScottP said:

I get similar errors from any 5.13 or 5.14 kernel on nanopct4 which also has rk3399. Works OK on 5.10

I have bisected the issue for helios64. See:

Could you try to revert this commit and check the issue vanish ?

You could also try to revert those commits against latest kernel :

06653ebc0ad2e0b7d799cd71a5c2933ed2fb7a66

and probably also aea6cb99703e17019e025aa71643b4d3e0a24413 (both commits are related).

Point is that helios64 is not a mainline supported architecture (the patches for it are not upstream but only Armbian) so maybe you will be in a better position to report the regression upstream and have one investigate the roots of the regression.