[Odroid C2] filesystem corrupted after boot from emmc

[michael]

I created a new image for my Odroid C2 with the most current version of Armbian 5.26. Configuration was pretty much standard, except BOOTSIZE=0 and ROOTFS_TYPE=ext4. The image works fine, if dumped to a sdcard. When dumping it to an emmc card, the system boots up, but 'find / > /dev/null' leads to a filesystem error:

 

[   52.799920] EXT4-fs error (device mmcblk0p1): htree_dirblock_to_tree:913: inode #41001: block 12032: comm find: bad entry in directory: rec_len % 4 != 0 - offset=0(0), inode=8213, rec_len=4097, name_len=0
[   52.812862] EXT4-fs (mmcblk0p1): Remounting filesystem read-only

 

Behavior is deterministic - I tried with two different boards and two different emmc cards. I also checked, that the filesystem is initially ok on the emmc.

 

I tracked the problem down to the Amlogic mmc driver in linux kernel 3.14.y. During kernel boot, the function aml_sd_emmc_execute_tuning in drivers/amlogic/mmc/aml_sd_emmc.c is invoked. pdata->need_cali == 1 and aml_sd_emmc_execute_tuning_index is called. At the end, this function writes to the emmc: mmc_write_internal(mmc->card, record_blk, 1, write_buf).

The first two bytes of the written data are 0x2015, 0x1001 - which perfectly corresponds to the inode and rec_len from the error message above.

The position where the data is written at is calculated in emmc_partitions.c (function get_reserve_partition_off) and depends on several constants. The position evaluates to block #98304, which is somewhere in the root partition of the disk. Since in my image, this block is used by a directory inode, the filesystem gets corrupted.

 

On an image with a 64MB boot partition and an ext4 root partition, the write operation goes to the boot partition. It does not seem to have a negative impact on the consistency of the filesystem - at least not in my case.

 

I could imagine different solutions of the problem:

(1) set OFFSET=64 -> write operation will go to the unused space before boot / root partition

(2) change constants in linux kernel -> write operation will go to a different block on emmc (i cannot guess potential side effects of this)

(3) change partition layout on emmc by changing calculation in debootstrap-ng.sh

 

 

Best regards,

Michael

[grirant]

Hi Michael,

 

I have the same trouble with my emmc. Thought it is an hardwarre issue (till today) and bought a new one. Today, I did an apt update && apt upgrade on my Odroid C2 that works as a headless Server. During the update, I got the following entry in the syslog:

 

[1192245.968722] EXT4-fs error (device mmcblk0p1): ext4_find_dest_de:1639: inode #43529: block 12032: comm dpkg: bad entry in directory: rec_len % 4 != 0 - offset=0(0), inode=8213, rec_len=4097, name_len=0

[1192245.985979] Aborting journal on device mmcblk0p1-8.

[1192245.991429] EXT4-fs (mmcblk0p1): Remounting filesystem read-only

[1192245.997249] EXT4-fs error (device mmcblk0p1) in ext4_evict_inode:241: Journal has aborted

[1192246.007682] EXT4-fs error (device mmcblk0p1) in ext4_create:2249: IO failure

 

When I compare it with your log entry, it looks like the same Problem. Hope, there will be a fix soon.

[zador.blood.stained]

I tracked the problem down to the Amlogic mmc driver in linux kernel 3.14.y. During kernel boot, the function aml_sd_emmc_execute_tuning in drivers/amlogic/mmc/aml_sd_emmc.c is invoked. pdata->need_cali == 1 and aml_sd_emmc_execute_tuning_index is called. At the end, this function writes to the emmc: mmc_write_internal(mmc->card, record_blk, 1, write_buf).

The first two bytes of the written data are 0x2015, 0x1001 - which perfectly corresponds to the inode and rec_len from the error message above.

The position where the data is written at is calculated in emmc_partitions.c (function get_reserve_partition_off) and depends on several constants. The position evaluates to block #98304, which is somewhere in the root partition of the disk. Since in my image, this block is used by a directory inode, the filesystem gets corrupted.

Thanks for investigations. This "calibration" looks strange and it will be disabled if not required for proper eMMC functioning (?), otherwise big OFFSET may work to solve this too.

[Igor]

https://github.com/igorpecovnik/lib/commit/5e36adca766b3afee8c1cbba5b2091552232646d

Try to reproduce now. 

Strangely, I can't reproduce even on default image ...

[zador.blood.stained]

It may depend on eMMC "type"/version - there are some checks in the code that enables this "calibration". I'll ad a patch that disables this later.

[michael]

Igor, thanks for the quick change to the board configuration file. Setting an OFFSET seems to be the best solution - at least for new installations.

Unfortunately, OFFSET=12 is too low to solve the issue. The data is written to block #98304, which means that OFFSET has to be greater than 98304 / 2048. So OFFSET must be at least 49.

 

I also reported the issue to the Odroid people: http://forum.odroid.com/viewtopic.php?f=141&t=26097&sid=bdee0ffe9e7193c430136dd2cc2450a3

The promised to have a look at it. I will let you know, if they come up with a fix to the kernel source.

[Igor]

Than I guess I made a wrong math  I looked upon a code but its messy and it's hard to understand ... proposed option B is possible to apply, but it would be nice to understand what does that means.

 

Thanks. I am looking forward for info / hint from Hardkernel.