OrangePI Zero: Card stuck in programming state

[MattCostamagna]

Hi,

 

I've been using Armbian for three years on some work's project and this is the first time I see this issue.

Basically I have an image of the operative system and all the programs and scripts already saved in it, so when I have to prepare a new SD card I just use the command "dd" and then I modify some registry stuff specific for the object I'm creating.

It has always worked without problem, but now we bought a new batch of SD cards and I'm facing this issue: the dd command goes succesfully and if I mount the partitions after it has finished I can see all my data, but when I boot the system sometimes I get some errors. Here is the last part of the boot log:

 

[   12.866043] systemd[1]: Mounted Kernel Configuration File System.
[   13.948285] thermal thermal_zone0: binding zone cpu_thermal with cdev thermal-cpufreq-0 failed:-22
[   93.570475] mmc0: Card stuck in programming state! mmc_do_erase
[   94.342442] sunxi-mmc 1c0f000.mmc: fatal err update clk timeout
[   95.110441] sunxi-mmc 1c0f000.mmc: fatal err update clk timeout
[   95.870438] sunxi-mmc 1c0f000.mmc: fatal err update clk timeout
[   95.877698] blk_update_request: I/O error, dev mmcblk0, sector 1863688 op 0x3:(DISCARD) flags 0x800 phys_seg 1 prio class 0
[   95.889014] blk_update_request: I/O error, dev mmcblk0, sector 1281568 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
[   95.889652] EXT4-fs error (device mmcblk0p1): __ext4_find_entry:1532: inode #13767: comm fstrim: reading directory lblock 0
[   95.900257] EXT4-fs error (device mmcblk0p2): __ext4_get_inode_loc:4699: inode #135743: block 524611: comm rc.local: unable to read itable block
[   95.911460] EXT4-fs (mmcblk0p1): I/O error while writing superblock
[   95.924553] EXT4-fs (mmcblk0p2): I/O error while writing superblock
[   95.937750] EXT4-fs error (device mmcblk0p1): __ext4_find_entry:1532: inode #13767: comm fstrim: reading directory lblock 0
[   95.938429] EXT4-fs error (device mmcblk0p1): __ext4_find_entry:1532: inode #13767: comm find: reading directory lblock 0
[   95.949147] EXT4-fs (mmcblk0p1): I/O error while writing superblock
[   95.950615] EXT4-fs error (device mmcblk0p2): __ext4_get_inode_loc:4699: inode #135743: block 524611: comm rc.local: unable to read itable block
[   95.950697] EXT4-fs (mmcblk0p2): I/O error while writing superblock
[   95.960128] EXT4-fs (mmcblk0p1): previous I/O error to superblock detected
[   95.967624] EXT4-fs error (device mmcblk0p1): __ext4_find_entry:1532: inode #13767: comm fstrim: reading directory lblock 0
[   95.979752] EXT4-fs (mmcblk0p1): I/O error while writing superblock
[   95.985982] EXT4-fs (mmcblk0p1): previous I/O error to superblock detected
[   95.993275] EXT4-fs error (device mmcblk0p1): __ext4_find_entry:1532: inode #13767: comm find: reading directory lblock 0
[   96.004308] EXT4-fs (mmcblk0p1): I/O error while writing superblock
[   96.010602] EXT4-fs (mmcblk0p1): previous I/O error to superblock detected
[   96.010702] EXT4-fs (mmcblk0p1): I/O error while writing superblock
[   96.018166] EXT4-fs error (device mmcblk0p1): __ext4_find_entry:1532: inode #13767: comm fstrim: reading directory lblock 0
[   96.029509] EXT4-fs error (device mmcblk0p1): __ext4_find_entry:1532: inode #13767: comm find: reading directory lblock 0
[   96.032375] EXT4-fs error (device mmcblk0p1): __ext4_find_entry:1532: inode #99: comm systemd-udevd: reading directory lblock 0
[   96.032420] EXT4-fs (mmcblk0p1): I/O error while writing superblock
[   96.035115] EXT4-fs (mmcblk0p1): I/O error while writing superblock
[   96.094862] EXT4-fs error (device mmcblk0p1): __ext4_find_entry:1532: inode #13767: comm find: reading directory lblock 0
[   96.094899] EXT4-fs error (device mmcblk0p1): __ext4_find_entry:1532: inode #13767: comm fstrim: reading directory lblock 0
[   96.918443] sunxi-mmc 1c0f000.mmc: fatal err update clk timeout
[   97.674443] sunxi-mmc 1c0f000.mmc: fatal err update clk timeout
[   98.442444] sunxi-mmc 1c0f000.mmc: fatal err update clk timeout
[   99.210440] sunxi-mmc 1c0f000.mmc: fatal err update clk timeout
[   99.974443] sunxi-mmc 1c0f000.mmc: fatal err update clk timeout
[  100.730444] sunxi-mmc 1c0f000.mmc: fatal err update clk timeout
[  100.810651] debugfs: Directory 'mmcblk0' with parent 'block' already present!
[  125.415973] EXT4-fs error (device mmcblk0p1): __ext4_find_entry:1532: inode #7779: comm cron: reading directory lblock 0
[  125.427247] EXT4-fs error (device mmcblk0p1): __ext4_find_entry:1532: inode #7779: comm cron: reading directory lblock 0
[  125.438653] EXT4-fs error (device mmcblk0p1): __ext4_find_entry:1532: inode #46845: comm cron: reading directory lblock 0
[  125.450113] EXT4-fs error (device mmcblk0p1): __ext4_find_entry:1532: inode #7779: comm cron: reading directory lblock 0
[  125.461370] EXT4-fs error (device mmcblk0p1): __ext4_find_entry:1532: inode #7779: comm cron: reading directory lblock 0
[  125.472605] EXT4-fs error (device mmcblk0p1): __ext4_find_entry:1532: inode #7779: comm cron: reading directory lblock 0
[  125.483833] EXT4-fs error (device mmcblk0p1): __ext4_find_entry:1532: inode #46845: comm cron: reading directory lblock 0

 

Sometimes it manages to exit from the error state, other times it keeps logging the same error message.

We have tested the SD cards using some broken sectors detector software, and none of them found any problems. The source image is working because if I use an old SD card, I don't see any errors.

I tried dd-ing a vanilla version of Armbian into one of the SD cards that is giving me problems and it is working nicely.

I tried the same "broken-OS SD cards" into differents OrangePI and none of them worked.

 

The problem happens sometimes at boot, sometimes after a couple of minutes (I saw some kernel panic).

This is the output of "uname -a": Linux smartbit 5.4.45-sunxi #20.05.3 SMP Wed Jun 10 12:09:20 CEST 2020 armv7l armv7l armv7l GNU/Linux

The "dd" command (which is working on old SD cards) is the following: dd if=./latest.img of=/dev/sdd bs=1M conv=fsync

 

So if the problem is in the source image, why is it working on the old SD cards batch?

Otherwise, if the problem is in the SD card, why an Armbian vanilla image isn't giving any problems?

 

Thanks a lot in advance for any help.

Mattia

 

[Werner]

Did you verify somehow that the written data actually matching the image?

[MattCostamagna]

Actually I didn't.

I supposed that if dd doesn't return any error it means that the operation went succesfully.

What can I do to verify it?

[Werner]

You can try using usbimager or Balenaetcher or any other tool that verifies written data.

[MattCostamagna]

Hi @Werner

I tried making an exact copy of the source image using etcher without modifying anything else.

The validation of the written data was successful and I didn't get any errors, but unfortunately the result was the same: card stuck in programming state and sometimes I can't even reach the login.

 

[TRS-80]

I will just interject to say that sdcard errors are so common that verification needs to be done first in order to be eliminated before proceeding to other troubleshooting steps.  So thanks for taking that necessary step.  Now if anyone knowledgeable read this, they are more likely to respond.

 

For example, only after writing above paragraph do I decide to actually go and  read OP. 

 

And now I think possibly the card might be starting to go out.  How old is it and how much use has it seen?  As you can see, we are right back to suspecting sdcard again (it is very, very common).

[MattCostamagna]

@TRS-80 Thanks for your reply.

The SD cards are brand new and they've never been used.

We have verified that all sectors are writable using h2testw and we've got zero errors.

We also tried different SD cards coming from the same batch but the outcome was the same.

 

[Werner]

What happens if you

On 11/19/2020 at 9:52 AM, MattCostamagna said:

I tried dd-ing a vanilla version of Armbian into one of the SD cards that is giving me problems and it is working nicely.

I assume you meant to say "giving me no problems", am I correct?

[MattCostamagna]

@Werner I meant that I tried flashing a vanilla version of Armbian into one of the SD cards that are not working with the production image I've been using, and with a vanilla image I don't see the errors that I see with the other image (which however works with the old SD cards).

Hopefully that's more clear.

 

[TRS-80]

4 hours ago, MattCostamagna said:

The SD cards are brand new and they've never been used.

We have verified that all sectors are writable using h2testw and we've got zero errors.

We also tried different SD cards coming from the same batch but the outcome was the same.

 

OK, this pretty much eliminate such concerns, good work.

 

Reading more carefully now your last post:

 

2 hours ago, MattCostamagna said:

I meant that I tried flashing a vanilla version of Armbian into one of the SD cards that are not working with the production image I've been using, and with a vanilla image I don't see the errors that I see with the other image (which however works with the old SD cards).

 

... made me go back and read OP (again) more closely, and I notice:

 

On 11/19/2020 at 8:52 AM, MattCostamagna said:

I have an image of the operative system and all the programs and scripts already saved in it, so when I have to prepare a new SD card I just use the command "dd" and then I modify some registry stuff specific for the object I'm creating.

It has always worked without problem, but now we bought a new batch of SD cards and I'm facing this issue: the dd command goes succesfully and if I mount the partitions after it has finished I can see all my data, but when I boot the system sometimes I get some errors.

 

OK, so some times things are updated in new Armbian versions, including perhaps even bootloaders, naming conventions, etc.  So perhaps some of those "other" things been updated in the meantime.  This may be why "vanilla" Armbian works fine, but whatever you have just keep copying forward, does not work any longer?

[MattCostamagna]

Thanks @TRS-80

10 minutes ago, TRS-80 said:

OK, so some times things are updated in new Armbian versions, including perhaps even bootloaders, naming conventions, etc.  So perhaps some of those "other" things been updated in the meantime. 

That might be the reason (also because I've excluded all the other possible causes), but I don't understand what could be the cause...

 

I didn't quite understand what you meant here:

8 minutes ago, TRS-80 said:

This may be why "vanilla" Armbian works fine, but whatever you have just keep copying forward, does not work any longer?

 

And yes, I can make a new version of the image of our project, but before doing that I wanted to be sure that I can't do anything else.

[TRS-80]

22 minutes ago, MattCostamagna said:

And yes, I can make a new version of the image of our project, but before doing that I wanted to be sure that I can't do anything else.

 

It's not that you "can't" but rather, tracking that down and finding out exactly what and where went wrong may take longer and be more effort than just sort of starting anew, blowing out the whole OS and flashing fresh, etc.  The decision will weigh on just how heavily configured your old setup is, and how much effort that is to reproduce on the new OS.

 

I ran in to, not exactly the same issue as you, recently, however it was a similar sort of decision to make (i.e., whether or not to re-flash OS again "from scratch").  In the end, I was able to get it working, luckily (as I had quite some config on that old Cubietruck!).

 

However the experience made me think more in general about how I organize my "services" which are running on top of base Armbian.  Now in my case I'm just a home gamer, running few services like an XMPP server for friends and family, etc.  In my case I start to play with containers, because I think that is a solution that will work for my needs while isolating the "services" from the underlying OS.  Then I can blow out the whole OS with the latest from scratch and it's no problem.  Such is the nature of these devices and the hardware support (well, at least on the relatively limited resources a community project such as Armbian has to work with, anyway).

 

I have no idea what "your project" is or if such a solution might work for you.  And those sort of questions are slightly outside the scope of the Armbian project to be honest (I just have a personal interest in this area, and thus am sharing my thoughts / experience).  However another path may be to hire @Igor and/or some of project devs to sort out whatever issue you are having, especially if this is some commercial project and you can afford to do so (I have no idea, just throwing it out there).

 

Another solution, if you don't like the idea of containers (or they are not appropriate for whatever reason), may be to come up with some sort of Ansible type thing, or even a Makefile to provision "your project" on top of a base install of Armbian.  There are many solutions, which is best depends a great deal on exactly what "your project" really is...

[MattCostamagna]

Yes you are right, making a new image won't take me that long and it's something I've been considering since I saw the sort of problem I was facing.

I'll just leave this thread as unresolved so if someone in the future will manage to find a solution they can post the answer here.

 

Thanks a lot for your support.

[TRS-80]

Please report back if that ends up working for you, or you ever figure out the problem otherwise, etc...

 

And, you are welcome!