rsync between two external hdds caused rootfs vanished

[Jerry Jyrer]

Board: Nanopi M4 with 4GB

OS: Armbian_5.65_Nanopim4_Debian_stretch_default_4.4.162

Boot from SD, rootfs on an external HDD 8TB (WD MyBook).

 

I attached another external HDD (Toshiba 2TB). I have been trying rsync -avh from the Toshiba one to the main MyBook one. It ran for under a minute before reporting files were vanished and exited with errors. It appeared the / is gone. nothing return from ls. Hard-reboot, fsck ran, everything's back to normal. 

 

I've been trying rsync with nice -n 20 as well as --bwlimit 10, 1MB. Nothing helped.  stress test the board for 10 minutes, (stress -c 4 -m 4 -d 4 -i 4). All looks good. 

 

Any help's appreciated.

 

Thanks,

Jerry

[guidol]

3 hours ago, Jerry Jyrer said:

Boot from SD, rootfs on an external HDD 8TB (WD MyBook).

I attached another external HDD (Toshiba 2TB). I have been trying rsync -avh from the Toshiba one to the main MyBook one. It ran for under a minute before reporting files were vanished and exited with errors. It appeared the / is gone. nothing return from ls. Hard-reboot, fsck ran, everything's back to normal. 

Any help's appreciated.

Please provide the complete rsync command (with directorynames).

Are you sure that you didnt overwrite something important with the rsync command on your root-device.
Why do you want to rsync from the Toshiba to the WD MyBook root-device?

 

BTW: I also got a "old" WD MyBook device, but wouldnt use it as a root device, because this "thing" goes very oft in sleep/ide-state and spinning down.

[Jerry Jyrer]

1 hour ago, guidol said:

Please provide the complete rsync command (with directorynames).

Are you sure that you didnt overwrite something important with the rsync command on your root-device.
Why do you want to rsync from the Toshiba to the WD MyBook root-device?

 

BTW: I also got a "old" WD MyBook device, but wouldnt use it as a root device, because this "thing" goes very oft in sleep/ide-state and spinning down.

 

$ sudo mount /dev/sdb1 /mnt

$ sudo mkdir -p /MyBackup

$ sudo rsync -avh --progress /mnt/ /MyBackup/

 

Well as for the reason, (1) testing and (2) mainly, 3-2-1 backup. 

 

Another observation, 

$ sudo smartctl -x /dev/sda1

 

showing:

 

SMART support is:     Unavailable - device lacks SMART capability.

 

pretty weird. 

 

I'm running 

 

$armbianmonitor -c "$HOME"

 

to test my new 8TB MyBook now. It will take awhile but so far so good. I believe it's finishing writing >1TB; 7TB to go.

[guidol]

1 hour ago, Jerry Jyrer said:

 

$ sudo mount /dev/sdb1 /mnt

$ sudo mkdir -p /MyBackup

$ sudo rsync -avh --progress /mnt/ /MyBackup/

the commands does look OK for me (referencing to https://www.howtogeek.com/135533/how-to-use-rsync-to-backup-your-data-on-linux/ )

 

just a shot in the blue - try to mount the Toshiba not directly as /mnt but as /mnt/toshiba

 

How are the drives connected to the NanoPi M4?  USB 3.0 or SATA?
Which filesystem do you use (MyBook I think ext4 - but is the Toshiba also ext4 or something like NTFS)?

 

Testing the MyBook is a very good idea

[martinayotte]

6 hours ago, Jerry Jyrer said:

$ sudo rsync -avh --progress /mnt/ /MyBackup/

If you are doing backup of currently running rootfs, be careful not to backup some virtual folder such as /dev, /proc, /run, /sys and /tmp, neither /lost+found ...

[Jerry Jyrer]

45 minutes ago, martinayotte said:

If you are doing backup of currently running rootfs, be careful not to backup some virtual folder such as /dev, /proc, /run, /sys and /tmp, neither /lost+found ...

Thank you. Yes I'll be aware. Currently, all is pure data.

 

5 hours ago, guidol said:

just a shot in the blue - try to mount the Toshiba not directly as /mnt but as /mnt/toshiba

 

How are the drives connected to the NanoPi M4?  USB 3.0 or SATA?
Which filesystem do you use (MyBook I think ext4 - but is the Toshiba also ext4 or something like NTFS)?

 

Testing the MyBook is a very good idea

Yes, the media test is still on going. I'll test out /mnt/toshiba mount point once the test finished.

 

My book is ext4. Toshiba is NTFS. Both of them are connected to USB 3.0 ports. There 4 of them on NanoPi M4. I tried all of them and got the result.

[Jerry Jyrer]

Just back from the holiday. I tested the media. Both hard drives are healthy. I tested to mount /mnt/toshiba. Same problem. rsync just quited and after that no files seem to be mounted. A hard restart (pulling of power) was required.

 

Here is $sudo armbianmonitor -u

Just after the restarted with the Toshiba unattached.

 

http://ix.io/1uz5 

 

Any help would be appreciated. 

 

Thanks,

Jerry

[Jerry Jyrer]

I'll dig a bit more but so far here is what looks like to be the problem:

 

[ 1008.854862] xhci-hcd xhci-hcd.6.auto: xHCI host not responding to stop endpoint command.

[ 1008.854871] xhci-hcd xhci-hcd.6.auto: Assuming host is dying, halting host.

[ 1008.854943] xhci-hcd xhci-hcd.6.auto: xHCI host not responding to stop endpoint command.

[ 1008.854958] xhci-hcd xhci-hcd.6.auto: Assuming host is dying, halting host.

[ 1008.868355] xhci-hcd xhci-hcd.6.auto: HC died; cleaning up

[ 1008.868414] xhci-hcd xhci-hcd.6.auto: HC died; cleaning up

[ 1008.868420] usb 3-1: USB disconnect, device number 2

[ 1008.868928] usb 4-1: USB disconnect, device number 2

[ 1008.868934] usb 4-1.3: USB disconnect, device number 3

[ 1008.875798] sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x01 driverbyte=0x00

[ 1008.875808] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x8a 8a 00 00 00 00 00 00 c5 88 00 00 00 00 f0 00 00

[ 1008.875812] blk_update_request: I/O error, dev sda, sector 12945408

[ 1008.875898] sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x01 driverbyte=0x00

[ 1008.875903] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x8a 8a 00 00 00 00 00 00 c5 88 f0 00 00 00 f0 00 00

[ 1008.875907] blk_update_request: I/O error, dev sda, sector 12945648

[ 1008.883757] EXT4-fs warning (device sda1): ext4_end_bio:330: I/O error -5 writing to inode 40435820 (offset 0 size 53248 starting block 323676765)

[ 1008.883767] Buffer I/O error on device sda1, logical block 323676496

[ 1008.883774] Buffer I/O error on device sda1, logical block 323676497

[ 1008.883778] Buffer I/O error on device sda1, logical block 323676498

[ 1008.883783] Buffer I/O error on device sda1, logical block 323676499

[ 1008.883786] Buffer I/O error on device sda1, logical block 323676500

[ 1008.883790] Buffer I/O error on device sda1, logical block 323676501

[ 1008.883793] Buffer I/O error on device sda1, logical block 323676502

[ 1008.883796] Buffer I/O error on device sda1, logical block 323676503

[ 1008.883800] Buffer I/O error on device sda1, logical block 323676504

[ 1008.883803] Buffer I/O error on device sda1, logical block 323676505

[ 1008.883852] EXT4-fs warning (device sda1): ext4_end_bio:330: I/O error -5 writing to inode 40438764 (offset 0 size 413696 starting block 323676866)

[ 1008.883975] EXT4-fs warning (device sda1): ext4_end_bio:330: I/O error -5 writing to inode 40501734 (offset 16777216 size 4980736 starting block 1618432)

[ 1008.884650] EXT4-fs warning (device sda1): ext4_end_bio:330: I/O error -5 writing to inode 40501734 (offset 16777216 size 4980736 starting block 1618688)

[ 1008.885322] EXT4-fs warning (device sda1): ext4_end_bio:330: I/O error -5 writing to inode 40501734 (offset 16777216 size 4980736 starting block 1618944)

[ 1008.885887] EXT4-fs warning (device sda1): ext4_end_bio:330: I/O error -5 writing to inode 40501734 (offset 16777216 size 4980736 starting block 1619200)

[ 1008.886444] EXT4-fs warning (device sda1): ext4_end_bio:330: I/O error -5 writing to inode 40501734 (offset 16777216 size 4980736 starting block 1619392)

[ 1008.886944] EXT4-fs warning (device sda1): ext4_end_bio:330: I/O error -5 writing to inode 40501734 (offset 21757952 size 1572864 starting block 1619648)

[ 1008.887502] EXT4-fs warning (device sda1): ext4_end_bio:330: I/O error -5 writing to inode 40501734 (offset 21757952 size 1572864 starting block 1619776)

[ 1008.887797] EXT4-fs warning (device sda1): ext4_end_bio:330: I/O error -5 writing to inode 40501734 (offset 23330816 size 1310720 starting block 1620032)

[ 1008.890389] JBD2: Detected IO errors while flushing file data on sda1-8

[ 1008.890521] Aborting journal on device sda1-8.

[ 1008.890657] JBD2: Error -5 detected when updating journal superblock for sda1-8.

[ 1008.890829] JBD2: Detected IO errors while flushing file data on sda1-8

[ 1008.901805] sd 1:0:0:0: [sdb] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=0x00

[ 1008.901819] sd 1:0:0:0: [sdb] tag#0 CDB: opcode=0x28 28 00 39 1d dd f0 00 00 f0 00

[ 1008.901825] blk_update_request: I/O error, dev sdb, sector 958258672

[ 1008.922339] usb 4-1.4: USB disconnect, device number 4

[ 1008.925822] sd 1:0:0:0: [sdb] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x01 driverbyte=0x00

[ 1008.925841] sd 1:0:0:0: [sdb] tag#0 CDB: opcode=0x28 28 00 39 1d de e0 00 00 10 00

[ 1008.925848] blk_update_request: I/O error, dev sdb, sector 958258912

[ 1008.925987] sd 1:0:0:0: [sdb] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x01 driverbyte=0x00

[ 1008.925995] sd 1:0:0:0: [sdb] tag#0 CDB: opcode=0x28 28 00 39 1d de f0 00 00 f0 00

[ 1008.926001] blk_update_request: I/O error, dev sdb, sector 958258928

[ 1008.927276] EXT4-fs error (device sda1): ext4_journal_check_start:56: Detected aborted journal

[ 1008.928055] EXT4-fs (sda1): Remounting filesystem read-only

[ 1008.928544] bdi-block not registered

[Jim MacKenzie]

I may be having a similar problem.  I am using a NanoPi Neo4 and my root filesystem is on eMMC, but I have a large RAID6 attached by USB, on which I back up my server (this box lives at a family member's home, offsite).  The RAID used to live on a SheevaPlug, but I've replaced it with the NanoPi because of performance issues.  (Got a good decade out of the Sheeva!)

 

The RAID devices seem to disappear as soon as heavy I/O occurs.  It first occurred when I started an rsync from my server.  Rebooting the system became impossible without disconnecting the USB hub because the RAID rebuild would cause the same problem to recur immediately.  This was all with a four-port hub connected to the USB 3.0 port.

 

I initially thought it was a power issue, so replaced the four-port unpowered hub with a ten-port powered hub (I don't need ten ports, but I had this unit kicking around).  This did nothing to cure the problem.  What did solve the problem was moving the hub's connection from the USB 3.0 port to the USB 2.0 port.

 

I'm not sure if it's significant, but I notice in /boot/armbianEnv.txt:

usbstoragequirks=0x2537:0x1066:u,0x2537:0x1068:u,0x0bc2:0x331a:u,0x0bc2:0x3322:u
 

And my hard disks are in that quirks "blacklist":

root@nobby:/boot# lsusb | grep Seagate
Bus 005 Device 009: ID 0bc2:3322 Seagate RSS LLC 
Bus 005 Device 008: ID 0bc2:331a Seagate RSS LLC 
Bus 005 Device 007: ID 0bc2:3322 Seagate RSS LLC 
Bus 005 Device 006: ID 0bc2:3322 Seagate RSS LLC 
 

Jim

[Igor]

13 hours ago, Jim MacKenzie said:

I'm not sure if it's significant, but I notice in /boot/armbianEnv.txt:

usbstoragequirks=0x2537:0x1066:u,0x2537:0x1068:u,0x0bc2:0x331a:u,0x0bc2:0x3322:u

This is because of:
http://linux-sunxi.org/USB/UAS

[Jerry Jyrer]

On 2/24/2019 at 7:19 AM, Jim MacKenzie said:

What did solve the problem was moving the hub's connection from the USB 3.0 port to the USB 2.0 port.

Yes, exactly what I finally solved my problem. Same to you Jim, I initially thought and be advised by many that it was the power problem but at the end what solved was to connect one with an USB 2.0 port.

[Jim MacKenzie]

1 hour ago, Jerry Jyrer said:

Yes, exactly what I finally solved my problem. Same to you Jim, I initially thought and be advised by many that it was the power problem but at the end what solved was to connect one with an USB 2.0 port.

So then the question is: is the USB 3.0 hardware bad on this device, or is it a driver issue and it can be corrected?

 

I've certainly found USB 3.0 to be a delicate thing compared to USB 2.0.  I've had many, many difficulties with 3.0 (all surmounted over time, thankfully, save this one).  The only issue I ever had with 2.0 and Linux was a using-the-port-at-1.1-speeds problem; everything still worked.

 

If I can do anything to help give diagnostic information to developers, please let me know, but it sounds like this problem is easily reproducible.

[Jerry Jyrer]

On 2/25/2019 at 11:14 AM, Jim MacKenzie said:

If I can do anything to help give diagnostic information to developers, please let me know, but it sounds like this problem is easily reproducible.

Totally agreed. I posted here and similar topics for the same issue but so far no real solutions. Many suggestions were around under voltage being supplied. I personally doubt if that is the root cause. 

 

Armbian software/driver itself? Not too sure I tried the stock one from FriendlyElec, the result is the same. Is it the upstream Linux? Possible but no verification. I have just tried after recent apt update and upgrade. The same problem is still reproducible.

[dkking]

is it the same issue faced by users of rock64? ayufan Rock64 and rock64 Armbian

[Jerry Jyrer]

On 3/19/2019 at 8:45 AM, dkking said:

is it the same issue faced by users of rock64? ayufan Rock64 and rock64 Armbian

 

Quick scan, it looks like the same. Solution / root cause?

[dkking]

1 hour ago, Jerry Jyrer said:

 

Quick scan, it looks like the same. Solution / root cause?

I have no clue but if the problem is shared across devices of Rockchip, could it be a Rockchip, chipset /usb host controller issue? something Rockchip has to fix? I have heard the Odroid XU4 has no such issues (not Rockchip CPU, Exynos) or the 'up board'(Intel)

 

this is after you rule out the adequate power question(which I believe is not the problem)
 

[Mathias]

Jumping in a little late, but on the RockPro64 (also Rk3399), the same kind of problems happen (also very easily reproducible, it happens every single time under load). See my post on Pine64's forums. I am in contact with them (well, so far they just keep bouncing me around to another forum / email / person...) to try learning from them if they ever managed to get some sort of usb3 working on this board. Other people are also wondering if the hardware is the root cause of these issues...

[Mathias]

Replying even later... This definitely has to do with USB suspend (see this link): on a freshly booted system, no problems at all. On a system that has been running for a day (ie the USB had enough time to suspend), the problem is back with tons of error messages...

 

[edit] The autosuspend is not the culprit... This has to do with some underpower, see this (long) thread. Sorry for the fake news and the disappointment!

Edited June 19, 2019 by Mathias
More information after more testing

[Jim MacKenzie]

Not solvable, then?

[Jerry Jyrer]

On 7/5/2020 at 8:16 AM, Jim MacKenzie said:

Not solvable, then?

 

Somehow as of today, I tried again and it seems the problem is no longer seen. At least so far, rsync between two hard drives about 30 hrs is working fine.

 

```

$ cat /etc/armbian-release 
# PLEASE DO NOT EDIT THIS FILE
BOARD=nanopim4
BOARD_NAME="NanoPi M4"
BOARDFAMILY=rk3399
BUILD_REPOSITORY_URL=https://github.com/armbian/build
BUILD_REPOSITORY_COMMIT=b0760915-dirty
DISTRIBUTION_CODENAME=bionic
DISTRIBUTION_STATUS=supported
VERSION=20.11
LINUXFAMILY=rk3399
BRANCH=legacy
ARCH=arm64
IMAGE_TYPE=stable
BOARD_TYPE=conf
INITRD_ARCH=arm64
KERNEL_IMAGE_TYPE=Image
```

[Jim MacKenzie]

On 12/5/2020 at 3:40 AM, Jerry Jyrer said:

 

Somehow as of today, I tried again and it seems the problem is no longer seen. At least so far, rsync between two hard drives about 30 hrs is working fine.

 

 

Looks like you're using the Ubuntu Armbian, not the Debian - do you mind telling me what kernel you're using?

 

(A pity I only saw this now.)

[Jerry Jyrer]

57 minutes ago, Jim MacKenzie said:

Looks like you're using the Ubuntu Armbian, not the Debian - do you mind telling me what kernel you're using?

 

(A pity I only saw this now.)

 

Right, between the original post and now, I had been switching and trying different Armbian Ubuntu, Debian with Buster, and other variations. None of them worked at the time. I stopped at some point and the workaround was to connect a usb 2.0 hub as a middle man. Recently, I found myself need to do a full local rsync between hard drives again and it works. I'm currently with 

 

Armbian 21.02.1 Bionic with Linux 4.4.213-rk3399

[Igor]

There are only two different hardware layers. Stock kernel (4.4.x) and mainline (5.10.y)  based. Debian or Ubuntu is just application lawyer which doesn't have any effects on board hardware. Check modern kernel - it should be much more stable. 

[Jim MacKenzie]

3 hours ago, Igor said:

There are only two different hardware layers. Stock kernel (4.4.x) and mainline (5.10.y)  based. Debian or Ubuntu is just application lawyer which doesn't have any effects on board hardware. Check modern kernel - it should be much more stable. 

 

Thanks, Igor.  What is the simplest way to shift to mainline?  (Preferably without reinstalling.)  i.e. Which linux-image package is the correct one for this platform?

[Igor]

armbian-config -> system -> alternative kernel -> rockchip64-current v21.02.2 ...

 

but I can't give you warranty that it will succeed. If it breaks down, we can't afford to guide you, so you will anyway need to start from scratch or try to fix this way. You will also lose certain functions that are present in 4.4.y but not yet in modern one.

 

[Jerry Jyrer]

On 2/15/2021 at 2:07 PM, Igor said:

armbian-config -> system -> alternative kernel -> rockchip64-current v21.02.2 ...

 

but I can't give you warranty that it will succeed. If it breaks down, we can't afford to guide you, so you will anyway need to start from scratch or try to fix this way. You will also lose certain functions that are present in 4.4.y but not yet in modern one.

 

I didn't know it is available now. I was so excited to try out. As you warned, I did the upgrade and bricked it. I couldn't get it boot up and ended up with restored from my backup back to 4.4 ;-(

 

Anyone success upgrading from 4.4 to 5.1 without reinstalling things?

 

My setup is the boot on SD with root on usb HDD, if that matters.