Jerry Jyrer Posted November 19, 2018 Posted November 19, 2018 Board: Nanopi M4 with 4GB OS: Armbian_5.65_Nanopim4_Debian_stretch_default_4.4.162 Boot from SD, rootfs on an external HDD 8TB (WD MyBook). I attached another external HDD (Toshiba 2TB). I have been trying rsync -avh from the Toshiba one to the main MyBook one. It ran for under a minute before reporting files were vanished and exited with errors. It appeared the / is gone. nothing return from ls. Hard-reboot, fsck ran, everything's back to normal. I've been trying rsync with nice -n 20 as well as --bwlimit 10, 1MB. Nothing helped. stress test the board for 10 minutes, (stress -c 4 -m 4 -d 4 -i 4). All looks good. Any help's appreciated. Thanks, Jerry
guidol Posted November 19, 2018 Posted November 19, 2018 3 hours ago, Jerry Jyrer said: Boot from SD, rootfs on an external HDD 8TB (WD MyBook). I attached another external HDD (Toshiba 2TB). I have been trying rsync -avh from the Toshiba one to the main MyBook one. It ran for under a minute before reporting files were vanished and exited with errors. It appeared the / is gone. nothing return from ls. Hard-reboot, fsck ran, everything's back to normal. Any help's appreciated. Please provide the complete rsync command (with directorynames). Are you sure that you didnt overwrite something important with the rsync command on your root-device. Why do you want to rsync from the Toshiba to the WD MyBook root-device? BTW: I also got a "old" WD MyBook device, but wouldnt use it as a root device, because this "thing" goes very oft in sleep/ide-state and spinning down.
Jerry Jyrer Posted November 19, 2018 Author Posted November 19, 2018 1 hour ago, guidol said: Please provide the complete rsync command (with directorynames). Are you sure that you didnt overwrite something important with the rsync command on your root-device. Why do you want to rsync from the Toshiba to the WD MyBook root-device? BTW: I also got a "old" WD MyBook device, but wouldnt use it as a root device, because this "thing" goes very oft in sleep/ide-state and spinning down. $ sudo mount /dev/sdb1 /mnt $ sudo mkdir -p /MyBackup $ sudo rsync -avh --progress /mnt/ /MyBackup/ Well as for the reason, (1) testing and (2) mainly, 3-2-1 backup. Another observation, $ sudo smartctl -x /dev/sda1 showing: SMART support is: Unavailable - device lacks SMART capability. pretty weird. I'm running $armbianmonitor -c "$HOME" to test my new 8TB MyBook now. It will take awhile but so far so good. I believe it's finishing writing >1TB; 7TB to go.
guidol Posted November 19, 2018 Posted November 19, 2018 1 hour ago, Jerry Jyrer said: $ sudo mount /dev/sdb1 /mnt $ sudo mkdir -p /MyBackup $ sudo rsync -avh --progress /mnt/ /MyBackup/ the commands does look OK for me (referencing to https://www.howtogeek.com/135533/how-to-use-rsync-to-backup-your-data-on-linux/ ) just a shot in the blue - try to mount the Toshiba not directly as /mnt but as /mnt/toshiba How are the drives connected to the NanoPi M4? USB 3.0 or SATA? Which filesystem do you use (MyBook I think ext4 - but is the Toshiba also ext4 or something like NTFS)? Testing the MyBook is a very good idea
martinayotte Posted November 19, 2018 Posted November 19, 2018 6 hours ago, Jerry Jyrer said: $ sudo rsync -avh --progress /mnt/ /MyBackup/ If you are doing backup of currently running rootfs, be careful not to backup some virtual folder such as /dev, /proc, /run, /sys and /tmp, neither /lost+found ...
Jerry Jyrer Posted November 19, 2018 Author Posted November 19, 2018 45 minutes ago, martinayotte said: If you are doing backup of currently running rootfs, be careful not to backup some virtual folder such as /dev, /proc, /run, /sys and /tmp, neither /lost+found ... Thank you. Yes I'll be aware. Currently, all is pure data. 5 hours ago, guidol said: just a shot in the blue - try to mount the Toshiba not directly as /mnt but as /mnt/toshiba How are the drives connected to the NanoPi M4? USB 3.0 or SATA? Which filesystem do you use (MyBook I think ext4 - but is the Toshiba also ext4 or something like NTFS)? Testing the MyBook is a very good idea Yes, the media test is still on going. I'll test out /mnt/toshiba mount point once the test finished. My book is ext4. Toshiba is NTFS. Both of them are connected to USB 3.0 ports. There 4 of them on NanoPi M4. I tried all of them and got the result.
Jerry Jyrer Posted November 27, 2018 Author Posted November 27, 2018 Just back from the holiday. I tested the media. Both hard drives are healthy. I tested to mount /mnt/toshiba. Same problem. rsync just quited and after that no files seem to be mounted. A hard restart (pulling of power) was required. Here is $sudo armbianmonitor -u Just after the restarted with the Toshiba unattached. http://ix.io/1uz5 Any help would be appreciated. Thanks, Jerry
Jerry Jyrer Posted November 29, 2018 Author Posted November 29, 2018 I'll dig a bit more but so far here is what looks like to be the problem: [ 1008.854862] xhci-hcd xhci-hcd.6.auto: xHCI host not responding to stop endpoint command. [ 1008.854871] xhci-hcd xhci-hcd.6.auto: Assuming host is dying, halting host. [ 1008.854943] xhci-hcd xhci-hcd.6.auto: xHCI host not responding to stop endpoint command. [ 1008.854958] xhci-hcd xhci-hcd.6.auto: Assuming host is dying, halting host. [ 1008.868355] xhci-hcd xhci-hcd.6.auto: HC died; cleaning up [ 1008.868414] xhci-hcd xhci-hcd.6.auto: HC died; cleaning up [ 1008.868420] usb 3-1: USB disconnect, device number 2 [ 1008.868928] usb 4-1: USB disconnect, device number 2 [ 1008.868934] usb 4-1.3: USB disconnect, device number 3 [ 1008.875798] sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x01 driverbyte=0x00 [ 1008.875808] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x8a 8a 00 00 00 00 00 00 c5 88 00 00 00 00 f0 00 00 [ 1008.875812] blk_update_request: I/O error, dev sda, sector 12945408 [ 1008.875898] sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x01 driverbyte=0x00 [ 1008.875903] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x8a 8a 00 00 00 00 00 00 c5 88 f0 00 00 00 f0 00 00 [ 1008.875907] blk_update_request: I/O error, dev sda, sector 12945648 [ 1008.883757] EXT4-fs warning (device sda1): ext4_end_bio:330: I/O error -5 writing to inode 40435820 (offset 0 size 53248 starting block 323676765) [ 1008.883767] Buffer I/O error on device sda1, logical block 323676496 [ 1008.883774] Buffer I/O error on device sda1, logical block 323676497 [ 1008.883778] Buffer I/O error on device sda1, logical block 323676498 [ 1008.883783] Buffer I/O error on device sda1, logical block 323676499 [ 1008.883786] Buffer I/O error on device sda1, logical block 323676500 [ 1008.883790] Buffer I/O error on device sda1, logical block 323676501 [ 1008.883793] Buffer I/O error on device sda1, logical block 323676502 [ 1008.883796] Buffer I/O error on device sda1, logical block 323676503 [ 1008.883800] Buffer I/O error on device sda1, logical block 323676504 [ 1008.883803] Buffer I/O error on device sda1, logical block 323676505 [ 1008.883852] EXT4-fs warning (device sda1): ext4_end_bio:330: I/O error -5 writing to inode 40438764 (offset 0 size 413696 starting block 323676866) [ 1008.883975] EXT4-fs warning (device sda1): ext4_end_bio:330: I/O error -5 writing to inode 40501734 (offset 16777216 size 4980736 starting block 1618432) [ 1008.884650] EXT4-fs warning (device sda1): ext4_end_bio:330: I/O error -5 writing to inode 40501734 (offset 16777216 size 4980736 starting block 1618688) [ 1008.885322] EXT4-fs warning (device sda1): ext4_end_bio:330: I/O error -5 writing to inode 40501734 (offset 16777216 size 4980736 starting block 1618944) [ 1008.885887] EXT4-fs warning (device sda1): ext4_end_bio:330: I/O error -5 writing to inode 40501734 (offset 16777216 size 4980736 starting block 1619200) [ 1008.886444] EXT4-fs warning (device sda1): ext4_end_bio:330: I/O error -5 writing to inode 40501734 (offset 16777216 size 4980736 starting block 1619392) [ 1008.886944] EXT4-fs warning (device sda1): ext4_end_bio:330: I/O error -5 writing to inode 40501734 (offset 21757952 size 1572864 starting block 1619648) [ 1008.887502] EXT4-fs warning (device sda1): ext4_end_bio:330: I/O error -5 writing to inode 40501734 (offset 21757952 size 1572864 starting block 1619776) [ 1008.887797] EXT4-fs warning (device sda1): ext4_end_bio:330: I/O error -5 writing to inode 40501734 (offset 23330816 size 1310720 starting block 1620032) [ 1008.890389] JBD2: Detected IO errors while flushing file data on sda1-8 [ 1008.890521] Aborting journal on device sda1-8. [ 1008.890657] JBD2: Error -5 detected when updating journal superblock for sda1-8. [ 1008.890829] JBD2: Detected IO errors while flushing file data on sda1-8 [ 1008.901805] sd 1:0:0:0: [sdb] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=0x00 [ 1008.901819] sd 1:0:0:0: [sdb] tag#0 CDB: opcode=0x28 28 00 39 1d dd f0 00 00 f0 00 [ 1008.901825] blk_update_request: I/O error, dev sdb, sector 958258672 [ 1008.922339] usb 4-1.4: USB disconnect, device number 4 [ 1008.925822] sd 1:0:0:0: [sdb] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x01 driverbyte=0x00 [ 1008.925841] sd 1:0:0:0: [sdb] tag#0 CDB: opcode=0x28 28 00 39 1d de e0 00 00 10 00 [ 1008.925848] blk_update_request: I/O error, dev sdb, sector 958258912 [ 1008.925987] sd 1:0:0:0: [sdb] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x01 driverbyte=0x00 [ 1008.925995] sd 1:0:0:0: [sdb] tag#0 CDB: opcode=0x28 28 00 39 1d de f0 00 00 f0 00 [ 1008.926001] blk_update_request: I/O error, dev sdb, sector 958258928 [ 1008.927276] EXT4-fs error (device sda1): ext4_journal_check_start:56: Detected aborted journal [ 1008.928055] EXT4-fs (sda1): Remounting filesystem read-only [ 1008.928544] bdi-block not registered
Jim MacKenzie Posted February 24, 2019 Posted February 24, 2019 I may be having a similar problem. I am using a NanoPi Neo4 and my root filesystem is on eMMC, but I have a large RAID6 attached by USB, on which I back up my server (this box lives at a family member's home, offsite). The RAID used to live on a SheevaPlug, but I've replaced it with the NanoPi because of performance issues. (Got a good decade out of the Sheeva!) The RAID devices seem to disappear as soon as heavy I/O occurs. It first occurred when I started an rsync from my server. Rebooting the system became impossible without disconnecting the USB hub because the RAID rebuild would cause the same problem to recur immediately. This was all with a four-port hub connected to the USB 3.0 port. I initially thought it was a power issue, so replaced the four-port unpowered hub with a ten-port powered hub (I don't need ten ports, but I had this unit kicking around). This did nothing to cure the problem. What did solve the problem was moving the hub's connection from the USB 3.0 port to the USB 2.0 port. I'm not sure if it's significant, but I notice in /boot/armbianEnv.txt: usbstoragequirks=0x2537:0x1066:u,0x2537:0x1068:u,0x0bc2:0x331a:u,0x0bc2:0x3322:u And my hard disks are in that quirks "blacklist": root@nobby:/boot# lsusb | grep Seagate Bus 005 Device 009: ID 0bc2:3322 Seagate RSS LLC Bus 005 Device 008: ID 0bc2:331a Seagate RSS LLC Bus 005 Device 007: ID 0bc2:3322 Seagate RSS LLC Bus 005 Device 006: ID 0bc2:3322 Seagate RSS LLC Jim
Igor Posted February 25, 2019 Posted February 25, 2019 13 hours ago, Jim MacKenzie said: I'm not sure if it's significant, but I notice in /boot/armbianEnv.txt: usbstoragequirks=0x2537:0x1066:u,0x2537:0x1068:u,0x0bc2:0x331a:u,0x0bc2:0x3322:u This is because of:http://linux-sunxi.org/USB/UAS
Jerry Jyrer Posted February 25, 2019 Author Posted February 25, 2019 On 2/24/2019 at 7:19 AM, Jim MacKenzie said: What did solve the problem was moving the hub's connection from the USB 3.0 port to the USB 2.0 port. Yes, exactly what I finally solved my problem. Same to you Jim, I initially thought and be advised by many that it was the power problem but at the end what solved was to connect one with an USB 2.0 port.
Jim MacKenzie Posted February 25, 2019 Posted February 25, 2019 1 hour ago, Jerry Jyrer said: Yes, exactly what I finally solved my problem. Same to you Jim, I initially thought and be advised by many that it was the power problem but at the end what solved was to connect one with an USB 2.0 port. So then the question is: is the USB 3.0 hardware bad on this device, or is it a driver issue and it can be corrected? I've certainly found USB 3.0 to be a delicate thing compared to USB 2.0. I've had many, many difficulties with 3.0 (all surmounted over time, thankfully, save this one). The only issue I ever had with 2.0 and Linux was a using-the-port-at-1.1-speeds problem; everything still worked. If I can do anything to help give diagnostic information to developers, please let me know, but it sounds like this problem is easily reproducible.
Jerry Jyrer Posted March 18, 2019 Author Posted March 18, 2019 On 2/25/2019 at 11:14 AM, Jim MacKenzie said: If I can do anything to help give diagnostic information to developers, please let me know, but it sounds like this problem is easily reproducible. Totally agreed. I posted here and similar topics for the same issue but so far no real solutions. Many suggestions were around under voltage being supplied. I personally doubt if that is the root cause. Armbian software/driver itself? Not too sure I tried the stock one from FriendlyElec, the result is the same. Is it the upstream Linux? Possible but no verification. I have just tried after recent apt update and upgrade. The same problem is still reproducible.
dkking Posted March 19, 2019 Posted March 19, 2019 is it the same issue faced by users of rock64? ayufan Rock64 and rock64 Armbian
Jerry Jyrer Posted March 24, 2019 Author Posted March 24, 2019 On 3/19/2019 at 8:45 AM, dkking said: is it the same issue faced by users of rock64? ayufan Rock64 and rock64 Armbian Quick scan, it looks like the same. Solution / root cause?
dkking Posted March 24, 2019 Posted March 24, 2019 1 hour ago, Jerry Jyrer said: Quick scan, it looks like the same. Solution / root cause? I have no clue but if the problem is shared across devices of Rockchip, could it be a Rockchip, chipset /usb host controller issue? something Rockchip has to fix? I have heard the Odroid XU4 has no such issues (not Rockchip CPU, Exynos) or the 'up board'(Intel) this is after you rule out the adequate power question(which I believe is not the problem)
Mathias Posted May 16, 2019 Posted May 16, 2019 Jumping in a little late, but on the RockPro64 (also Rk3399), the same kind of problems happen (also very easily reproducible, it happens every single time under load). See my post on Pine64's forums. I am in contact with them (well, so far they just keep bouncing me around to another forum / email / person...) to try learning from them if they ever managed to get some sort of usb3 working on this board. Other people are also wondering if the hardware is the root cause of these issues...
Mathias Posted June 13, 2019 Posted June 13, 2019 (edited) Replying even later... This definitely has to do with USB suspend (see this link): on a freshly booted system, no problems at all. On a system that has been running for a day (ie the USB had enough time to suspend), the problem is back with tons of error messages... [edit] The autosuspend is not the culprit... This has to do with some underpower, see this (long) thread. Sorry for the fake news and the disappointment! Edited June 19, 2019 by Mathias More information after more testing
Jerry Jyrer Posted December 5, 2020 Author Posted December 5, 2020 On 7/5/2020 at 8:16 AM, Jim MacKenzie said: Not solvable, then? Somehow as of today, I tried again and it seems the problem is no longer seen. At least so far, rsync between two hard drives about 30 hrs is working fine. ``` $ cat /etc/armbian-release # PLEASE DO NOT EDIT THIS FILE BOARD=nanopim4 BOARD_NAME="NanoPi M4" BOARDFAMILY=rk3399 BUILD_REPOSITORY_URL=https://github.com/armbian/build BUILD_REPOSITORY_COMMIT=b0760915-dirty DISTRIBUTION_CODENAME=bionic DISTRIBUTION_STATUS=supported VERSION=20.11 LINUXFAMILY=rk3399 BRANCH=legacy ARCH=arm64 IMAGE_TYPE=stable BOARD_TYPE=conf INITRD_ARCH=arm64 KERNEL_IMAGE_TYPE=Image ```
Jim MacKenzie Posted February 15, 2021 Posted February 15, 2021 On 12/5/2020 at 3:40 AM, Jerry Jyrer said: Somehow as of today, I tried again and it seems the problem is no longer seen. At least so far, rsync between two hard drives about 30 hrs is working fine. Looks like you're using the Ubuntu Armbian, not the Debian - do you mind telling me what kernel you're using? (A pity I only saw this now.)
Jerry Jyrer Posted February 15, 2021 Author Posted February 15, 2021 57 minutes ago, Jim MacKenzie said: Looks like you're using the Ubuntu Armbian, not the Debian - do you mind telling me what kernel you're using? (A pity I only saw this now.) Right, between the original post and now, I had been switching and trying different Armbian Ubuntu, Debian with Buster, and other variations. None of them worked at the time. I stopped at some point and the workaround was to connect a usb 2.0 hub as a middle man. Recently, I found myself need to do a full local rsync between hard drives again and it works. I'm currently with Armbian 21.02.1 Bionic with Linux 4.4.213-rk3399
Igor Posted February 15, 2021 Posted February 15, 2021 There are only two different hardware layers. Stock kernel (4.4.x) and mainline (5.10.y) based. Debian or Ubuntu is just application lawyer which doesn't have any effects on board hardware. Check modern kernel - it should be much more stable. 1
Jim MacKenzie Posted February 15, 2021 Posted February 15, 2021 3 hours ago, Igor said: There are only two different hardware layers. Stock kernel (4.4.x) and mainline (5.10.y) based. Debian or Ubuntu is just application lawyer which doesn't have any effects on board hardware. Check modern kernel - it should be much more stable. Thanks, Igor. What is the simplest way to shift to mainline? (Preferably without reinstalling.) i.e. Which linux-image package is the correct one for this platform? 1
Igor Posted February 15, 2021 Posted February 15, 2021 armbian-config -> system -> alternative kernel -> rockchip64-current v21.02.2 ... but I can't give you warranty that it will succeed. If it breaks down, we can't afford to guide you, so you will anyway need to start from scratch or try to fix this way. You will also lose certain functions that are present in 4.4.y but not yet in modern one.
Jerry Jyrer Posted February 17, 2021 Author Posted February 17, 2021 On 2/15/2021 at 2:07 PM, Igor said: armbian-config -> system -> alternative kernel -> rockchip64-current v21.02.2 ... but I can't give you warranty that it will succeed. If it breaks down, we can't afford to guide you, so you will anyway need to start from scratch or try to fix this way. You will also lose certain functions that are present in 4.4.y but not yet in modern one. I didn't know it is available now. I was so excited to try out. As you warned, I did the upgrade and bricked it. I couldn't get it boot up and ended up with restored from my backup back to 4.4 ;-( Anyone success upgrading from 4.4 to 5.1 without reinstalling things? My setup is the boot on SD with root on usb HDD, if that matters.
Recommended Posts