Orange Pi Prime: USB problem under heavy I/O (and CPU) load

[grep]

Just for information (i know there is no support): under heavy load i have such error (for 3 times on different re-runs)

 

Spoiler

[76957.139777] monerod[1798]: undefined instruction: pc=000000000054a3e0

[76957.139788] Code: 00942f40 00000000 00000402 00001c60 (00000000) 

[76988.126797] sd 0:0:0:0: [sda] tag#1 uas_eh_abort_handler 0 uas-tag 2 inflight: IN 

[76988.126811] sd 0:0:0:0: [sda] tag#1 CDB: opcode=0x28 28 00 03 00 08 18 00 00 08 00

[76988.130108] scsi host0: uas_eh_device_reset_handler start

[76988.254794] usb 4-1: reset high-speed USB device number 2 using ehci-platform

[76988.411953] scsi host0: uas_eh_device_reset_handler success

[81988.438809] monerod[9811]: unhandled level 0 translation fault (11) at 0xfffffffffffffffe, esr 0x92000004, in libc-2.23.so[ffff9ac8c000+12e000]

[81988.438835] CPU: 0 PID: 9811 Comm: monerod Tainted: G         C      4.14.4-sunxi64 #11

[81988.438837] Hardware name: Xunlong Orange Pi Prime (DT)

[81988.438841] task: ffff80007411c380 task.stack: ffff000010718000

[81988.438847] PC is at 0xffff9acfe9a8

[81988.438850] LR is at 0xffff9b325bc4

[81988.438853] pc : [<0000ffff9acfe9a8>] lr : [<0000ffff9b325bc4>] pstate: 80000000

[81988.438854] sp : 0000ffffe529ebc0

[81988.438857] x29: 0000ffffe529ebc0 x28: 0000000000000000 

[81988.438862] x27: 00000000033b0fa0 x26: 00000000033b0fc0 

[81988.438866] x25: 00000000033af000 x24: 0000000000000001 

[81988.438871] x23: 0000ffffe529ee08 x22: 000000000054ba70 

[81988.438875] x21: 0000000000000000 x20: 0000000000000000 

[81988.438880] x19: 0000ffffe529ec58 x18: 0000000000000a03 

[81988.438884] x17: 0000ffff9b33b110 x16: 0000ffff9acfe988 

[81988.438889] x15: 0000ffff9b5dc000 x14: 0000000000000000 

[81988.438894] x13: 6436336136666334 x12: 6135356266363861 

[81988.438898] x11: 0000000000000030 x10: 0101010101010101 

[81988.438903] x9 : 000000000054c180 x8 : 0000000000943538 

[81988.438907] x7 : 0000000000000000 x6 : 000000000094354c 

[81988.438912] x5 : 0000000000000000 x4 : 0000000000000001 

[81988.438916] x3 : 0000000000000000 x2 : 0000000000000000 

[81988.438920] x1 : fffffffffffffff6 x0 : 0000000000000006 

 

[Igor]

If you have internet access please provide the output of: armbianmonitor -u

[grep]

@Igor i've made a reboot and after that board became unaccessible via ssh (while there's still icmp-connectivity ) it's weird a bit. Need to check and fix.

 

 

 

[grep]

@Igor http://sprunge.us/NJFS

Looks like my USB-HDD fs is corrupted, but when i connected device with this fs to "single" usb port on board i could not see the device at all  Is it normal ?

 

[guidol]

3 minutes ago, grep said:

Looks like my USB-HDD fs is corrupted, but when i connected device with this fs to "single" usb port on board i could not see the device at all  Is it normal ?

 

Did you power the drive from the Orange Pi  Prime via USB or has it an external power supply?

If from USB then the power supply hasnt enough power for booth - because on heavy load the Prime need more power.

[grep]

@guidol Looks like you're right: power led was not "bright red", but "muted red", i've replaced USB 3.0 cable (i use external case with 3.0 interface) with USB 2.0 dual-plug cable hoping that "theoretically" (it's just an assumption) it could use power from both USB-lines attached. And it is working now.

e2fsck -f did not gave any errors.

So, do you think "2-plug USB cable" will help me, or there is a "common power circuit" for all USB on this board and i need external power supply to use for USB-HDD ?

[grep]

Same symptoms after reboot (icmp-availability with unability to boot from HDD). In fact this is strange: there was no such behavior before last crash.

And it was ok, after i connected it to BOOTED system.

Probably: something wrong had happended with board.

 

But looks like first of all i need to replace PSU to overcome "not enough power on boot" possible problem. I have one more (not usual "black cube with plug", but this one https://ru.aliexpress.com/item/Mini-switching-power-supply-15W-5V-3A-Single-Output-dc-power-supply-for-Led-Strip-AC110V/32800882848.html) but its TOO hot (really ~70-80 celcius deg. under load).

Can anyone help to find reliable & proven & not too hot 5V/3A PSU ?

[chwe]

Why not going for the 'official one' which can be bought in xunglongs online store?

[guidol]

7 hours ago, chwe said:

Why not going for the 'official one' which can be bought in xunglongs online store?

Yes - this power supply is fine. I did buy it at a local store and isnt getting real warm:
https://www.robotistan.com/orange-pi-plus-2-guc-adaptoru

[grep]

@guidol @chwe

 Thanks. Ordered one at AliExpress, but need to wait for delivery.

[guidol]

10 hours ago, grep said:

So, do you think "2-plug USB cable" will help me, or there is a "common power circuit" for all USB on this board and i need external power supply to use for USB-HDD ?

i dont think it would help (maybe on a Raspberry where one USB Port has a power limit).

If you got the right power-supply it can work (for a 2.5" HDD and not a 3.5" HDD) - but I would be much better if the HDD has its own power supply.

With the 2 plug-cable you could try to plug the data/power connector to the Prime and the second power only to a additional power supply.

But the best will be a seperate power supply for the drive and a normal one plug cable.

[grep]

And once again (now i used BTRFS, armbianmonitor -u = http://sprunge.us/agje ):

Spoiler

[ 6952.597076] BTRFS warning (device sda1): csum failed root 5 ino 33666 off 108867584 csum 0x0fed3478 expected csum 0x6c67fb8c mirror 1

[ 6952.621597] BTRFS warning (device sda1): csum failed root 5 ino 33666 off 109916160 csum 0x848872d2 expected csum 0x1f9de071 mirror 1

[ 6952.645809] BTRFS warning (device sda1): csum failed root 5 ino 33666 off 110964736 csum 0x083e7e00 expected csum 0xc9b10f86 mirror 1

[ 6952.646792] BTRFS warning (device sda1): csum failed root 5 ino 23358 off 532480 csum 0x69584541 expected csum 0x9c398cf5 mirror 1

[ 6952.646885] BTRFS warning (device sda1): csum failed root 5 ino 15937 off 148037632 csum 0x2e705571 expected csum 0xc8560bb4 mirror 1

[ 6952.653387] BTRFS warning (device sda1): csum failed root 5 ino 19910 off 380928 csum 0xf2daacac expected csum 0x010ee954 mirror 1

[ 6952.653394] BTRFS warning (device sda1): csum failed root 5 ino 23358 off 532480 csum 0x69584541 expected csum 0x9c398cf5 mirror 1

[ 6952.658312] BTRFS warning (device sda1): csum failed root 5 ino 15937 off 148037632 csum 0x2e705571 expected csum 0xc8560bb4 mirror 1

[ 6952.658405] monerod[5376]: unhandled level 3 translation fault (7) at 0xffff6cd2e00a, esr 0x92000007, in monerod[400000+531000]

[ 6952.658439] CPU: 0 PID: 5376 Comm: monerod Tainted: G         C      4.14.4-sunxi64 #11

[ 6952.658442] Hardware name: Xunlong Orange Pi Prime (DT)

[ 6952.658445] task: ffff800001b1ec00 task.stack: ffff00000e518000

[ 6952.658453] PC is at 0x77b350

[ 6952.658455] LR is at 0x77b30c

[ 6952.658458] pc : [<000000000077b350>] lr : [<000000000077b30c>] pstate: 80000000

[ 6952.658460] sp : 0000ffffaa1fb270

[ 6952.658463] x29: 0000ffffaa1fb270 x28: 000000000000db6a 

[ 6952.658468] x27: 0000ffffaa1fb5c8 x26: 000000002dc1de80 

[ 6952.658472] x25: 00000000033af000 x24: 0000ffffaa1fb6b8 

[ 6952.658477] x23: 0000000000000000 x22: 0000000000000000 

[ 6952.658482] x21: 0000000000000000 x20: 0000000000000006 

[ 6952.658486] x19: 0000ffff3c7cd378 x18: 0000000000000000 

[ 6952.658491] x17: 0000ffffbec26408 x16: 0000000000942fa0 

[ 6952.658496] x15: 000037fc2683d2a4 x14: 003277699e200b08 

[ 6952.658501] x13: ffffffffffff0000 x12: 0000000000000000 

[ 6952.658506] x11: 0000000000000010 x10: 0000000000001b28 

[ 6952.658510] x9 : 0000ffff3cfdf630 x8 : 0101010101010101 

[ 6952.658515] x7 : 0000ffffb9b13010 x6 : 0000000000000001 

[ 6952.658519] x5 : 000000000000019c x4 : 0000000000000000 

[ 6952.658524] x3 : 0000000000000003 x2 : 0000ffff6cd2e000 

[ 6952.658528] x1 : 0000ffff3c7cd37c x0 : 0000ffff3c7cd388 

[ 6952.659552] BTRFS warning (device sda1): csum failed root 5 ino 19910 off 380928 csum 0xf2daacac expected csum 0x010ee954 mirror 1

[ 6952.663692] BTRFS warning (device sda1): csum failed root 5 ino 23358 off 532480 csum 0xc9eefb90 expected csum 0x9c398cf5 mirror 1

[ 6952.696845] BTRFS error (device sda1): bad tree block start 0 480018432

[ 6952.705417] BTRFS error (device sda1): bad tree block start 0 480018432

[ 6964.460183] BTRFS error (device sda1): bad tree block start 0 533659648

[ 6964.475414] BTRFS error (device sda1): bad tree block start 8942583155399852085 817430528

[ 6964.500910] BTRFS error (device sda1): bad tree block start 0 533659648

[ 6964.860928] BTRFS error (device sda1): bad tree block start 8942583155399852085 817430528

[ 6964.869918] BTRFS error (device sda1): bad tree block start 3833842346553536713 764493824

[ 6964.878552] BTRFS error (device sda1): bad tree block start 8942583155399852085 817430528

[ 6964.887427] BTRFS error (device sda1): bad tree block start 3833842346553536713 764493824

[ 6975.905343] BTRFS error (device sda1): bad tree block start 8942583155399852085 817430528

[ 7006.454298] BTRFS error (device sda1): bad tree block start 518144854589440 787824640

[ 7006.454657] sd 0:0:0:0: [sda] Unaligned partial completion (resid=988, sector_sz=512)

[ 7006.455004] sd 0:0:0:0: [sda] tag#4 data cmplt err -75 uas-tag 5 inflight: 

[ 7006.455013] sd 0:0:0:0: [sda] tag#4 CDB: opcode=0x12 12 00 00 00 24 00

[ 7006.455419] BTRFS error (device sda1): bad tree block start 0 480018432

[ 7006.456535] BTRFS info (device sda1): read error corrected: ino 0 off 480018432 (dev /dev/sda1 sector 953920)

[ 7006.457024] BTRFS info (device sda1): read error corrected: ino 0 off 480022528 (dev /dev/sda1 sector 953928)

[ 7006.457509] BTRFS info (device sda1): read error corrected: ino 0 off 480026624 (dev /dev/sda1 sector 953936)

[ 7006.457919] BTRFS info (device sda1): read error corrected: ino 0 off 480030720 (dev /dev/sda1 sector 953944)

[ 7006.474931] BTRFS info (device sda1): read error corrected: ino 0 off 787824640 (dev /dev/sda1 sector 1555104)

[ 7006.475529] BTRFS info (device sda1): read error corrected: ino 0 off 787828736 (dev /dev/sda1 sector 1555112)

[ 7006.476089] BTRFS info (device sda1): read error corrected: ino 0 off 787832832 (dev /dev/sda1 sector 1555120)

[ 7006.476655] BTRFS info (device sda1): read error corrected: ino 0 off 787836928 (dev /dev/sda1 sector 1555128)

 

 

PSU is this one:  https://ru.aliexpress.com/item/Mini-switching-power-supply-15W-5V-3A-Single-Output-dc-power-supply-for-Led-Strip-AC110V/32800882848.html 

The new one did not arrived yet.

But "something" makes me think that problem is not related to PSU.

 

 

[zador.blood.stained]

In the past "translation fault" errors usually indicated CPU underpowering, so IMO the best course of action would be a stress&stability test with cpuburn and xhpl, without connecting the HDD first and with the HDD conected after that. Ideally this test should be done for each available OPP (by setting the userspace governor).

[grep]

@zador.blood.stained

 

I suppose simultaneous run a lot of IO/CPU-intensive tasks (monerod & bitcoind threads) is like a stress-test. But, there is no stability (latest armbianmonitor -u = http://sprunge.us/hUZM ).

And line "reset high-speed USB device number 2 using ehci-platform" becomes a bit annoying. What causes "USB device reset" ? Is underpower the only reason ? Or "most likely reason because we haven't seen things like this before ?". I ordered "reference/original" PSU (it did not arrived yet) but i still think the problem is not with PSU (i used 2 different and got the same error).

Looks like next step is going to be "using external power for usb hdd" as @guidol  adviced, and, if error will happen once again,  it will reject all args for assumption "cpu underpower because of usb device".

 

Spoiler

[25591.714792] sd 0:0:0:0: [sda] tag#5 uas_eh_abort_handler 0 uas-tag 6 inflight: IN 

[25591.714806] sd 0:0:0:0: [sda] tag#5 CDB: opcode=0x28 28 00 03 81 0f 78 00 00 08 00

[25591.718092] scsi host0: uas_eh_device_reset_handler start

[25591.842788] usb 3-1: reset high-speed USB device number 2 using ehci-platform

[25592.051961] scsi host0: uas_eh_device_reset_handler success

[41806.259711] INFO: rcu_sched self-detected stall on CPU

[41806.264872] 1-...: (5249 ticks this GP) idle=c02/140000000000001/0 softirq=5263189/5263189 fqs=2493 

[41806.267704] INFO: rcu_sched detected stalls on CPUs/tasks:

[41806.267712] 1-...: (5249 ticks this GP) idle=c02/140000000000001/0 softirq=5263189/5263189 fqs=2493 

[41806.267713] (detected by 0, t=5252 jiffies, g=2656881, c=2656880, q=6069)

[41806.267720] Task dump for CPU 1:

[41806.267723] monerod         R  running task        0  9071      1 0x0000000b

[41806.267729] Call trace:

[41806.267743] [<ffff000008084e48>] __switch_to+0x98/0xb0

[41806.267751] [<ffff00000839e5dc>] btrfs_buffer_uptodate+0x54/0x78

[41806.267755] [<0000000000000019>] 0x19

[41806.295623]  (t=5258 jiffies g=2656881 c=2656880 q=6070)

[41806.301018] Task dump for CPU 1:

[41806.301020] monerod         R  running task        0  9071      1 0x0000000b

[41806.301025] Call trace:

[41806.301032] [<ffff0000080884d8>] dump_backtrace+0x0/0x3a8

[41806.301036] [<ffff000008088894>] show_stack+0x14/0x20

[41806.301043] [<ffff0000080d6b60>] sched_show_task+0x170/0x198

[41806.301047] [<ffff0000080d7b58>] dump_cpu_task+0x40/0x50

[41806.301053] [<ffff0000081085ac>] rcu_dump_cpu_stacks+0x94/0xd8

[41806.301057] [<ffff000008107b5c>] rcu_check_callbacks+0x564/0x788

[41806.301062] [<ffff00000810b6e4>] update_process_times+0x2c/0x58

[41806.301070] [<ffff00000811b2c0>] tick_sched_handle.isra.5+0x30/0x48

[41806.301074] [<ffff00000811b318>] tick_sched_timer+0x40/0x90

[41806.301079] [<ffff00000810c2f4>] __hrtimer_run_queues+0xec/0x168

[41806.301083] [<ffff00000810c570>] hrtimer_interrupt+0xa0/0x228

[41806.301091] [<ffff0000087d3d48>] arch_timer_handler_phys+0x28/0x40

[41806.301096] [<ffff0000080fbe18>] handle_percpu_devid_irq+0x80/0x138

[41806.301101] [<ffff0000080f623c>] generic_handle_irq+0x24/0x38

[41806.301106] [<ffff0000080f68ec>] __handle_domain_irq+0x5c/0xb0

[41806.301110] [<ffff000008080df8>] gic_handle_irq+0x58/0xa8

[41806.301113] Exception stack(0xffff000010c737a0 to 0xffff000010c738e0)

[41806.301118] 37a0: ffff80005c9b8bc0 000000008770a000 00000000876ce000 ffff800074bea000

[41806.301123] 37c0: 0000000000000010 ffff000010c73c00 ffff000010c73a8f 0000000000000000

[41806.301127] 37e0: ffff80005f392358 0000000000000000 0000000035d24000 0000000000004000

[41806.301131] 3800: 3213d1102470ca00 0321e1000000005a 0000005a3213d12b aef692e44fad9ecd

[41806.301136] 3820: 5e55bbfaaa4008d8 0e1b0135dfabdb3b 0000000000000000 00000000a8358000

[41806.301140] 3840: 0000000000001000 ffff80005c9b8b60 ffff8000557702d0 ffff000010c73c00

[41806.301145] 3860: ffff000010c73970 ffff800002cd8d10 ffff800002cd8ee0 ffff80000225e000

[41806.301149] 3880: 0000000000000000 ffff000010c738e0 ffff0000083e9934 ffff000010c738e0

[41806.301153] 38a0: ffff0000083e93f4 0000000080000145 ffff000010c738e0 ffff000008172994

[41806.301157] 38c0: 0001000000000000 0000000000000000 ffff000010c738e0 ffff0000083e93f4

[41806.301161] [<ffff0000080828f0>] el1_irq+0xb0/0x140

[41806.301169] [<ffff0000083e93f4>] wait_ordered_extents.isra.2.constprop.6+0xa4/0x3a0

[41806.301174] [<ffff0000083e9934>] btrfs_log_changed_extents+0x244/0x5c0

[41806.301179] [<ffff0000083ea4d8>] btrfs_log_inode+0x828/0xe20

[41806.301185] [<ffff0000083eada4>] btrfs_log_inode_parent+0x22c/0x928

[41806.301188] [<ffff0000083ec35c>] btrfs_log_dentry_safe+0x64/0x90

[41806.301194] [<ffff0000083bd280>] btrfs_sync_file+0x260/0x380

[41806.301201] [<ffff000008224770>] vfs_fsync_range+0x48/0xc0

[41806.301205] [<ffff000008224840>] do_fsync+0x38/0x78

[41806.301209] [<ffff000008224b28>] SyS_fdatasync+0x10/0x20

[41806.301212] Exception stack(0xffff000010c73ec0 to 0xffff000010c74000)

[41806.301216] 3ec0: 0000000000000011 0000000000000002 0000ffff84d535ac 0000ffff84d542e8

[41806.301220] 3ee0: 0000000000000002 0000ffff84d541e0 0000ffff84d548d0 0000000000000000

[41806.301225] 3f00: 0000000000000053 0000ffff300008c0 000000000000a333 0000000000000010

[41806.301229] 3f20: 0000000000000000 ffffffffffff0000 002264e5fffa46c5 000054bf5272a99a

[41806.301233] 3f40: 0000000000000000 0000ffff89295c70 0000000000000000 0000000000000000

[41806.301237] 3f60: 0000000000000000 00000000033af000 0000000038d52a40 0000ffff84d53640

[41806.301241] 3f80: 0000000038c388c8 000000000087e000 0000000000880000 000025ff70067304

[41806.301245] 3fa0: 0000ffff6305cb30 0000ffff84d535f0 0000ffff89295c84 0000ffff84d535b0

[41806.301250] 3fc0: 0000ffff89295c94 0000000080000000 0000000000000011 0000000000000053

[41806.301254] 3fe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000

[41806.301258] [<ffff000008082fb0>] el0_svc_naked+0x24/0x28

[44272.067835] btrfs_print_data_csum_error: 3 callbacks suppressed

[44272.067849] BTRFS warning (device sda1): csum failed root 5 ino 148933 off 630784 csum 0x9e7b37d9 expected csum 0x69ee511f mirror 1

[44272.071240] BTRFS warning (device sda1): csum failed root 5 ino 148933 off 733184 csum 0x8d1370b9 expected csum 0x90b4caf2 mirror 1

[44272.071760] BTRFS warning (device sda1): csum failed root 5 ino 148933 off 901120 csum 0x87012a6e expected csum 0x79a2aa34 mirror 1

[44272.073152] BTRFS warning (device sda1): csum failed root 5 ino 148933 off 946176 csum 0x5a51f5bf expected csum 0x03fd7c36 mirror 1

[44272.073877] BTRFS warning (device sda1): csum failed root 5 ino 148933 off 1015808 csum 0x5875be69 expected csum 0x710f2af6 mirror 1

[44272.077437] BTRFS warning (device sda1): csum failed root 5 ino 148933 off 1052672 csum 0xeadfc254 expected csum 0xffb19f33 mirror 1

[44272.079156] BTRFS warning (device sda1): csum failed root 5 ino 148933 off 1212416 csum 0xc0dc84dd expected csum 0x2b26ffbc mirror 1

[44272.081935] BTRFS warning (device sda1): csum failed root 5 ino 148933 off 1294336 csum 0xd093812d expected csum 0x9fd5ef6f mirror 1

[44272.082155] BTRFS warning (device sda1): csum failed root 5 ino 148933 off 1540096 csum 0x61155f26 expected csum 0xae3459e9 mirror 1

[44272.083394] BTRFS warning (device sda1): csum failed root 5 ino 148933 off 1658880 csum 0xa8a4f5ba expected csum 0xdb469f15 mirror 1

[44272.088908] bitcoin-msghand[6750]: unhandled level 2 translation fault (7) at 0xffff8b3c3a97, esr 0x92000006, in bitcoind[aaaab5738000+52a000]

[44272.088941] CPU: 2 PID: 6750 Comm: bitcoin-msghand Tainted: G         C      4.14.4-sunxi64 #11

[44272.088944] Hardware name: Xunlong Orange Pi Prime (DT)

[44272.088948] task: ffff8000740c0000 task.stack: ffff00000f6b0000

[44272.088955] PC is at 0xaaaab5aa9b54

[44272.088957] LR is at 0xaaaab5aa9a2c

[44272.088960] pc : [<0000aaaab5aa9b54>] lr : [<0000aaaab5aa9a2c>] pstate: 60000000

[44272.088962] sp : 0000ffff73ffa2e0

[44272.088964] x29: 0000ffff73ffa2e0 x28: 0000ffff73ffa410 

[44272.088969] x27: 0000000000001008 x26: 0000aaaab5ae7000 

[44272.088974] x25: 0000ffff73ffa408 x24: 0000ffff60b714e0 

[44272.088979] x23: 0000ffff60b24410 x22: 000000005d7d1528 

[44272.088983] x21: 0000ffff8b3c2a8e x20: 0000aaaab5c7c000 

[44272.088988] x19: 0000ffff73ffa440 x18: 00000000996c87f7 

[44272.088993] x17: 0000ffff9f2763b0 x16: 0000ffff9f5d0088 

[44272.088997] x15: 00000000878b26d0 x14: 00000000b63f8b0c 

[44272.089002] x13: ffffff0000000000 x12: 0000000000000000 

[44272.089007] x11: 000000000000001c x10: 000000000000d1cc 

[44272.089011] x9 : 0000ffff60000020 x8 : 0000ffff73ffa358 

[44272.089016] x7 : 0000000000000000 x6 : 0000ffff9fb25cb8 

[44272.089020] x5 : 0000000000156a9b x4 : 0000000000214e4b 

[44272.089025] x3 : 0000ffff73ffa368 x2 : 0000000000001009 

[44272.089030] x1 : 0000ffff8b3c2a8e x0 : 0000000000001009 

[44279.385651] btrfs_print_data_csum_error: 7 callbacks suppressed

[44279.385673] BTRFS warning (device sda1): csum failed root 5 ino 161155 off 1548288 csum 0x9675b7a4 expected csum 0x62ca83f2 mirror 1

[44279.385799] bitcoind[6739]: unhandled level 3 translation fault (7) at 0xffff8664393a, esr 0x92000007, in bitcoind[aaaab5738000+52a000]

[44279.385918] CPU: 3 PID: 6739 Comm: bitcoind Tainted: G         C      4.14.4-sunxi64 #11

[44279.385920] Hardware name: Xunlong Orange Pi Prime (DT)

[44279.385924] task: ffff800001aba880 task.stack: ffff00000f670000

[44279.385932] PC is at 0xaaaab5aa9b54

[44279.385934] LR is at 0xaaaab5aa9a2c

[44279.385938] pc : [<0000aaaab5aa9b54>] lr : [<0000aaaab5aa9a2c>] pstate: 60000000

[44279.385939] sp : 0000ffff9527d280

[44279.385941] x29: 0000ffff9527d280 x28: 0000ffff9527d3b0 

[44279.385947] x27: 0000000000001009 x26: 0000aaaab5ae7000 

[44279.385952] x25: 0000ffff9527d3a8 x24: 0000ffff9001f0e0 

[44279.385956] x23: 0000ffff900333f0 x22: 000000005d7d1528 

[44279.385961] x21: 0000ffff86642930 x20: 0000aaaab5c7c000 

[44279.385966] x19: 0000ffff9527d3e0 x18: 0000000000000014 

[44279.385970] x17: 0000ffff9f2763b0 x16: 0000ffff9f5d0088 

[44279.385975] x15: 0000503953582f74 x14: 0000000a00000efe 

[44279.385980] x13: ffffffffff000000 x12: 0000000000000000 

[44279.385984] x11: 0000000000000018 x10: 0000000000000efe 

[44279.385989] x9 : 0000ffff90000020 x8 : 0000ffff9527d2f8 

[44279.385994] x7 : 0000000000000000 x6 : 0000ffff9fb25cb8 

[44279.385998] x5 : 000000000017a93e x4 : 000000000021a00e 

[44279.386002] x3 : 0000ffff9527d308 x2 : 000000000000100a 

[44279.386007] x1 : 0000ffff86642930 x0 : 000000000000100a 

[44294.737203] BTRFS warning (device sda1): csum failed root 5 ino 15937 off 1718964224 csum 0xd025b212 expected csum 0xb21fb46e mirror 1

[44294.737886] BTRFS error (device sda1): bad tree block start 1544664303443902517 371834880

[44294.747564] BTRFS warning (device sda1): csum failed root 5 ino 15937 off 1718964224 csum 0xd025b212 expected csum 0xb21fb46e mirror 1

[44294.747655] monerod[9085]: unhandled level 3 translation fault (7) at 0xfffd8606500a, esr 0x92000007, in monerod[400000+531000]

[44294.747709] CPU: 3 PID: 9085 Comm: monerod Tainted: G         C      4.14.4-sunxi64 #11

[44294.747715] Hardware name: Xunlong Orange Pi Prime (DT)

[44294.747723] task: ffff800001b1a880 task.stack: ffff000010cf0000

[44294.747736] PC is at 0x77b350

[44294.747742] LR is at 0x77b30c

[44294.747751] pc : [<000000000077b350>] lr : [<000000000077b30c>] pstate: 80000000

[44294.747755] sp : 0000ffff747fb270

[44294.747761] x29: 0000ffff747fb270 x28: 0000000000030f26 

[44294.747775] x27: 0000000000877000 x26: 0000fffd34e20fc0 

[44294.747788] x25: 00000000033af000 x24: 0000ffff747fb6b8 

[44294.747800] x23: 0000000000000000 x22: 0000000000000000 

[44294.747813] x21: 0000000000000000 x20: 0000000000000013 

[44294.747826] x19: 0000ffff603b3de8 x18: 0000000000000014 

[44294.747838] x17: 0000ffff8924df00 x16: 0000000000942f10 

[44294.747896] x15: 000072ffd3f721fa x14: 0011323f51ffc609 

[44294.747919] x13: ffffffffffff0000 x12: 0000000000000001 

[44294.747932] x11: 0000000000000005 x10: 00000000000007d1 

[44294.747945] x9 : 0000000038d59000 x8 : 0000fffd1882c554 

[44294.747957] x7 : 06023b1a073a0807 x6 : 0000000000006755 

[44294.747970] x5 : 00000000000a0000 x4 : 0000000000000000 

[44294.747982] x3 : 0000000000000004 x2 : 0000fffd86065000 

[44294.747994] x1 : 0000ffff603b3dee x0 : 0000ffff603b3e00 

[44325.465014] sd 0:0:0:0: [sda] tag#0 uas_eh_abort_handler 0 uas-tag 1 inflight: IN 

[44325.465027] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 00 2b 5c e0 00 00 20 00

[44325.468270] scsi host0: uas_eh_device_reset_handler start

[44325.593022] usb 3-1: reset high-speed USB device number 2 using ehci-platform

[44325.802178] scsi host0: uas_eh_device_reset_handler success

[44325.815135] BTRFS info (device sda1): read error corrected: ino 0 off 371834880 (dev /dev/sda1 sector 742624)

[44325.817491] BTRFS info (device sda1): read error corrected: ino 0 off 371838976 (dev /dev/sda1 sector 742632)

[44325.817986] BTRFS info (device sda1): read error corrected: ino 0 off 371843072 (dev /dev/sda1 sector 742640)

[44325.818485] BTRFS info (device sda1): read error corrected: ino 0 off 371847168 (dev /dev/sda1 sector 742648)

 

[grep]

Same problem with "externally powered" HDD (I use 2-plug USB cable and power connected to separate 2.1A USB-charger; cannot post armbianmonitor -u "500 Internal server error"):

Spoiler

[ 5263.346836] nr_pdflush_threads exported in /proc is scheduled for removal

[ 9309.154150] BTRFS warning (device sda1): csum failed root 5 ino 188237 off 1220608 csum 0xc57cb381 expected csum 0xeb57aed1 mirror 1

[ 9309.154396] BTRFS warning (device sda1): csum failed root 5 ino 188237 off 1273856 csum 0x69f266d3 expected csum 0x2688e7f8 mirror 1

[ 9309.154786] BTRFS warning (device sda1): csum failed root 5 ino 188237 off 1347584 csum 0x5dbb75ac expected csum 0x38a199da mirror 1

[ 9309.155934] BTRFS warning (device sda1): csum failed root 5 ino 188237 off 1425408 csum 0xd2698be9 expected csum 0x6ec754f0 mirror 1

[ 9309.156558] BTRFS warning (device sda1): csum failed root 5 ino 188237 off 1474560 csum 0xa4797809 expected csum 0x696e5611 mirror 1

[ 9309.156826] BTRFS warning (device sda1): csum failed root 5 ino 188237 off 1511424 csum 0xeee17752 expected csum 0xbc5d808e mirror 1

[ 9309.157138] BTRFS warning (device sda1): csum failed root 5 ino 188237 off 1560576 csum 0xefa26d11 expected csum 0x5b943b6f mirror 1

[ 9309.157766] BTRFS warning (device sda1): csum failed root 5 ino 188237 off 1736704 csum 0x9d3d20d1 expected csum 0x5bdd3ccd mirror 1

[ 9309.159047] BTRFS warning (device sda1): csum failed root 5 ino 188237 off 1806336 csum 0x43308e88 expected csum 0x0cd414bf mirror 1

[ 9309.159399] BTRFS warning (device sda1): csum failed root 5 ino 182903 off 102400 csum 0xf341c701 expected csum 0xbc5ddfee mirror 1

[ 9320.031268] btrfs_print_data_csum_error: 25 callbacks suppressed

[ 9320.031282] BTRFS warning (device sda1): csum failed root 5 ino 182903 off 1273856 csum 0xc1983611 expected csum 0xf03942a1 mirror 1

[ 9320.031364] bitcoin-msghand[4677]: unhandled level 3 translation fault (7) at 0xffff6fe87886, esr 0x92000007, in bitcoind[aaaac0205000+52a000]

[ 9320.031501] CPU: 1 PID: 4677 Comm: bitcoin-msghand Tainted: G         C      4.14.4-sunxi64 #11

[ 9320.031504] Hardware name: Xunlong Orange Pi Prime (DT)

[ 9320.031508] task: ffff8000742a0d80 task.stack: ffff00000e690000

[ 9320.031519] PC is at 0xaaaac056c2f8

[ 9320.031522] LR is at 0xaaaac056c104

[ 9320.031525] pc : [<0000aaaac056c2f8>] lr : [<0000aaaac056c104>] pstate: 80000000

[ 9320.031527] sp : 0000ffff667fa290

[ 9320.031530] x29: 0000ffff667fa290 x28: 0000ffff667fa410 

[ 9320.031535] x27: 000000000000101b x26: 0000aaaac05b4000 

[ 9320.031540] x25: 0000ffff667fa408 x24: 000000000000101c 

[ 9320.031545] x23: 0000000000000000 x22: 0000000000000000 

[ 9320.031550] x21: 0000aaaac0756000 x20: 0000aaaac0749000 

[ 9320.031554] x19: 0000ffff6fe87886 x18: 000000001a60b802 

[ 9320.031559] x17: 0000ffff87a83880 x16: 0000aaaac0749818 

[ 9320.031564] x15: 00000000decdfdaf x14: 00000000678bc9cc 

[ 9320.031569] x13: ffffffff00000000 x12: 0000000000000000 

[ 9320.031574] x11: 0000000000000020 x10: 000000000000a0a1 

[ 9320.031579] x9 : 0000ffff5c000020 x8 : 0000ffff667fa358 

[ 9320.031584] x7 : 0000000000000000 x6 : 0000ffff880d4cb8 

[ 9320.031589] x5 : 0000ffff6fe87888 x4 : 0000ffff6fe87886 

[ 9320.031593] x3 : 0000aaaac0684e78 x2 : 0000ffff6fe888a2 

[ 9320.031598] x1 : 0000aaaac0684000 x0 : 00000000ffffffff 

[ 9323.838204] BTRFS warning (device sda1): csum failed root 5 ino 15937 off 2995359744 csum 0x46d3d7a0 expected csum 0x07dea30e mirror 1

[ 9323.838646] BTRFS error (device sda1): bad tree block start 14820449172188037173 51750158336

[ 9323.847426] BTRFS warning (device sda1): csum failed root 5 ino 15937 off 2995359744 csum 0x46d3d7a0 expected csum 0x07dea30e mirror 1

[ 9323.847458] monerod[5332]: unhandled level 3 translation fault (7) at 0xffff0452100a, esr 0x92000007, in monerod[400000+531000]

[ 9323.847485] CPU: 1 PID: 5332 Comm: monerod Tainted: G         C      4.14.4-sunxi64 #11

[ 9323.847487] Hardware name: Xunlong Orange Pi Prime (DT)

[ 9323.847490] task: ffff800074a19b00 task.stack: ffff00000ed90000

[ 9323.847497] PC is at 0x77b350

[ 9323.847499] LR is at 0x77b30c

[ 9323.847502] pc : [<000000000077b350>] lr : [<000000000077b30c>] pstate: 80000000

[ 9323.847504] sp : 0000ffffafffc870

[ 9323.847506] x29: 0000ffffafffc870 x28: 0000000000877000 

[ 9323.847511] x27: 0000ffffa02f4980 x26: 0000ffff05ed0fc0 

[ 9323.847515] x25: 00000000033af000 x24: 0000ffffafffcca8 

[ 9323.847520] x23: 0000000000000000 x22: 0000000000000000 

[ 9323.847525] x21: 0000000000000000 x20: 000000000000000d 

[ 9323.847529] x19: 0000ffffa02cecd8 x18: 0000000000000014 

[ 9323.847534] x17: 0000ffffbaa112c0 x16: 0000ffffbaeb9290 

[ 9323.847539] x15: 00004d9a4bf92824 x14: 0aea9513d965a506 

[ 9323.847544] x13: ffffffffffff0000 x12: 0000000000000000 

[ 9323.847548] x11: 0000000000000010 x10: ac95b46849e835ed 

[ 9323.847553] x9 : 0000000032bc0f50 x8 : 0000ffff3650e562 

[ 9323.847557] x7 : 0000000000020000 x6 : 0000000000002899 

[ 9323.847562] x5 : 00000000000a0000 x4 : 0000000000000000 

[ 9323.847566] x3 : 0000000000000003 x2 : 0000ffff04521000 

[ 9323.847570] x1 : 0000ffffa02cecdc x0 : 0000ffffa02cece8 

[ 9354.568112] sd 0:0:0:0: [sda] tag#0 uas_eh_abort_handler 0 uas-tag 1 inflight: IN 

[ 9354.568125] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 06 36 8e 60 00 00 20 00

[ 9354.571404] scsi host0: uas_eh_device_reset_handler start

[ 9354.696121] usb 3-1: reset high-speed USB device number 2 using ehci-platform

[ 9354.853278] scsi host0: uas_eh_device_reset_handler success

[ 9354.876009] BTRFS info (device sda1): read error corrected: ino 0 off 51750158336 (dev /dev/sda1 sector 103188064)

[ 9354.876503] BTRFS info (device sda1): read error corrected: ino 0 off 51750162432 (dev /dev/sda1 sector 103188072)

[ 9354.876998] BTRFS info (device sda1): read error corrected: ino 0 off 51750166528 (dev /dev/sda1 sector 103188080)

[ 9354.877498] BTRFS info (device sda1): read error corrected: ino 0 off 51750170624 (dev /dev/sda1 sector 103188088)

 

 

So, @Igor @guidol @zador.blood.stained +1 to "this is not underpower caused by HDD connection"

 

[Igor]

Thanks. It will be investigated when possible. 

Please use:
 

Spoiler

code <> under spoiler for better forum readability when pasting walls of logs

 

 

 

[Naguissa]

Maybe overheat problem?

Enviado desde mi Jolla mediante Tapatalk

[grep]

@Naguissa Max temp under load is not more then 50, usual range is ~44..48. Not too much IMHO, but need to see CPU specification (but lazy a bit ... )

@Igor i've set 1.1Ghz via "userspace" governor to evaluate stability with low freq (and, therefore, voltage). 

And, BTW,  http://sprunge.us/ still returns 500, for example: http://sprunge.us/hUZM (i don't know is that yours or external service, but just informing)

 

 

[Igor]

47 minutes ago, grep said:

And, BTW,  http://sprunge.us/ still returns 500 (i don't know is that your or external service, but just informing)

https://github.com/armbian/build/commit/7f2337ed538abcbc09845617c2cc0ba0522ed25a

[grep]

ok, took latest with this commit (strange, but see no changes in this commit except "if-ing" & more strict checking to detect absence of  temp-sensors, but doest not matter) 

Anyway, crash with 1104MHz ( http://sprunge.us/cacb ). Reasonable to assume problems with my USB external case (thing i could test by myself)?... Will check in a couple of days.

@Igor So, unleashing 1152MHz  (instead of previous max of 1108Mhz) does not look successfull yet. At least with my board

+more: with 1104MHz & current load (monerod & bitcoind with cpulimit 200+150) i could not even perform a login via ssh until it crashed ("key exchange timeout"), so, i suppose, it was a "real-life stress-test". 

[Igor]

1 hour ago, grep said:

ok, took latest with this commit

This was referring only to a quoted problem "armbianmonitor returns 500", while the rest is a low priority. Board kernel is still in the development phase which means there is no support/you are on your own if you try to use it.

[tkaiser]

3 hours ago, grep said:

BTW,  http://sprunge.us/ still returns 500

 

When you see 'Error 500' accessing any page on sprunge.us this means always the same: Internal server error at sprunge.us (this is an online pasteboard service Armbian is only using since majority of users fails with providing log output -- so we integrated an automatic upload mechanism in Armbian that works almost all the time but sometimes not, simply try it again)

 

And you can stop flooding this thread with more and more underpowering reports since it's pointless. I also have no idea why this thread is not already in the correct subforum.

[grep]

14 minutes ago, Igor said:

This was referring only to a quoted problem "armbianmonitor returns 500", while the rest is a low priority. Board kernel is still in the development phase which means there is no support/you are on your own if you try to use it.

I've already got it  So, i try to do my best to solve my problems by myself with help of volunteered people around.

[grep]

11 minutes ago, tkaiser said:

And you can stop flooding this thread with more and more underpowering reports since it's pointless. I also have no idea why this thread is not already in the correct subforum.

Then move it to correct subforum, see no problem.

 

11 minutes ago, tkaiser said:

And you can stop flooding this thread with more and more underpowering reports since it's pointless.

This is just your  POW, nothing more, agreed ? 

I really should "hide" those walls with quoting, but in fact, there was at least two ideas based on output: about external power supply for USB-HDD and "cpu underpowering => test with lower freq".  This is "more then no support", so 1) thanks for all 2) walls are working.

And that's my opinion, and, as i guess, it does not contraversal to forum usage rules.

 

[tkaiser]

24 minutes ago, grep said:

This is just your  POW, nothing more, agreed ? 

Feel free to waste your time further on this, I only answer in the hope that some of the 8 moderators in this forum later do what they originally wanted to do (some housekeeping to help users):

• 'usb 3-1: reset high-speed USB device number 2 using ehci-platform' this message tells you that USB controller 3-1 lost control to device 3-2 (that's your external JMS578). Usually this is the result of undervoltage, the external disk doesn't get enough voltage (which HDD do you use? Some of them start to get in trouble when voltage drops below 4.7V, some others are even fine with 4.4V), disconnects from JMS578's SATA PHY which in turn let the JMS578 disappear from the USB bus. Reduced consumption let's the voltage rise again, the disk reappears and so the JMS578 does
• In the meantime your filesystem is already corrupted (which is perfectly understandable if there are continous USB resets) so I would better check with 'btrfs scrub /home' now
• If you want to switch from trial&error mode into investigation mode you need a Multimeter to check voltage available on testpoints under load and in idle
• And if you want to solve the problem with an external disk that is not able to be powered on its own you need to provide a reliable power source able to provide a STABLE voltage and enough current (using those Y-cable and connecting them to two different power sources with different voltages available is nothing I would call 'stable power source')

 

Edit: Realized that this sounds all too harsh but I really just want you to stop wasting your time since it's really pointless. Even if OPi Prime is not supported it's IMO misleading to not tell you that you need a 3A PSU with stable voltage to do the next step. All of your symptoms are well known and with 99% probability underpowering related. So again it's pointless to spend any time on this before this common problem source is not eliminated.

[tkaiser]

On 13.12.2017 at 9:30 PM, grep said:

PSU is this one:  https://ru.aliexpress.com/item/Mini-switching-power-supply-15W-5V-3A-Single-Output-dc-power-supply-for-Led-Strip-AC110V/32800882848.html 

 

If this thing overheats there's only one solution: do not use it. In case you want to address your problem one more time in trial&error mode check the voltage available on the screw terminals. If you measure below 5V (which I would assume) use the blue thing next to it to adjust it to 5.25V, then try again.

[grep]

51 minutes ago, tkaiser said:

Feel free to waste your time further on this, I only answer in the hope that some of the 8 moderators in this forum later do what they originally wanted to do (some housekeeping to help users):

• 'usb 3-1: reset high-speed USB device number 2 using ehci-platform' this message tells you that USB controller 3-1 lost control to device 3-2 (that's your external JMS578). Usually this is the result of undervoltage, the external disk doesn't get enough voltage (which HDD do you use? Some of them start to get in trouble when voltage drops below 4.7V, some others are even fine with 4.4V), disconnects from JMS578's SATA PHY which in turn let the JMS578 disappear from the USB bus. Reduced consumption let's the voltage rise again, the disk reappears and so the JMS578 does
• In the meantime your filesystem is already corrupted (which is perfectly understandable if there are continous USB resets) so I would better check with 'btrfs scrub /home' now
• If you want to switch from trial&error mode into investigation mode you need a Multimeter to check voltage available on testpoints under load and in idle
• And if you want to solve the problem with an external disk that is not able to be powered on its own you need to provide a reliable power source able to provide a STABLE voltage and enough current (using those Y-cable and connecting them to two different power sources with different voltages available is nothing I would call 'stable power source')

 

Edit: Realized that this sounds all too harsh but I really just want you to stop wasting your time since it's really pointless. Even if OPi Prime is not supported it's IMO misleading to not tell you that you need a 3A PSU with stable voltage to do the next step. All of your symptoms are well known and with 99% probability underpowering related. So again it's pointless to spend any time on this before this common problem source is not eliminated.

@tkaiser Thanks, a lot of info, need to think about all you said.

[tkaiser]

32 minutes ago, grep said:

need to think about all you said

 

But please provide the requested information:

• Which HDD are you using (it's important since different brands behave differently wrt undervoltage)
• How do you really power your board? On the led strip PSU there's only a screw terminal. What is connected to this and how do the cable diameters look like?

The real problem is that the average user is not aware of voltage drops and that amperage ratings printed on the PSU are less relevant (ok, often it's both -- cheap PSUs show faked amperage ratings or there is at least the following relationship: a cheap 5V/3A PSU either provides 5V or 3A since if a device really would want to draw 3A the voltage already dropped to such low levels that the device to be powered crashed due to undervoltage).

 

In case your PSU is adjusted to provide less than 5V and you use tiny wires between PSU and board every single symptom you report is just the well known and pretty common undervoltage problem no normal user is willing to accept  (and that's what this subforum should serve for: giving a nice overview of 'real life underpowering stories' for other users to read through and learn from -- but if threads aren't readable any more due to confusion spread and walls of text for no reason this subforum becomes as useless as every piece of instructions we provide since no one ever will read them)

[grep]

5 minutes ago, tkaiser said:

Which HDD are you using (it's important since different brands behave differently wrt undervoltage)

SSD Samsung 850 Pro 512 Gb (...please don't remind me about "hunting squirrel with a howitzer"). Did no try to search deep, but official specification (https://www.samsung.com/us/system/consumer/product/mz/7k/e2/mz7ke256bw/850PRO.pdf) says (for "active" state) V = "5", Power = 3.5W (5Vx0.75A, as i assume); ok, let them lie and it's 7.5W under load (5Vx1.5A); i have a USB-charger with 2.1A (let it lie too, and there are only 1.7-1.8A which is enough). BUT: i also do have a SATA2USB with"unknown power characteristics" (so, there are at least 2 points to measure Voltage/Amp: SATA OUT to SSD, USB-IN to SATA2USB) and first one is "not the easiest thing to measure"

8 minutes ago, tkaiser said:

How do you really power your board? On the led strip PSU there's only a screw terminal. What is connected to this and how do the cable diameters look like?

PSU screw terminal <=> two thick wires taken from "standard black computer PSU power connector internals"  <=> twist, covered with insulating tape <=> two a bit thiner (but thick enough) wires ended with 1.7mm jack (cut from 3A PSU). All wires are thick enough IMHO, but i agreed, that real voltage/amp level is a subject to check (potential losses / additional resistance in points of twist i hope are negligible because of "contact surface length" which is formed as ~10mm of "naked" wire from each side )

 

I will try to measure voltage and amperage under load, but it will take a while.

 

P.S. +I also ordered Odroid-HDR-HC1 (because i just want things to work) so, maybe OPiPrime will be added to my collection of "things that just does not work out-of-the-box as needed for me" 

[tkaiser]

29 minutes ago, grep said:

(https://www.samsung.com/us/system/consumer/product/mz/7k/e2/mz7ke256bw/850PRO.pdf) says (for "active" state) V = "5", Power = 3.5W (5Vx0.75A, as i assume); ok, let them lie and it's 7.5W under load (5Vx1.5A); i have a USB-charger with 2.1A (let it lie too, and there are only 1.7-1.8A which is enough)

 

Hmm... you're focused on amperage/consumption but why? The specs talk about 'Standard Operating Voltage 5V ± 5%' and that's the most important sentence there: 4.75V is the minimum allowed according to these specs and 5.25V is the maximum. So in case your PSU is currently adjusted to 4.7V you're well below specs. You use a PSU with adjustable 5V output voltage so which voltage do you feed your board with? Unless you're able to answer this question it's really absolutely pointless to do any further step in any direction...

 

Besides that: I've always 3 Samsung SSDs flying around for tests only and they all run fine with just 4.5V (but none of them is a 'Pro', it's all cheap EVO stuff). One of those, the EVO750, is somewhat nasty since with this SSD I can power down every Micro USB powered board connected with an insufficient cable to the PSU. This thing seems to have an insanely high peak consumption at startup which then leads to either a huge voltage drop or consumption peak on the DC-IN input line immediately killing each board I tried with already. Works of course fine when powered from a separate PSU.

 

PS: I really start to hate this SBC crap... https://linux-sunxi.org/Xunlong_Orange_Pi_Prime (no one took the time to add a device page with PCB pictures so we could look for voltage testpoints)