[SOLVED, PSU issue, SMB server issue] Asus thinkerboard hangs

[Tido]

8 hours ago, marcopete87 said:

hot removing/plugging sd card is the best way to get corrupted filesystem.

sorry for confusion, I should have written on the next boot instead of reboot. I even remove the power cord.

Still I face the above written surprise. Can be the PSU or the board.

[marcopete87]

i've just submitted an assistance request to Asus.

 

update: i'm running stress-ng to stress-test cpu.

i'm also running memtester for ram.

 

update 1: during test i tried access internal apache server, this partially hanged tinkerboard (but memtest was running fine) then, after 5 minutes, it resumed from hang.

now, i'm suspecting something on swap file disabled.

 

update2: enabled swap, testing for system stability. System unstable as usual; on htop, kernel threads were at 100%

update3: i measured amperage, at full cpu load is 1.5A (rounded up).

Now, i've contacted seller for an rma.

[marcopete87]

Hi all, new tinkerboard, same issues.

[botfap]

23 hours ago, marcopete87 said:

Hi all, new tinkerboard, same issues.

We bought 250 tinkerboards in July and have deployed approx 180 for a specific project and not had a single hardware failure.  They are all deployed with micro USB PSU's. The chances of you getting 2 faulty devices in a row are pretty slim.

 

What PSU and what battery have you tried? Make, model and serial numbers, not current ratings, they are mostly horse shit. And what exact make and model of SD card?

 

Update: I got one of our engineers to load test 3 tinks this morning. They stay stable up to 2.18-2.21A current draw through the micro USB, above 2.3A we get hard lockups. We use switched 5.25V PSU's (5.26-5.31v measured) which drops to 5.12-5.15v at ~2.2A current draw. If you are really only drawing 1.5A max when in use then you have a PSU problem, workload issue or you have a mis-behaving peripheral connected.

[marcopete87]

As told before, raspberry pi branded psu, no usb devices connected.

I tried also an ikea powerbank and a no-name powerbank, which proved to support 2A.

I ruled out usb connector issue, because i used GPIO in board without differences.

I'm thinking it's something about RAM and SWAP use: i can replicate this issue running memtester, which left some 50MB of free ram, i try to open it apache2 server and... BAM, board hanged!

 

I measured amperage with stress-ng -c 4 (quad core cpu)

[botfap]

Problem is that there are at least 4 different RasPi PSU's that I have seen and not a single one of them is suitable for powering the tinkerboard which draws upto 3 x more current under load than any of the Pi models. The 2.5A Pi PSU we tested (comes with the Pi3 Kit) would only actually deliver ~1.4A reliably and when it gets hot it gets very unreliable. Thats why I asked  for the numbers off the PSU. The most power hungry PI model (3B) uses 30-40% of the total current than the tinkerboard at max load (see attached chart). Even if you connect the Pi PSU through the GPIO pins there still isnt enough current from any Pi branded PSU I have seen to power the tinkerboard reliably. As a general rule, RasPi accessories (like the boards) are made to very low standards and budgets and should be avoided.

 

If you can provide details for me to replicate your software, ramdisk, swap and workload I can have a look at reproducing the problem and finding a solution or at least a cause. The fact you have swap on SD make me shudder!

 

 

[tkaiser]

11 hours ago, botfap said:

The fact you have swap on SD make me shudder!

 

Check his log: http://sprunge.us/FJib (it makes no difference where swap that is never used is located on  )

 

I don't know why I'm posting in this thread since I wanted to stay away from shitty boards but anyway... I would recommend this test (while running 'sudo armbianmonitor -m' in another terminal):

sudo armbianmonitor -p
minerd --benchmark

Takes a minute to install cpuminer 2.4.5. If the board locks up within a second after starting minerd it's underpowered, otherwise armbianmonitor -m output might be interesting.

[marcopete87]

Standard armbian OS upgraded until 2 weeks ago kernel/patches, zoneminder, apache and mysql.

Everything you need is posted above, in firsts comments (with results).

 

However, raspberry psu is cold, it's rated 2.5A peak and 2A continuous usage.

Measured amperage says 1.5A peak current, so it's a lot lower than continuous permitted current, i also added an capacitor in case of voltage sag, but it seems to do nothing.

 

tkaiser, thank you, it was one of think i tried.

Now, i'm testing a new configuration, then i'll try with armbian image.

[marcopete87]

update: casually, i found this error on official thinkerboard debian image:

 

Message from syslogd@linaro-alip at Oct  7 13:37:07 ...

 kernel:[ 1440.791144] Kernel panic - not syncing: hung_task: blocked tasks

[marcopete87]

Update 2: i'm using both official Asus image and Armbian_5.33.171008_Tinkerboard_Ubuntu_xenial_dev_4.14.0-rc3_desktop.img nightly build.

First image seems stable with one camera @ 20fps (powered by Raspberry pi PSU, i won't add any other camera due hanging issue)

Second image seems stable with two cameras @ 10 fps each (powered by ikea SOLBANA), last image hanged after 10 - 20 minutes, this is still running after 1 hour.

I hope i can make this topic solved (due something been updated), but stay tuned.

 

Update3: issue is not solved.  

[chwe]

On 8.10.2017 at 4:24 PM, marcopete87 said:

First image ... Second image 

Don't make it more confusing than it is..
So first image means official asus image and second image means armbian nightly?

Call it first, second and last makes it hard to follow.. 

 

On 7.10.2017 at 1:00 PM, marcopete87 said:

Standard armbian OS upgraded until 2 weeks ago kernel/patches, zoneminder, apache and mysql.

Everything you need is posted above, in firsts comments (with results).

IMO it would be smarter to repeat once your setup for him. If someone offers you to nail down your issue for free you should make it as easy as possible for him.  He still has to set up a tinker with all your program dependencies (I would be annoyed if I had to figure out your setup before debugging too).

Quote

after 15 minutes hanged.

I don't have any good powered usb hub.

with 3 cameras, hangs after 5 minutes.

when hanged, it had 3 ssh sessions opened:

This sounds really like a power issue. 

 

Something to consider:
To my knowledge ZoneMinder uses FFmpeg. Does FFmpeg on the tinker support hardware accelerated decoding? (I'm not up to date, maybe @TonyMac32 or @Myy knows this better). This would reduce a lot of load from the CPU which is fore sure not the worst idea. 

 

[marcopete87]

Yes, i'm using two different distribution in two different boards, i exchanged microsd card to check if one board is more stable than other, but they seems to be

twins (compare against other Linux version may be useful).

 

Armbian version, now is the latest (as i told on Sunday), base image, desktop disabled, only zoneminder + apache2 installed (sql server is on other pc); i use smb shares to store images.

Cameras are IP cameras, not USB, there is nothing attached on USB ports.

However, i replaced power supply, boards seems to be stable, but i'm still testing both (power over micro usb).

 

Yes, zoneminder uses FFmpeg, but jpg encoding is still an cpu task.

 

My doubt is about why board hanged after an progressive slow down in performances at normal cpu speed?

If was only an power issue, board would crash or reboot without slowing.

[marcopete87]

Hi all, as you told me before, it's likely an 5V power issue; i was stupid i didn't double checked with another different cable-power supply combination.

 

At the end, i changed power supply with an SBS 7A 4 port usb charger SBS model TTTRAV5USB7A and i replaced 1 micro USB cable, now both boards are working fine from yesterday morning (now, they are online from 30 hours).

So, i had 2 power supply which weren't capable to deliver enought current and 1 usb cable with too much resistance despite is 30cm long.

 

My test were:

Raspberry pi PSU, crash on both boards

Ikea battery with 30cm cable, crash on both boards

Onite battery with 30 and 60cm cable, crash on both boards.

PC capable delivering 2A, 30cm cable, crash on both boards.

 

Yesterday, before buying new power supply, i tried ikea battery with 60cm usb cable, board was stable.

Then i replaced raspberry pi psu and 30cm usb cable and they seems to work fine.

 

So, thank you all for time you spent on me, next time i'll triple check power supply issue before posting.

[Tido]

Hey @tkaiser, one more to add to your list

[botfap]

@marcopete87 that’s good news  Not entirely unexpected but at least you know for next time

 

 

[tkaiser]

5 minutes ago, Tido said:

one more to add to your list

 

The issue was described in the thread title from the very beginning (Tinkerboard + hangs = underpowering issue) and that's why I suggested numerous times to add a cpuburn run to our firstrun script simply killing all those boards with insufficient powering before all the useless drama starts over and over again. But no, we know that no one is reading documentation, we know that no one is willing to accept the simple fact that hardware might not be sufficient and we allow both our users directly affected and other users trying to help wasting their time over and over again.

[marcopete87]

1 hour ago, tkaiser said:

 

The issue was described in the thread title from the very beginning (Tinkerboard + hangs = underpowering issue) and that's why I suggested numerous times to add a cpuburn run to our firstrun script simply killing all those boards with insufficient powering before all the useless drama starts over and over again. But no, we know that no one is reading documentation, we know that no one is willing to accept the simple fact that hardware might not be sufficient and we allow both our users directly affected and other users trying to help wasting their time over and over again.

i used stress-ng + memtester.

 

however, after previous reply, armbian running board hanged, so i'm giving up.

[tkaiser]

20 minutes ago, marcopete87 said:

i used stress-ng + memtester

 

The latter is not sufficient to test for this kind of underpowering and stress-ng is just a lightweight joke. I suggested to you testing with cpuminer (using NEON instructions and pretty good in deadlocking underpowered boards, I personally know just one other tool that is better in this regard: cpuburn -- both tools are at least twice as heavy as stress-ng or other lightweight tests).

[TonyMac32]

I power directly over GPIO for that reason,  this board was simply not designed well power wise.  I also put a heat sink on the RK808 PMIC, that little chip gets smoking hot when under heavy load.

[marcopete87]

16 hours ago, tkaiser said:

 

The latter is not sufficient to test for this kind of underpowering and stress-ng is just a lightweight joke. I suggested to you testing with cpuminer (using NEON instructions and pretty good in deadlocking underpowered boards, I personally know just one other tool that is better in this regard: cpuburn -- both tools are at least twice as heavy as stress-ng or other lightweight tests).

i'm trying minerd, but board seems stable as voltages.
Maybe there is something on software level...

[marcopete87]

Update: after issue was unsolved on both boards, i moved usb disks from smb server to tinkerboards, mounted storage and i'm using usb storage without issues.

So, issue was 30% caused by power supply and 70% due heavy smb usage which caused threads hangs as smb server disconnected clients.

 

Now, i have 3 boards with usb powered 2.5" hdd (all powered via board micro usb connector), they are stable.

[tkaiser]

1 hour ago, marcopete87 said:

Now, i have 3 boards with usb powered 2.5" hdd (all powered via board micro usb connector), they are stable.

 

They're stable until the whole thingie gets instable again. Powering through Micro USB is ALWAYS WRONG and spending time on such support dramas like yours (especially again and again) is only wasting time for everyone.

 

Seriously:

On 10.10.2017 at 4:23 PM, tkaiser said:

I suggested numerous times to add a cpuburn run to our firstrun script simply killing all those boards with insufficient powering before all the useless drama starts over and over again.

@Igor and @zador.blood.stained: Why can't we add this to firstrun? Saves everyone a lot of time, efforts and trouble. Users will complain in the beginning but in half a year it gets accepted as most simple test for shitty hardware designs equipped with Micro USB for powering: 'If Armbian is available, try this out first. If it not boots then there's something wrong with powersupply'

 

Of course we can introduce a new board config setting, eg. BOARD_DESIGNER_IS_MORON or PROBABLY_UNDERPOWERED or something like that to limit such an underpowering test only to hardware design fails like this board here, MiQi, Pine64 and the other Micro USB victims...

 

BTW: Can a moderator please move the thread where it belonged to from post #1 on: https://forum.armbian.com/index.php?/forum/31-sd-card-and-power-supply/

[zador.blood.stained]

4 minutes ago, tkaiser said:

@Igor and @zador.blood.stained: Why can't we add this to firstrun?

Anybody with a push access and enough time to implement and test this can do it.

It wasn't added yet because it needs time and needs testing (so again time). Different CPUs need different cpuburn variants (and maybe a "stress" call as a fallback/alternative), and I have enough problems and enhancements on my TODO list that I probably won't touch this.

 

13 minutes ago, tkaiser said:

'If Armbian is available, try this out first. If it not boots then there's something wrong with powersupply'

Or with your SD card. Or with the SD card slot. Or with onboard regulators fried due to defects or bad power supply (replacing PSU won't change anything with a damaged board). Or it's a BGA soldering defect / damage (I've read about several such cases with some very popular H3 boards).