Helios4 Support

[Igor]

36 minutes ago, gprovost said:

Any chance you can trigger a build of just Helios4 u-boot and publish the u-boot .deb in the armbian repo ?

It will be fixed automagically at next rebuild, nightly builds already have this fixed if someone needs an urgent fix.

[JakeK]

I just switched to nightly builds to test and so far I'm getting an IP address each boot. Thanks for your @gprovost hard work tracking down this issue and fixing it @Igor!

[soydemadrid]

Hi I hope someone can please help. I keep finding my Helios4 with its lights out and inaccessible. I can't ssh in and have to just power it off and reboot. I can't find much to debug it, but I was using ssh today with it to grep some files and try find a few things and it suddenly spat out loads of errors all over the place as follows:

 

Quote

grep: warning: sys/devices/platform/reg-dummy/subsystem/devices/system-leds/driver/io-leds/leds/helios4:green:ata3/subsystem/mmc0::/device/mmc_host/mmc0/mmc0:aaaa/block/mmcblk0/subsystem/sdd/device/scsi_disk/3:0:0:0/subsystem/1:0:0:0/device/subsystem/devices/2:0:0:0/driver/0:0:0:0/block/sda/bdi/subsystem/8:16/subsystem: recursive directory loop
sys/devices/platform/reg-dummy/subsystem/devices/system-leds/driver/io-leds/leds/helios4:green:ata3/subsystem/mmc0::/device/mmc_host/mmc0/mmc0:aaaa/block/mmcblk0/subsystem/sdd/device/scsi_disk/3:0:0:0/subsystem/1:0:0:0/device/subsystem/devices/2:0:0:0/driver/0:0:0:0/block/sda/bdi/subsystem/8:16/stable_pages_required:0
sys/devices/platform/reg-dummy/subsystem/devices/system-leds/driver/io-leds/leds/helios4:green:ata3/subsystem/mmc0::/device/mmc_host/mmc0/mmc0:aaaa/block/mmcblk0/subsystem/sdd/device/scsi_disk/3:0:0:0/subsystem/1:0:0:0/device/subsystem/devices/2:0:0:0/driver/0:0:0:0/block/sda/bdi/subsystem/8:16/power/runtime_suspended_time:0
grep: sys/devices/platform/reg-dummy/subsystem/devices/system-leds/driver/io-leds/leds/helios4:green:ata3/subsystem/mmc0::/device/mmc_host/mmc0/mmc0:aaaa/block/mmcblk0/subsystem/sdd/device/scsi_disk/3:0:0:0/subsystem/1:0:0:0/device/subsystem/devices/2:0:0:0/driver/0:0:0:0/block/sda/bdi/subsystem/8:16/power/autosuspend_delay_ms: Input/output error
sys/devices/platform/reg-dummy/subsystem/devices/system-leds/driver/io-leds/leds/helios4:green:ata3/subsystem/mmc0::/device/mmc_host/mmc0/mmc0:aaaa/block/mmcblk0/subsystem/sdd/device/scsi_disk/3:0:0:0/subsystem/1:0:0:0/device/subsystem/devices/2:0:0:0/driver/0:0:0:0/block/sda/bdi/subsystem/8:16/power/runtime_active_time:0
sys/devices/platform/reg-dummy/subsystem/devices/system-leds/driver/io-leds/leds/helios4:green:ata3/subsystem/mmc0::/device/mmc_host/mmc0/mmc0:aaaa/block/mmcblk0/subsystem/sdd/device/scsi_disk/3:0:0:0/subsystem/1:0:0:0/device/subsystem/devices/2:0:0:0/driver/0:0:0:0/block/sda/bdi/subsystem/8:16/power/control:auto
sys/devices/platform/reg-dummy/subsystem/devices/system-leds/driver/io-leds/leds/helios4:green:ata3/subsystem/mmc0::/device/mmc_host/mmc0/mmc0:aaaa/block/mmcblk0/subsystem/sdd/device/scsi_disk/3:0:0:0/subsystem/1:0:0:0/device/subsystem/devices/2:0:0:0/driver/0:0:0:0/block/sda/bdi/subsystem/8:16/power/runtime_status:unsupported
sys/devices/platform/reg-dummy/subsystem/devices/system-leds/driver/io-leds/leds/helios4:green:ata3/subsystem/mmc0::/device/mmc_host/mmc0/mmc0:aaaa/block/mmcblk0/subsystem/sdd/device/scsi_disk/3:0:0:0/subsystem/1:0:0:0/device/subsystem/devices/2:0:0:0/driver/0:0:0:0/block/sda/bdi/subsystem/8:16/max_ratio:100
sys/devices/platform/reg-dummy/subsystem/devices/system-leds/driver/io-leds/leds/helios4:green:ata3/subsystem/mmc0::/device/mmc_host/mmc0/mmc0:aaaa/block/mmcblk0/subsystem/sdd/device/scsi_disk/3:0:0:0/subsystem/1:0:0:0/device/subsystem/devices/2:0:0:0/driver/0:0:0:0/block/sda/bdi/subsystem/8:16/min_ratio:0
sys/devices/platform/reg-dummy/subsystem/devices/system-leds/driver/io-leds/leds/helios4:green:ata3/subsystem/mmc0::/device/mmc_host/mmc0/mmc0:aaaa/block/mmcblk0/subsystem/sdd/device/scsi_disk/3:0:0:0/subsystem/1:0:0:0/device/subsystem/devices/2:0:0:0/driver/0:0:0:0/block/sda/bdi/subsystem/7:5/read_ahead_kb:128
grep: warning: sys/devices/platform/reg-dummy/subsystem/devices/system-leds/driver/io-leds/leds/helios4:green:ata3/subsystem/mmc0::/device/mmc_host/mmc0/mmc0:aaaa/block/mmcblk0/subsystem/sdd/device/scsi_disk/3:0:0:0/subsystem/1:0:0:0/device/subsystem/devices/2:0:0:0/driver/0:0:0:0/block/sda/bdi/subsystem/7:5/subsystem: recursive directory loop
sys/devices/platform/reg-dummy/subsystem/devices/system-leds/driver/io-leds/leds/helios4:green:ata3/subsystem/mmc0::/device/mmc_host/mmc0/mmc0:aaaa/block/mmcblk0/subsystem/sdd/device/scsi_disk/3:0:0:0/subsystem/1:0:0:0/device/subsystem/devices/2:0:0:0/driver/0:0:0:0/block/sda/bdi/subsystem/7:5/stable_pages_required:0
sys/devices/platform/reg-dummy/subsystem/devices/system-leds/driver/io-leds/leds/helios4:green:ata3/subsystem/mmc0::/device/mmc_host/mmc0/mmc0:aaaa/block/mmcblk0/subsystem/sdd/device/scsi_disk/3:0:0:0/subsystem/1:0:0:0/device/subsystem/devices/2:0:0:0/driver/0:0:0:0/block/sda/bdi/subsystem/7:5/power/runtime_suspended_time:0
grep: sys/devices/platform/reg-dummy/subsystem/devices/system-leds/driver/io-leds/leds/helios4:green:ata3/subsystem/mmc0::/device/mmc_host/mmc0/mmc0:aaaa/block/mmcblk0/subsystem/sdd/device/scsi_disk/3:0:0:0/subsystem/1:0:0:0/device/subsystem/devices/2:0:0:0/driver/0:0:0:0/block/sda/bdi/subsystem/7:5/power/autosuspend_delay_ms: Input/output error

 

I don't know if the above is related to my problem but I hope someone can fix as it is a real pain when I try to print something (I have an office Cups network printer on the Helios) or access Samba or such and they aren't there or available so I then have to reboot it...

[soydemadrid]

I'm also getting loads of other weird errors before the above too like this:

 

Quote

# grep -R "sometexthere"
Binary file sys/kernel/notes matches
sys/kernel/uevent_helper:
Binary file sys/kernel/mm/page_idle/bitmap matches
sys/kernel/mm/swap/vma_ra_enabled:true
sys/kernel/mm/ksm/use_zero_pages:0
sys/kernel/mm/ksm/pages_volatile:0
sys/kernel/mm/ksm/stable_node_dups:0
sys/kernel/mm/ksm/stable_node_chains:0
sys/kernel/mm/ksm/full_scans:0
sys/kernel/mm/ksm/pages_to_scan:100
sys/kernel/mm/ksm/pages_unshared:0
sys/kernel/mm/ksm/pages_shared:0
sys/kernel/mm/ksm/stable_node_chains_prune_millisecs:2000
sys/kernel/mm/ksm/sleep_millisecs:20
sys/kernel/mm/ksm/pages_sharing:0
sys/kernel/mm/ksm/max_page_sharing:256
sys/kernel/mm/ksm/run:0
sys/kernel/security/lsm:capability
sys/kernel/fscaps:1

 

 

Tons and tons of them that go on for hundreds and thousands of entries - too much to list here but hopefully you get the idea...

[Igor]

23 minutes ago, soydemadrid said:

Hi I hope someone can please help.

Please provide more date - which kernel are you using, the best way is to run (as superuser):

armbianmonitor -u

 

[gprovost]

@soydemadrid First, as Igor suggested, please use armbianmonitor -u and share the link it will generate here.

 

The issue you are witnessing is unrelated to the grep command output you are sharing. However your grep command that you are running from the root directory effectively highlights a weird recursive symlink loops in sysfs. We will look at it but in any case it's unrelated to your unreachable Helios4 issue.

 

When you say all lights are out, Do you also mean the LED8 is also off ? This led is located close to the DC input power connector.

Or do you mean LED8 is ON but LED1 and the others one are not blinking ?

 

Can you also check with the following command last -x | less if you see some reboot or shutdown events that seems suspicious.

 

[soydemadrid]

Hi thanks for the replies - I think you're right in that the grep stuff doesn't have anything to do with my Helios becoming unreachable.

 

Here is the output of the diagnostics: http://ix.io/1mNq

 

With the LEDS I mean that 8 is on but the activity isn't blinking when the unit crashes or becomes unavailable.

 

The output of less -x | less looks ok. There are listings of just my ssh remote access user and a few runlevel/reboot/shutdown entries but nothing looks weird...

 

Thanks again for any help. I did wonder if it might be something going on with my network cups printer causing the crash or unavailability?

[gprovost]

20 hours ago, soydemadrid said:

With the LEDS I mean that 8 is on but the activity isn't blinking when the unit crashes or becomes unavailable.

 

If LED1 is not blinking, it either means that the system hanged OR the system was shutdown || put in suspend-mode.

 

First I would advice that you update to the latest kernel, for that you will need to change your repo to nightly build. Use armbian-config utility for that.

 

armbian-config > System > Nightly

 

This will automatically update your system with latest kernel and u-boot.

 

Beside that, I don't see anything wrong with your system. Next time your system hangs, can you post your system log (all the syslog files in /var/log and /var/log.hdd).  You can send it to me by PM.

 

[nemo19]

On 9/16/2018 at 12:14 PM, soydemadrid said:

With the LEDS I mean that 8 is on but the activity isn't blinking when the unit crashes or becomes unavailable.

 

On 9/14/2018 at 3:09 PM, soydemadrid said:

Hi I hope someone can please help. I keep finding my Helios4 with its lights out and inaccessible. I can't ssh in and have to just power it off and reboot. I can't find much to debug it, [...]

I don't know if the above is related to my problem but I hope someone can fix as it is a real pain when I try to print something (I have an office Cups network printer on the Helios) or access Samba or such and they aren't there or available so I then have to reboot it...

 

Hi,

 

I don't have a solution, but thought I should let you know that you're not alone with that problem. A while back I had that happen very frequently, about every or every other day. Turned out that my Helios had no thermal pad. Thanks to @gprovost I got a thermal pad and the frequent hanging stopped and the board runs much cooler and the fans are much more quiet now.

 

Unfortunately the Helios still hangs every couple of weeks at random times. Mostly when idle, sometimes while watching a movie from a samba share. Symptoms are the same as yours: Power LED on. all other LEDs off. No SSH connection. Green LED on Ethernet port on, orange LED periodically flashing in short bursts. I meant to report this a while ago, but didn't get around to it.

 

I'll try to have a look at the logs the next time it happens and might switch to the nightly builds again. If someone has some other advice I'd happily try it.

 

Cheers!

[Igor]

5 minutes ago, nemo19 said:

Unfortunately the Helios still hangs every couple of weeks at random times.

I am running storage services and TVheadend and so far it never hanged. Now it's almost two weeks since the last kernel upgrades so I need to wait a little longer. Logs: http://ix.io/1mVr

[gprovost]

@nemo19 Sorry to hear that now you are facing some other instability with the board. Please next time it occurs, can you send me your syslog files along the link generated by armbianmonitor -u

 

 

[NickS]

Helios4 Fan problems again!

 

OK, I got hold of an OLED display and have it installed and set up - it works really well - thankyou.

 

I significantly changed your script and included a panel for CPU load, system temperature and system fan speeds.

 

This is where I noticed that fancontrol does indeed increase the fan speed values in /dev/fan-j10/pwm1 and /dev/fan-j17/pwm1 as the temperature value in

/dev/thermal-board/temp1_input is seen to rise. However I never see or hear any change to the physical fans' speed. The fans' speed remain constant at boot,

after boot, during shutdown and even after shutdown.

I turned on DEBUG in fancontrol and it appears to be doing what it is supposed to do. Every 10 seconds or so it reads the temp and calculates a new fan speed value

which it then writes to the two fan control files.  I even modified fancontrol to write values 1 to 255 incremented every 10 seconds and saw absolutely no change to

the physical fan speed.

 

It looks as if either the fan driver is not using the values set in /dev/fan-j1x/pwm1 to control fan speeds or the physical fans are not responding to the driver.

Can you suggest how I debug this please?

 

Overall the continuous low fan speed appears to keep the temperature of the board & CPU at a reasonable level (below 50C) - but the

HDDs do get a bit hot from time to time - so I would like to try and fix this.

 

I am running Jessie 4.4.112-mvebu with OMV 3.0.99 (Erasmus)

[Igor]

1 hour ago, NickS said:

I am running Jessie 4.4.112-mvebu with OMV 3.0.99 (Erasmus)

That's an old build ... fan control works properly on current stable 4.14.70-mvebu build (NEXT kernel branch) ... which is recommended build in any case. OLED display must also work on this kernel.  

 

Debian Stretch based OMV is also a better choice now.

[NickS]

OK yesterday I installed a brand new test system (well it was until you guys slipped 4.14.66 in overnight!)

Armbian_Helios4_Debian_Stretch_4.14.20.img.xz
Debian 9 Stretch (Kernel 4.14.20)
Build date : 17/02/2018
Size : 229 MB

 

I'm seeing exactly the same symptoms.

I turned DEBUG=1 on in fancontrol and let it rip and pasted the output below.

I then hacked fancontrol to report a temperature value of 99C (99000) and tested again.

You will note that fancontrol sets the fan speeds to 255 (max allowed) so fancontrol seems to be working as designed

However the actual fans don't spin any faster?

 

... here's the DEBUG output of regular fancontrol first followed by my hacked 99C version

 

=====================================================

Loading configuration from /etc/fancontrol ...

Common settings:
  INTERVAL=10

Settings for /dev/fan-j10/pwm1:
  Depends on /dev/thermal-cpu/temp1_input
  Controls
  MINTEMP=55
  MAXTEMP=95
  MINSTART=50
  MINSTOP=50
  MINPWM=50
  MAXPWM=255

Settings for /dev/fan-j17/pwm1:
  Depends on /dev/thermal-cpu/temp1_input
  Controls
  MINTEMP=55
  MAXTEMP=95
  MINSTART=50
  MINSTOP=50
  MINPWM=50
  MAXPWM=255

Enabling PWM on fans...
Starting automatic fan control...
pwmo=/dev/fan-j10/pwm1
tsens=/dev/thermal-cpu/temp1_input
fan=
mint=55000
maxt=95000
minsa=50
minso=50
minpwm=50
maxpwm=255
tval=74642
pwmpval=255
fanval=
min_fanval=1
new pwmval=150
pwmo=/dev/fan-j17/pwm1
tsens=/dev/thermal-cpu/temp1_input
fan=
mint=55000
maxt=95000
minsa=50
minso=50
minpwm=50
maxpwm=255
tval=73690
pwmpval=255
fanval=
min_fanval=1
new pwmval=145

========================================================

Loading configuration from /etc/fancontrol ...

Common settings:
  INTERVAL=10

Settings for /dev/fan-j10/pwm1:
  Depends on /dev/thermal-cpu/temp1_input
  Controls
  MINTEMP=55
  MAXTEMP=95
  MINSTART=50
  MINSTOP=50
  MINPWM=50
  MAXPWM=255

Settings for /dev/fan-j17/pwm1:
  Depends on /dev/thermal-cpu/temp1_input
  Controls
  MINTEMP=55
  MAXTEMP=95
  MINSTART=50
  MINSTOP=50
  MINPWM=50
  MAXPWM=255

Enabling PWM on fans...
Starting automatic fan control...
***************** +99000
pwmo=/dev/fan-j10/pwm1
tsens=/dev/thermal-cpu/temp1_input
fan=
mint=55000
maxt=95000
minsa=50
minso=50
minpwm=50
maxpwm=255
tval=99000
pwmpval=255
fanval=
min_fanval=1
new pwmval=255
***************** +99000
pwmo=/dev/fan-j17/pwm1
tsens=/dev/thermal-cpu/temp1_input
fan=
mint=55000
maxt=95000
minsa=50
minso=50
minpwm=50
maxpwm=255
tval=99000
pwmpval=255
fanval=
min_fanval=1
new pwmval=255

=================================================================

 

 

 

 

[gprovost]

@NickS First to test that your fans are responding properly to the different PWM value, you should eliminate fancontrol from the equation.

 

1. Stop fancontrol service

$> sudo systemctl stop fancontrol.service

 

2. Play with PWM value and see how the fan react. (accept value 0 to 255). Change to root user for that.

The different of speed / noise should very noticeable between the 3 speeds below, proving the fans respond properly to the different PWM value.

 

Full speed :

#> echo 255 > /dev/fan-j10/pwm1
#> echo 255 > /dev/fan-j17/pwm1

Medium speed :

#> echo 150 > /dev/fan-j10/pwm1
#> echo 150 > /dev/fan-j17/pwm1

Low speed :

#> echo 50 > /dev/fan-j10/pwm1
#> echo 50 > /dev/fan-j17/pwm1

 

So let me know the outcome of the above test ;-)

 

 

On 9/21/2018 at 8:13 PM, NickS said:

 

OK, I got hold of an OLED display and have it installed and set up - it works really well - thankyou.

 

I significantly changed your script and included a panel for CPU load, system temperature and system fan speeds.

 

Would be nice that you share your tweaks. Why not forking our sys-oled repo ;-)

 

[NickS]

Hello gprovost,

thanks for the quick reply.

I stopped fancontrol & tried all 3x2 direct updates to the pwm1 files.

A cat of the 2 files showed they had both taken the new values.

I saw/heard no change in physical fan speeds - they both just run at a constant low speed.

 

test system used...

Linux Helios4 4.14.70-mvebu #266 SMP Wed Sep 19 10:35:09 CEST 2018 armv7l GNU/Linux

 

Kind regards ... Nick

[gprovost]

@NickS Are you using the fan provided with the Helios4 kit ?

 

Any clue when you started to witness the issue ? I mean do you remember if the fan speed worked previous and it’s just recently that the fans seem to be stuck at low speed all the time ?

 

I’m wondering if something could be wrong with the fans themselves.

[NickS]

Hi gprovost,

Yes - I am using the original fans supplied with the Helios4 kit.

They were working fine after I applied the fan patch. ie they would go to full speed on

boot then settle down to a slower speed and occasionally I would hear them change pitch.

I noticed that the fans stopped going to full speed on boot about 6 weeks ago - and only

just got around to trying to understand what's going on. 

 

I may have some old fans lying around that i could substitute one of them with.

Will any 4-pin fan work?

 

Kind regards ... Nick

[Igor]

1 hour ago, NickS said:

then settle down to a slower speed and occasionally I would hear them change pitch.

 

Full at power on, going low when systems boot up and going up while stressing the board is expected ... and wanted behavior. If it works this way all is fine.

[NickS]

Igor - that's how it should work I know!

But if you read the thread you'll see that they just stay at low speed all the time now.

 

Anyway ... moving on ... following on from gprovosts suggestion... I found a brand new Arctic F8 PWM rev.2 4 pin fan

in my workshop and substituted that for one of the Helios4 kit fans.

 

#> echo 255 > /dev/fan-j10/pwm1

#> echo 255 > /dev/fan-j17/pwm1

#> echo 50 > /dev/fan-j10/pwm1

#> echo 50 > /dev/fan-j17/pwm1

 

Above commands have no effect. Both fans (kit & new replacement) run exactly the same speed  - unchanging.

So I find it hard to believe that its a fan problem. I wish I had a scope to put across the fan pins - but sadly

I don't. But PWM fans in general work at full power if no PWM signal is present - so I'm surmising

that there is a signal on the PWM pin that is providing a short duty cycle keep the speed low. Very odd.

 

Any idea how I might locate the driver and trace what its doing?

[Igor]

1 hour ago, NickS said:

But if you read the thread you'll see that they just stay at low speed all the time now.

Even you start with a clean image from the download section?

[NickS]

Hi Igor,

Yes, this is a test system - fresh new latest Stretch image downloaded and installed yesterday.

 

[gprovost]

@NickS Ok so at least you confirmed that the fans are not the problem.

 

Any chance you have at least a DC voltmeter ? If yes can you measure the voltage on the PWM pin.

Before doing that, please stop fan-control and set the PWM value to 255 on the header you are doing the measure.

 

Here the pin out of the Fan header.  You will see on the PCB the small white arrow that indicates pin 1 of the header.

 

 

19 hours ago, NickS said:

But PWM fans in general work at full power if no PWM signal is present - so I'm surmising

that there is a signal on the PWM pin that is providing a short duty cycle keep the speed low. Very odd. 

 

We are investigating on your side what could be this effectively odd issue.

It could be a too low voltage on the PWM pin that acts like a Duty Cycle of 0%.

 

[iamwithstupid]

Bad news, I contacted them via Mail and it seems like this was the last batch sold (at least that's what support told me).

 

The next one will be most likely another product

 

Why oh why can't this be sold regulary ???

 

I mean I could even live with preorder and wait a few months until produced - but this is almost like a guranteed death

 

(imagine one of the Dev-Units getting destroyed -> Sajonara development*).

 

*ok that's a bit overdramatized, but yea ... when there's no supply development will stall as soon as enough of the units are dead. I don't think anyone ordered 10+ on backorder just in case ... ???

[NickS]

Hi gprovost

 

Well more and more strange - I checked voltages across ground and all pins for both J10 & J17 at fan values of 255, 50 & 1 (always with fancontrol stopped).

Of course with a meter I can only see the mean voltage and not the waveform and duty cycle length.

The PWM voltages look a bit random.

Does this help you?

 

          12V        SENSE     PWM-255          PMW-50            PWM-1

J17    12.19      9.2           0.14                  0.04                   0.01

J10    12.19      9.1           1.19                 1.18                   1.18               

 

Kind regards ... Nick

[gprovost]

@NickS It's quite helpful measures. The voltage should be 3.3v on the PWM pin when PWM255.

 

We are experimenting on our side to try to reproduce what you are observing. Something might be wrong with your board GPIO 55 and 41.

 

Will keep you updated.

 

 

 

 

@iamwithstupid Sorry that you missed out the 2nd campaign. The reason we are not manufacturing a lot of additional units on top the pre-order ones is because we can't take the risk yet to have a big inventory. This will change with the next project. As to your comments, don't worry we have a enough dev units on our side to perform our dev / support tasks. Plus we have a small stock of spare units that are reserved in case of customer return / exchange.

[gprovost]

@NickS We are still investigating. If we come to the conclusion that the GPIO used for PWM might be spoiled / faulty on your board, then we will proceed to an exchange ;-) Will keep you updated this week.

[gprovost]

@NickS The conclusion is that the two GPIOs driving the PWM pins of each fan header are dead on your board. After spending quite some time investigating we are still puzzled on what could have created the issue, we still don't have a theory that could explain both occurrence. Anyhow you can PM me and we see how we can arrange a replacement of your board.

 

 

[NickS]

OLED panel

Here's my version of the sys-oled script with various panels for...

System Logo;

System Load, Fan & Temperature;

HDD activity;

Network activity;

Temperature & Clock

 

Anyone wishing to deploy it will need to change the location of their HDDs in status_hdd()

Also change location of  temperature sensor in cpu_temp() depending on your Kernel

 

Apologies for the crude code - Python is not my favourite!

oled.py

[gprovost]

@NickS Thanks for sharing. I will test your mod later.

 

However it would have been nice to fork the sys-oled project on github and put your modification there ;-)