Orange Pi zero - freezing randomly


Recommended Posts

Armbianmonitor:

Hello,

 

I have been struggling for some time with random freezes. I am running application with i2c0 Fram and UART1 for transferring/receiving data based on MAX3232. The freezes cause no response from Orange PI (debug UART too), only unplug power helps.
I read that it may be a problem with powering but i tried many USB power adapters 2A, 3A and ATX power (28A) connected to GND and 5v, nothing helps. I have lowered cpu max frequency to 816Mhz  and still the same.  All necessary data uploaded via armbianmonitor.

Link to post
Share on other sites
Armbian is a community driven open source project. Do you like to contribute your code?

I have some 20 Orange Pi Zero's around and I tried quite a few images, the last one was Devuan. I installed it two days ago and this morning I found it crashed. Anyway, I fear the the hardware is definitly unstable. I tried to limit the CPU freq, but no way. A reboot does not solve the issue, you really have to give it a power cycle (!) Some boards can run for months, other will crash after a few hours. I ordered some Raspberry Pi Zero-W now and I will replace these boards. And I have two Orange Pi PC boards (H3): same issue. Switching on the WiFi reduces the uptime a lot so I blacklisted the xr819 module on all boards. I just installed Armbian buster, just to see what happens before they end up in the trash bin. I definitively lost confiance in these boards. I have lost too much time with this crap. Sorry boys.

Link to post
Share on other sites

Oh, and freezing also means that some services still work. E.g. ssh still listens on tcp port 22 but no ssh connection possible. Time is sometimes weird, like 1937 or 2178. Sometimes nullmailer works as I receive messages that a certificate is out of date, due to a local time of 21 dec 1937 when the cert was not valid yet.

Link to post
Share on other sites

 

2 hours ago, Richard Lucassen said:

I tried quite a few images, the last one was Devuan


If Armbian doesn't work, then you can suspect this:
 

2 hours ago, Richard Lucassen said:

I fear the the hardware is definitly unstable.

Link to post
Share on other sites

Here, I have two GPIO OPiZero switches running Armbian. Same Armbian, same software, same hardware, wifi module blacklisted. One is rarely used and works fine. The other one is behaving weirdly. Fortunately this time I'm able to access the board by ssh (which is not always the case):

 

$ timeout 1s sleep 5
timeout: error while loading shared libraries: ld-lijux-armhb.so.3: cannot open shared object file: No such file or directory

 

The one that works:

 

$ timeout 1s sleep 5

(sleep just killed after 1 second)

 

Ok, to make it weirder:

The crashed one:

find /usr/ -name '*ld-lijux*'

find /lib/ -name '*ld-lijux*'
[nothing]

 

The working device:

find /usr/ -name '*ld-lijux*'

find /lib/ -name '*ld-lijux*'
[nothing]

 

Searching for the error message I stumble upon cross compiler issues. Anyway, another issue:

 

# apt update
Segmentation fault

 

Reboot: nothing changes, same issues (!!!) The uSD is ok. Already replaced the uSD twice. Power supply is ok (5.10 V) Only a power cycle resolves these issues (temporarely). Ok, it might be a production batch issue. But: I have two H3 Orange Pi PC's having the same weird behaviour. I also run some Raspberry Pi's: never a problem. And btw: switch on the WiFi chip and you can wait for it.

 

I just installed Armbian Buster and set the governor to "performance" as mentioned above, just to see what happens. And I suspect the hardware because a power cycle is needed. Bit rot. And I did not complain about Armbian! :-)

Link to post
Share on other sites
38 minutes ago, Richard Lucassen said:

Power supply is ok


Most of the troubles are caused by a bad PSU/cable combo.

 

How can you prove that its OK? I have around 30 mUSB cables at home and around 70% of them are broken - can only slow charge (low current) mobile phones / transfer data, but my PSU is O.K.

Our test rig reports https://dl.armbian.com/_test-reports/2020-05-31_17.23.38.html that most of the boards (1-3 are sometimes having issues) are surviving extreme stress testings. With wifi enabled and tested. 

 

I can't say its not a hardware problem. Possible. It can be a board quality issue, but before claiming that ... triple check PSU and cabling.

Link to post
Share on other sites
39 minutes ago, Richard Lucassen said:

Power supply is ok (5.10 V)

Do not measure the output voltage of your supply. That value is worth nothing. It does not include the resistance of the wire and the connectors and these values can add up badly.

 

Measure for example at the GPIO 5V and GND pins. You will notice quite some fluctuations when putting a load to the board.

Link to post
Share on other sites

I got a old but often updated installation on my OPi Zero which does run stable (not much used performance).

 

Do you got the same hardware-limits like

hardware limits: 480 MHz - 1.01 GHz
available frequency steps: 480 MHz, 648 MHz, 816 MHz, 960 MHz, 1.01 GHz

when doing a query with

cpufreq-info

?

Because most H2/H3 CPUs are unstable over 1008Mhz/1.01GHz

 

My stable OPi Zero has the following config:
System diagnosis information has been uploaded to http://ix.io/2obt

Link to post
Share on other sites

The 5.10V is applied directly to the board, not via USB. There is a PCB connected to the OPi Zero with a 5V/4A PSU on it. I know USB power supplies are crap. I got some usb chargers here that have as low as 4.65V as output voltage. I don't use these ones. But most important phenomenon: after a reboot the problems remain!

And that's quite weird, must be some bitrot somewhere. And almost all of these boards suffer from these problems. As I bought most of these boards all in one order (also the PC H3 version) must be some hardware glitch somewhere.

 

@ Guidol: just the default cpu frequencies. I never overclock a cpu. I wanted to insert this image with the "insert image from URL" button, but that does not work.

Link to post
Share on other sites
On 6/3/2020 at 3:09 PM, Richard Lucassen said:

Here, I have two GPIO OPiZero switches running Armbian. Same Armbian, same software, same hardware, wifi module blacklisted. One is rarely used and works fine. The other one is behaving weirdly. Fortunately this time I'm able to access the board by ssh (which is not always the case):

 

$ timeout 1s sleep 5
timeout: error while loading shared libraries: ld-lijux-armhb.so.3: cannot open shared object file: No such file or directory

 

The one that works:

 

$ timeout 1s sleep 5

(sleep just killed after 1 second)

 

Ok, to make it weirder:

The crashed one:

find /usr/ -name '*ld-lijux*'

find /lib/ -name '*ld-lijux*'
[nothing]

 

The working device:

find /usr/ -name '*ld-lijux*'

find /lib/ -name '*ld-lijux*'
[nothing]

 

Searching for the error message I stumble upon cross compiler issues. Anyway, another issue:

 

# apt update
Segmentation fault

 

Reboot: nothing changes, same issues (!!!) The uSD is ok. Already replaced the uSD twice. Power supply is ok (5.10 V) Only a power cycle resolves these issues (temporarely). Ok, it might be a production batch issue. But: I have two H3 Orange Pi PC's having the same weird behaviour. I also run some Raspberry Pi's: never a problem. And btw: switch on the WiFi chip and you can wait for it.

 

I just installed Armbian Buster and set the governor to "performance" as mentioned above, just to see what happens. And I suspect the hardware because a power cycle is needed. Bit rot. And I did not complain about Armbian! :-)

 

Try adding mem=512M to kernel parameters in boot.cmd/boot.scr, smells like defective memory.

Link to post
Share on other sites

I got no problems with kernel 5.9.x while using a good power-supply and a 80MB/Sec FAST SanDisk Ultra MicroSD-Card.

(Only 12.5 Days because we get here in Turkey from time to time a power-interruption)

While using other cards I also got freezes on my 2nd OPi Zero :(
 

  ___  ____  _   _____
 / _ \|  _ \(_) |__  /___ _ __ ___
| | | | |_) | |   / // _ \ '__/ _ \
| |_| |  __/| |  / /|  __/ | | (_) |
 \___/|_|   |_| /____\___|_|  \___/

Welcome to Debian GNU/Linux 10 (buster) with Linux 5.9.0-rc5-sunxi

System load:   0.00 0.00 0.00   Up time:       12 days 12:36
Memory usage:  11 % of 492MB    IP:            192.168.6.99
CPU temp:      73°C

 

Link to post
Share on other sites
14 minutes ago, guidol said:

I got no problems with kernel 5.9.x while using a good power-supply and a 80MB/Sec FAST SanDisk Ultra MicroSD-Card.

(Only 12.5 Days because we get here in Turkey from time to time a power-interruption)

While using other cards I also got freezes on my 2nd OPi Zero :(
 


  ___  ____  _   _____
 / _ \|  _ \(_) |__  /___ _ __ ___
| | | | |_) | |   / // _ \ '__/ _ \
| |_| |  __/| |  / /|  __/ | | (_) |
 \___/|_|   |_| /____\___|_|  \___/

Welcome to Debian GNU/Linux 10 (buster) with Linux 5.9.0-rc5-sunxi

System load:   0.00 0.00 0.00   Up time:       12 days 12:36
Memory usage:  11 % of 492MB    IP:            192.168.6.99
CPU temp:      73°C

 

I'm running the legacy armbian on an orange pi pc for CCTV. Four IP + Zoneminder cameras with the CPU reaching 75º without a cooler. The system is rock solid. Migrating to the most current version of armbian, I couldn't keep him alive for more than 2 days. Constant freezes. The funny thing is that in legacy armbian, with the same hardware configurations the system works wonderfully. I love the legacy Armbian. For me it works spectacularly.

Link to post
Share on other sites

My Orange Pi Zero (256MB RAM) is also instable with recent kernels.

 

I have tried:

 

Armbian 20.08.3 with kernel 5.4.63 - custom built image: unexpected SSH disconnects, freezes (only ping works, but no ssh login), random problems with "apt install" - system freezes. "echo 0 > /sys/devices/system/cpu/cpu2/online"  leads to kernel dumps or SSH freezes.

Most recent custom armbian build with kernel 5.8.x: same as above

Armbian 19.11.6 with kernel 5.4.8 - official image: same problems as above but putting cpu2 offline is working.

 

Armbian 5.91 with kernel 4.19.59 - official image: everything working as expected.

 

What I can eliminate as a source for these troubles is the SD card. I tried multiple SD cards (same SanDisk Ultra 16GB A1 Model).

In my opinion there is something wrong with the kernel. The temperature readings (if correct) looked normal when installing packages ~50-60 degrees.

 

Something in the armbian image broke after kernel 4.19.59.

 

I am using a Xiaomi Mi5 Charger (rated for 2A) as power source, but I also have a programmable lab power supply available. I could try it with the lab power supply too but haven't done so. 

Link to post
Share on other sites
23 minutes ago, maracuja said:

My Orange Pi Zero (256MB RAM) is also instable with recent kernels.

 

I have tried:

 

Armbian 20.08.3 with kernel 5.4.63 - custom built image: unexpected SSH disconnects, freezes (only ping works, but no ssh login), random problems with "apt install" - system freezes. "echo 0 > /sys/devices/system/cpu/cpu2/online"  leads to kernel dumps or SSH freezes.

Most recent custom armbian build with kernel 5.8.x: same as above

Armbian 19.11.6 with kernel 5.4.8 - official image: same problems as above but putting cpu2 offline is working.

 

Armbian 5.91 with kernel 4.19.59 - official image: everything working as expected.

 

What I can eliminate as a source for these troubles is the SD card. I tried multiple SD cards (same SanDisk Ultra 16GB A1 Model).

In my opinion there is something wrong with the kernel. The temperature readings (if correct) looked normal when installing packages ~50-60 degrees.

 

Something in the armbian image broke after kernel 4.19.59.

 

I am using a Xiaomi Mi5 Charger (rated for 2A) as power source, but I also have a programmable lab power supply available. I could try it with the lab power supply too but haven't done so. 

I agree.

Link to post
Share on other sites

Updated my OPi Zero (512MB) to Armbian 20.08.9 Buster with Linux 5.9.0-rc7-sunxi

 

I got no problems with time/date, because armbian does use chronyd for ntp

Is chrony already as ntp-standard included in 4.19.59?

 

And I did read some/most(?) have the 256MB Ram version versus me using the 512MB Ram version.

My OPi Zero is using the USB-extension + black cube-case from OPi - connected via Ethernet and not WiFi. 

 

BUT armbianmonitor -u doesnt give me a URL at this time:
 

root@opi-zero(192.168.6.99):~# armbianmonitor -u
System diagnosis information will now be uploaded to
Please post the URL in the forum where you've been asked for.

 

Link to post
Share on other sites
33 minutes ago, guidol said:

 

BUT armbianmonitor -u doesnt give me a URL at this time:
 


root@opi-zero(192.168.6.99):~# armbianmonitor -u
System diagnosis information will now be uploaded to
Please post the URL in the forum where you've been asked for.

 

Try -U and pipe it into a file. Maybe it is just too big?

Link to post
Share on other sites

The causes for the system freezes are most probably NOT:

the expansion board (I have none in use)

the WiFi module (I have blacklisted xradio_wlan and use an ethernet connection)

 

The causes could have to do with:
*) the kernel

*) system load / temperature / current draw / cpu frequency

Link to post
Share on other sites

For what it's worth, I'm not seeing any issues on a 256MB OrangePi R1 running 5.8.14-current (I only have a 512MB Pi Zero).  It works without issue - e.g.,

 

root@orangepi-r1:~# uname -a
Linux orangepi-r1 5.8.14-sunxi #trunk SMP Sun Oct 11 14:48:02 UTC 2020 armv7l GNU/Linux
root@orangepi-r1:~# cpufreq-info -c 1
cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009
Report errors and bugs to cpufreq@vger.kernel.org, please.
analyzing CPU 1:
  driver: cpufreq-dt
  CPUs which run at the same hardware frequency: 0 1 2 3
  CPUs which need to have their frequency coordinated by software: 0 1 2 3
  maximum transition latency: 5.44 ms.
  hardware limits: 480 MHz - 1.01 GHz
  available frequency steps: 480 MHz, 648 MHz, 816 MHz, 960 MHz, 1.01 GHz
  available cpufreq governors: conservative, userspace, powersave, ondemand, performance, schedutil
  current policy: frequency should be within 480 MHz and 1.01 GHz.
                  The governor "ondemand" may decide which speed to use
                  within this range.
  current CPU frequency is 480 MHz (asserted by call to hardware).
  cpufreq stats: 480 MHz:86.78%, 648 MHz:0.30%, 816 MHz:0.18%, 960 MHz:0.06%, 1.01 GHz:12.69%  (193)
root@orangepi-r1:~# free
              total        used        free      shared  buff/cache   available
Mem:         243992       78052       79360        1936       86580      155792
Swap:        121992           0      121992
root@orangepi-r1:~# echo 0 > /sys/devices/system/cpu/cpu2/online
root@orangepi-r1:~# apt-get update
Hit:1 http://deb.debian.org/debian buster InRelease
Hit:2 http://deb.debian.org/debian buster-updates InRelease
Hit:3 http://deb.debian.org/debian buster-backports InRelease
Hit:4 http://security.debian.org buster/updates InRelease
Hit:5 https://armbian.systemonachip.net/apt buster InRelease
Reading package lists... Done
root@orangepi-r1:~# wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.8.14.tar.xz
--2020-10-11 15:52:46--  https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.8.14.tar.xz
Resolving cdn.kernel.org (cdn.kernel.org)... 2a04:4e42:600::432, 2a04:4e42::432, 2a04:4e42:200::432, ...
Connecting to cdn.kernel.org (cdn.kernel.org)|2a04:4e42:600::432|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 114506716 (109M) [application/x-xz]
Saving to: ‘linux-5.8.14.tar.xz’

linux-5.8.14.tar.xz 100%[===================>] 109.20M  3.19MB/s    in 30s

2020-10-11 15:53:17 (3.68 MB/s) - ‘linux-5.8.14.tar.xz’ saved [114506716/114506716]

root@orangepi-r1:~# md5sum linux-5.8.14.tar.xz
b4ed25c9a13f2ac90405b74da26e31d6  linux-5.8.14.tar.xz
root@orangepi-r1:~# cat /sys/class/thermal/thermal_zone0/temp
60297
root@orangepi-r1:~#

 

@maracuja Might it be possible to hook a serial console to the board, and then capture the output when your board freezes/crashes?  (It might also be interesting to try using Wi-Fi instead of Ethernet to eliminate that vector as a possibility?)

 

Link to post
Share on other sites

@5kft

It is possible to hook a serial console to the board, but what kind of outcome do you expect?

I can even reproduce the freezing - all I have to do is to run a script that has some "apt install ..." calls with custom deb packages that I've built. Today it got stuck when configuring php7.3-cli (a dependency) for example.

Then it said "process killed". The very same script runs without error on kernel 4.9.

Link to post
Share on other sites
2 hours ago, Werner said:

Try -U and pipe it into a file. Maybe it is just too big?

Is that too big? around 1.5MB unzipped

 

root@opi-zero(192.168.6.99):~# armbianmonitor -U > /home/guido/armbianmonitor.txt
root@opi-zero(192.168.6.99):~# ls -l /home/guido/armbianmonitor.txt
-rw-r--r-- 1 root root 1547948 Oct 11 19:49 /home/guido/armbianmonitor.txt

root@opi-zero(192.168.6.99):~# gzip /home/guido/armbianmonitor.txt
root@opi-zero(192.168.6.99):~# ls -l /home/guido/armbianmonitor.*
-rw-r--r-- 1 root root 500078 Oct 11 19:49 /home/guido/armbianmonitor.txt.gz

 

armbianmonitor.txt.gz

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...