Jump to content

Cubietruck Ubuntu 14.04 crashes after 1-3 days


joaofl

Recommended Posts

Hi.

 

Can anybody give me any idea on how to find out why it is crashing?

 

I recently moved to this distro, and it is now crashing after 1 to 4 days of continuous usage. The thing is, I have no idea how to find out why it is happening, and how to fix it. Which log should I go for?

 

I was running this Aruntu before, and never had this issue, which seems to show that this is not an hardware issue.

 

Best,

 

João

Link to comment
Share on other sites

Some releases had stability issues. Which build do you have?

 

Check: /etc/armbian.txt or /root/readme.txt 

 

Last one is stable.

From /etc/armbian.txt:

 

Title:                  Armbian 4.2 Cubietruck Ubuntu trusty default (unofficial)

Kernel:                 Linux 3.4.108

Build date:             26.08.2015

 

I think it is the last one. I use to have instability issues before too.

 

Is ubuntu worst in any way compared to the debian distro? Should I upgrade the kernel?

 

How can I figure out why, and maybe help on that?

 

Thanks a lot.

 

Link to comment
Share on other sites

Have you come here with upgrades or you started with clean image?

Stability issues are in most cases related to U-boot and board configuration. Board conf hasn't been changed for a while, uboot is changing constantly. There was the same issue with R1 not long time ago.

 

Kernel and distribution doesn't play any role here.

Link to comment
Share on other sites

Have you come here with upgrades or you started with clean image?

 

Stability issues are in most cases related to U-boot and board configuration. Board conf hasn't been changed for a while, uboot is changing constantly. There was the same issue with R1 not long time ago.

 

Kernel and distribution doesn't play any role here.

 

Ok, it makes sense. I did the apt-get upgrades available.

 

What would u suggest me to try? Or should I go back to the orignal image?

 

Thanks again.

Link to comment
Share on other sites

Just upgrade the u-boot. I assume you already add my repository? (howto: http://forum.armbian.com/index.php/topic/211-kernel-update-procedure-has-been-changed/)

apt-get install linux-u-boot-cubietruck
reboot

I think this is all you need to do and you should be fine.

Thanks Igor.

 

I did that, in a brand new image. Worked out for since then, but have just crashed.  I'm running a demanding software there, but it does not hurt even weaker boards I have.

 

What could it be?

 

Cheers.

Link to comment
Share on other sites

Tell me more:

 

- What kind of power supply are you using and where it is attached to? Power connector or host USB.

- do you have any additional hardware attached to USB?

- have you installed system to NAND?

- have you modify anything else?

- what is the exact build and build date of Uboot.

- rather power cycle then reboot

Link to comment
Share on other sites

On your text:

 

Tell me more:

 

- What kind of power supply are you using and where it is attached to? Power connector or host USB.

 

Power supply is connected to the Jack. A good one, 3Amps.

 

- do you have any additional hardware attached to USB?

 

No. I have a sata disk, connected externally to another output of the same power supply (like a PC power supply).

 

- have you installed system to NAND?

 

No, it is on the SSD card so far.

 

- have you modify anything else?

 

No modifications, except for installing some new apps, and changing the GUI boot to my "user" instead of root.

 

- what is the exact build and build date of Uboot.

 

It is the last one available at the repo. Not sure of the version number, but ill check it out.

 

- rather power cycle then reboot

 

should also try it out when I'm back home.

 

Thanks again.

Link to comment
Share on other sites

Same thing happens to me. kernel crashes, there is nothing in log files.

 

- What kind of power supply are you using and where it is attached to? Power connector or host USB.

true 3A, connected to power jack.

 

- do you have any additional hardware attached to USB?

NO. well yes, now I do, it's a KVM switch. but it doesn't matter if switch is in USB not. kernel crashes without that switch attached to USB. I actually attached KVM switch just to try to switch to cubietruck console in case if it stops responding over the network.

 

- have you installed system to NAND?

no, it boots a kernel from SD-card, then rest of it from SATA disk.

 

- have you modify anything else?

just copied system to SATA disk and installed some additional packages from Debian repositories. Not much actually – postfix, dovecot, apache, mysql, transmission, some old FIDO–net packages, vim, mc, some small libs and utils.

 

- what is the exact build and build date of Uboot.

last one from you repo.

 

- rather power cycle then reboot

been there done that. doesn't help.

 

What can I also add is that I had a setup using Igor's early images (about a year and a half old Wheezy, v1.8, with 3.4.79 kernel) and system uptime was almost a year. Now with latest images it stops every 4–5 days in best case, sometimes it runs for 1 or 2 days only.

Link to comment
Share on other sites

I'll try to recreate. The most problematic thing here is probably u-boot ... perhaps host compiler?

 

If you have chance to experiment too, try 2015.04 or older. I don't recall how far back is safe to go - we need legacy boot support - that we can boot old kernel too.

Link to comment
Share on other sites

It seems, that stability issues are indeed directly related to the version/assembly of u-boot. I've built complete image of Ubuntu 14.04.03 LTS on 11.11.2015 which identifies itself as:

 

Title:                  Armbian 4.6 Cubietruck Ubuntu trusty default (unofficial)
Kernel:                 Linux 3.4.110
 

It was quite unstable and could be easily made to hang completely by running "memtester 1500'. Then u-boot was downgraded to 4.5 via apt-get while keeping kernel the same. After that, the board become stable and survived several days under the load of memtester.

 

There seems to be an additional issue of stability with 2.5" hdd connected, but it manifests itself even with original Cubian image. Preliminary fix for it includes soldering of 220uF electrolytic capacitor directly to the pins of DC-IN jack. In both cases, board was powered via "official" 12to5V DC-DC converter.

Link to comment
Share on other sites

I've been having these sorts of problems on my Cubieboard2 running Debian Jessie on the sunxi kernel in the past few months also. The older wheezy images were stable. I'm not sure when the issue started. I won't have physical access to the machine for a couple days, so I can't give very detailed information at the moment, but I will report back then.

 

I'd like to move to the mainline kernel but I am waiting for audio support. I believe this will be in 4.4 based on the sunxi wiki [1] although I haven't noticed it on the RC kernel changelogs published by Torvalds.

 

What are the major changes between Armbian releases? Is there a downgrade process or do I have to reinstall the OS?

 

What is the full list of information I can give that may help diagnose the issue? For example, should anything be added to this list?

Kernel version

Contents of /etc/armbian.txt

u-boot build date? How can I check that? Should I provide a hash of something?

Description of power supply

Connected peripherals

Results of a memtest? What provides this on ARM?

Anything else? I'm willing to spend some time running tests and providing information.

 

[1]

http://linux-sunxi.org/Linux_mainlining_effort#Merged_into_4.4

Link to comment
Share on other sites

 

 

I'd like to move to the mainline kernel but I am waiting for audio support. I believe this will be in 4.4 based on the sunxi wiki [1] although I haven't noticed it on the RC kernel changelogs published by Torvalds.

 

Audio (analog only) is already merged into mainline 4.4 RC1 kernel, you can compile it yourself by setting KERNELTAG="v4.4-rc1" in build.sh and enabling certain kernel options. Edit: and fixing packaging process.

It is enabled in Device Tree for cubieboard2 already.

 

Edit:

If it's a kernel crash, then the most useful info will be log from UART console (with additional kernel parameters), you will need to set up logging to file (i.e. using Putty) on another board/PC/something else.

U-boot also will print its version to UART console.

Description of PSU and peripherials, list of loaded modules (lsmod) can also be useful.

 

Also you can try to enable hardware watchdog which should reboot board in case it hangs, and periodically check board uptime.

Link to comment
Share on other sites

The hardware watchdog works in tests, but in my cases never in real crashes.

Did you set nowayout=1 for watchdog? If it doesn't restart system with this parameter, it may be some hardware/power supply issues.

If setting performance governor helps, then it may be the correct solution/workaround for this problem as long as it doesn't cause overheating.

Link to comment
Share on other sites

Hi guys, I have stability issues matching title. System kernel is 3.4.110-sun7i and I am running root on 8GB microSD, mythtv media folders on 1TB hdd attached to SATA port and 2 usb tuners connected to each port. Board is powered via DC-DC converter and everytime when I checked Rpi-Monitor graphs after crash I did not see nothing weird with voltage or higher current. Cpu load is stable about 2 all time (caused by mysql, mythtv-backend or boinc-client processes). Board / drive temperatures are 63.4°C / 49°C

now.

 

After about 1 day I could not see it on network or sometimes I cannot log in using ssh client while I could browse web pages on still running apache2 server or I could watch recording from SATA drive. Power on/off did not help. 

 

It is about 23h ago when I changed these setting to keep cpu more stable.

 

I edited /boot/bin/cubietruck.bin to keep cpu on one frequency all time and this is dmesg output after reboot:

 

[    1.932372] [cpu_freq] INF:-------------------V-F Table-------------------
[    1.937421] [cpu_freq] INF:  voltage = 1450mv        frequency =  912MHz
[    1.942474] [cpu_freq] INF:  voltage = 1425mv        frequency =  864MHz
[    1.947519] [cpu_freq] INF:  voltage = 1350mv        frequency =  720MHz
[    1.952579] [cpu_freq] INF:  voltage = 1250mv        frequency =  528MHz
[    1.957624] [cpu_freq] INF:  voltage = 1150mv        frequency =  312MHz
[    1.962681] [cpu_freq] INF:  voltage = 1100mv        frequency =  144MHz
[    1.967726] [cpu_freq] INF:  voltage = 1000mv        frequency =    0MHz
[    1.973560] [cpu_freq] INF:-----------------------------------------------
[    1.983297] [cpu_freq] INF:sunxi_cpufreq_initcall, get cpu frequency from sysconfig, max freq: 912MHz, min freq: 912MHz
 

... and next edit in file /etc/init.d/cpufrequtils

 

ENABLE="true"
GOVERNOR="performance"
MAX_SPEED="912000"
MIN_SPEED="912000"

 

Next step is solder a capacitor to power connector (it can be good idea, posted by bxm) but I do not want to do it now during Christmas time.

But if somebody thinks this is a software problem I am ready to hear you ;-).

Link to comment
Share on other sites

Uptime: 2 days 08 hours 49 minutes 26 seconds

It seems higher cpu voltage and locked cpu frequency are working... This is longest uptime from beginning of December when I have started using new ubuntu image on my cubietruck. If somebody knows what voltages are available I can test higher frequencies ;).

Link to comment
Share on other sites

BTW: Instead of trial&error you could also have a look whether the dvfs settings are stable: http://linux-sunxi.org/Hardware_Reliability_Tests#Reliability_of_cpufreq_voltage.2Ffrequency_settings

 

Then you seem to confuse Vcore voltage (the voltage with which the CPU cores are being fed by the AXP209 PMU) with powering the board itself. It's possible to overvolt the CPU while running in undervoltage situations (voltage dropping below 4.6V from DC-IN).

 

On the cubietruck you should also be aware that a disk is not powered through the PMU but directly from DC-IN. And if you just use a 2.5" disk I fail to understand why you use the 12V/5V DC-DC converter?

 

BTW: What do you refer to regarding 'board temperature'? Is this PMU or SoC? Anyway: 63.4°C is way too high: http://forum.lemaker.org/forum.php?mod=redirect&goto=findpost&ptid=7937&pid=37527&fromuid=33332

 

http://linux-sunxi.org/AXP209#Overview

 

AXP209 integrated over / under voltage (OVP / UVP), over temperature (OTP), overcurrent protection (OCP) circuit.
Link to comment
Share on other sites

Thank you for "Hardware Reliability Tests" link. It can help me do stability tests now and in a future.

 

Yes, I took voltages for lower frequencies, but at the cost of loosing higher frequency, coz I do not know what voltage should I choose or what is more important if choose smthing this voltage is available (I have no time to do blind test).

 

I am using 12V power adapter connected to DC/DC converter and output 5V is connected to Cubietruck. I am not using DC/DC converter for 3.5" sized disks which you can buy separately for cubietruck if you mean this. I have decided for an 12V adapter + DC/DC converter because it works more reliable than 99% of 5V adapters on market even label on the top says 4A. (If you found a good power adapter probably I am not a lucky devil).

 

I checked sys tree and I have only 2 temp here. One should be PMU and second is for battery (not connected). But system is on 100% load 24h/day because boinc-client is running on background.

Link to comment
Share on other sites

Thank you for "Hardware Reliability Tests" link. It can help me do stability tests now and in a future. [...]

 

I checked sys tree and I have only 2 temp here. One should be PMU and second is for battery (not connected). But system is on 100% load 24h/day because boinc-client is running on background.

 

You should immediately run cpuburn-a7 and the cpufreq-ljt-stress-test in parallel to sort undervoltage situations out. But since the PMU is getting that hot you might also run in overvoltage/overtemperature situations (the more Vcore you provide the hotter the PMU gets). 

 

BTW: If you use my enhanced RPi-Monitor version available here http://forum.armbian.com/index.php/topic/155-testers-wanted-sunxi-adjustments-for-rpi-monitor/you can also read out the SoC's temperature with the included  sunxi_tp_temp binary. By installing the sunxi additions everything should work automagically.

Link to comment
Share on other sites

I returned back from work and could not access cubietruck from network again => reset => little check of Rpi-Monitor, voltage had been stable, very stable before it stopped working, straight line 5V and current between ranges like usually.

I do not know what is happening on the background but I must I think move /var from tmpfs back to sdcard and lower commit of root fs to keep logs.

And because I received pair of Banana Pro is time to replace this non reliable device. Temps of soc and pmu temps are higher than should be but my BananaPi was able working on higher temperatures reliable and stable.

 

I going to try vanilla kernel and wait if it not helps next step will be adding fan on soc and if it will not help again I will add capacitor on power input.

Link to comment
Share on other sites

Update:

 

Runtime is 1day and 19hours now.

I changed/optimized:

1) mysql - I used mysqltuner.pl script and it showed me which settings are good to change to keep memory usage low VERY IMPORTANT !!! while performance is on good level.

I added 4GB swap to hdd but it can solve every high memory demand cases which is mysql indeed.

2) cpu frequency - When I tried run cubietruck on 1008Mhz mythbackend send back invalid pointer followed with hexa value very quickly. Because this error reminded me times when I was doing overclocking of desktop computer cpu's I set maximum cpu frequency to lower value 864Mhz in script.bin and set both low and high values in /etc/init.d/cpufrequtils daemon in same values like in that script. (I left system starting on 720Mhz value like in original config)

3) I set higher voltage on my buck converter. RPi-Monitor shows average value 5,1V but sometimes it drops lower than 4,9V. Sometimes I see lower values on front page than in graph (I added it there for quick check). Difference is only between 0,1-0,2V and it shows some hardware related problems in power source design. It can be caused by me or cubietruck designers. When 7200rm hardrive is seeking even it is 2,5" in combination with changes in cpu load and usb devices load make sense why voltage is dropping sometimes.

 

Another problems/issues:

1) I see battery charging spikes even I have already unplugged battery. Problem si in RPi-Monitor because it shows battery voltage 2.677V now and it should be 0V but it reads value from sys. No, problem is in kernel than, in PMU driver.

Link to comment
Share on other sites

Another problems/issues:

1) I see battery charging spikes even I have already unplugged battery. Problem si in RPi-Monitor because it shows battery voltage 2.677V now and it should be 0V but it reads value from sys. No, problem is in kernel than, in PMU driver.

Battery detection is broken in legacy kernel, so this is normal

 

2) cpu frequency - When I tried run cubietruck on 1008Mhz mythbackend send back invalid pointer followed with hexa value very quickly. Because this error reminded me times when I was doing overclocking of desktop computer cpu's I set maximum cpu frequency to lower value 864Mhz in script.bin and set both low and high values in /etc/init.d/cpufrequtils daemon in same values like in that script. (I left system starting on 720Mhz value like in original config)

A20 on 1008MHz can be unstable, this frequency is even removed from mainline kernel; setting max frequency to 960MHz should be safe.

 

3) I set higher voltage on my buck converter. RPi-Monitor shows average value 5,1V but sometimes it drops lower than 4,9V. Sometimes I see lower values on front page than in graph (I added it there for quick check). Difference is only between 0,1-0,2V and it shows some hardware related problems in power source design. It can be caused by me or cubietruck designers. When 7200rm hardrive is seeking even it is 2,5" in combination with changes in cpu load and usb devices load make sense why voltage is dropping sometimes.

PMU input range is 3.8V - 6.3V, so input voltage shouldn't be a problem. Voltage drop can be expected with long power cables and relatively cheap power supplies.

Link to comment
Share on other sites

Battery detection is broken in legacy kernel, so this is normal

 

A20 on 1008MHz can be unstable, this frequency is even removed from mainline kernel; setting max frequency to 960MHz should be safe.

 

PMU input range is 3.8V - 6.3V, so input voltage shouldn't be a problem. Voltage drop can be expected with long power cables and relatively cheap power supplies.

I changed cubietruck.bin file to start otg usb port working for normal devices. Swapped one of 2 usb tuners to this port and added usb voltage meter to last usb free hole.

I expected usb voltage is on same level like input voltage or very little lower due board paths and usb <-> cable contact resistence and cables resistence but it is not true. (It is good thing to read board schema on begining sometimes). And USB voltage has started showing in RPi-Monitor. Even I raised input board voltage to 5,3V usb-otg voltage is 4.864V in Rpi-monitor and 4.94V on usb multimeter. And second thing is I discovered USB voltage is raising up when input voltage of cubietruck board is raising. Difference is about 0.4V so I can say usb voltage was lower than minimal allowed value for usb devices and therefore It colud send system down from this reason. I forget add when I unplugged usb tuner it set kernel in error and I must reboot cubietruck.

Link to comment
Share on other sites

Guest
This topic is now closed to further replies.
×
×
  • Create New...

Important Information

Terms of Use - Privacy Policy - Guidelines