Jessie ans Wheeze on cubietruck

[FanM]

Hi friends!

 

Here is the situation. Trying the armbian jessie 3.4.109 and 4.2.3. When installing a new soft it takes to much time.

For example, when running apt-get upgrade after fresh image install, it takes about 17 minutes.

Of course it contain the core update. But that upgrade part on jessie takes about 7min, on wheezy about 3min.

The same thing when just install some software, like apache2 or mc. It takes to much time (for mc it takes about 1 min on jessie, about 30sec on wheezy). The same with reloading. It takes about 1min.

Situation is the same on both jessie 3.4.109 and 4.2.3

Tried the same with Banana PI. The same thing.

 

I use the fastest SD card.

 

What is your expirience? Any comments?

[FanM]

Meanwhile i was trying to find some information on that thing. Found some info about syslog and systemd. Is there any connection between my problem and systemd?

Does the systemd writes the log on SD card, but syslog in cache memory?

[Igor]

17 min is way too much. Some speed decrease could happen if you had terrible bad luck and httpredir choose some mirror that is broken or our servers very in trouble ... but they are working fine.

 

Fastest card. Can you measure r/w speed within Linux.

 

What is your download speed? How many files are downloaded. Can you post a upgrade log?

[FanM]

Igor, hi!

 

R speed is more than 11.5 (can not find faster destination storage) , W speed is 11.5mb/s.

 

I wrote an image jessie 4.2.3. Ran installing mc, ntpdate, and apt-get upgrade

Downloading from repository is not a problem, it is fast process everytime. 

The main thing is that unpacking takes a lot of time.

As you can see in log, today that process took 35min. Incredable long time.

 

Preparing to unpack .../linux-headers-next-sunxi_4.81_armhf.deb ...

Unpacking linux-headers-next-sunxi (4.81) over (4.5) ...

That took about 10 minutes!!!

 

That is all kinda strange. For example, i downloaded a file with size 11M that contain 2519 files. So unpacking that file needs only 5 seconds.

 

Do not know what to do with it.

 

17 min is way too much. Some speed decrease could happen if you had terrible bad luck and httpredir choose some mirror that is broken or our servers very in trouble ... but they are working fine.

 

Fastest card. Can you measure r/w speed within Linux.

 

What is your download speed? How many files are downloaded. Can you post a upgrade log?

apt.tar.gz

[Igor]

Kernel upgrade can take 10+ minutes, that's somehow normal. It's a lot of files ... there are modules and headers, so it take some time.

[FanM]

Hmmm, yesturday you told me that 17min is to much for upgrade. But today that operation tooks 35min. and you telling me that it is normal and in fact needs 10+ minutes. I am kinda confused

[tkaiser]

So unpacking that file needs only 5 seconds.

 

Can you try it again, this time the unpack command followed by " && sync"?

[zador.blood.stained]

@FanM

Also try reinstalling kernel headers package (as it has many small files) and while it's installing, take a look at output of "top" command in another tty/SSH session.

Values that should tell something:

• load average (end of first row)
• %cpu (3rd row)
• top processes and their CPU usage

Post output here in code or spoiler block (1 "page" should be enough).

Also check (and post here) output of "cpufreq-info" in case there are issues with cpufreq settings.

 

Edit:

Does the systemd writes the log on SD card, but syslog in cache memory?

journald on Jessie by default should use only volatile storage (/run/systemd/journal), logs are written on SD card by rsyslog, which reads it from journald. It's a little CPU overhead, but it shouldn't generate extra disk writes.

[tkaiser]

@FanM

Also try reinstalling kernel headers package (as it has many small files) and while it's installing, take a look at output of "top" command in another tty/SSH session.

 

I would suspect we're talking here about a horribly slow SD card. I would better go with 'apt-get install sysstat' and then let an 'iostat 5' run and watch %iowait. You might use '--force-unsafe-io option' with dpkg since it has been adjusted to use synchronous I/O a few years ago. But I would better check the SD card first (using f3 for example or checking both sequential and random I/O)

[zador.blood.stained]

@tkaiser

It might be, but it's quick enough to exclude from possible reasons extra CPU usage or wrong cpufreq settings. And "top" outputs %wait too.

from "man top"

wa, IO-wait : time waiting for I/O completion

[FanM]

That info took when installing gdebi software. Totally it itakes 11.5min for installing.

I get top info twice when it was unpacking deb packets.

 

 

top - 19:26:21 up 38 min, 2 users, load average: 1.16, 0.82, 0.68
Tasks: 110 total, 2 running, 108 sleeping, 0 stopped, 0 zombie
%Cpu(s): 5.9 us, 9.8 sy, 0.0 ni, 42.0 id, 42.2 wa, 0.0 hi, 0.0 si, 0.0 st

top - 19:27:52 up 39 min, 2 users, load average: 1.45, 1.00, 0.76
Tasks: 110 total, 1 running, 109 sleeping, 0 stopped, 0 zombie
%Cpu(s): 11.5 us, 8.4 sy, 0.0 ni, 39.3 id, 40.7 wa, 0.0 hi, 0.0 si, 0.0 st

 

   84 root        20   0       0      0      0 S  20.4  0.0   2:28.70 mmcqd/0

 7054 root      20   0   11732   8616   1120 S  14.8  0.4   0:03.78 dpkg-deb

 7064 root      20   0    9084   4152   3428 S   8.6  0.2   0:00.98 sshd

 5849 root      20   0   32132  27564  23636 S   7.9  1.3   0:39.25 apt-get

 5974 root      20   0   13856  11636   1824 D   3.0  0.6   0:20.79 dpkg

 

root@cubietruck:/etc/apt# cpufreq-info
cpufrequtils 008: cpufreq-info © Dominik Brodowski 2004-2009
Report errors and bugs to cpufreq@vger.kernel.org, please.
analyzing CPU 0:
driver: cpufreq-dt
CPUs which run at the same hardware frequency: 0 1
CPUs which need to have their frequency coordinated by software: 0 1
maximum transition latency: 244 us.
hardware limits: 144 MHz - 960 MHz
available frequency steps: 144 MHz, 312 MHz, 528 MHz, 720 MHz, 864 MHz, 912 MHz, 960 MHz
available cpufreq governors: conservative, userspace, powersave, ondemand, performance
current policy: frequency should be within 480 MHz and 960 MHz.
The governor "ondemand" may decide which speed to use
within this range.
current CPU frequency is 528 MHz (asserted by call to hardware).
cpufreq stats: 144 MHz:0.42%, 312 MHz:0.14%, 528 MHz:89.03%, 720 MHz:4.92%, 864 MHz:0.83%, 912 MHz:0.31%, 960 MHz:4.34% (555)
analyzing CPU 1:
driver: cpufreq-dt
CPUs which run at the same hardware frequency: 0 1
CPUs which need to have their frequency coordinated by software: 0 1
maximum transition latency: 244 us.
hardware limits: 144 MHz - 960 MHz
available frequency steps: 144 MHz, 312 MHz, 528 MHz, 720 MHz, 864 MHz, 912 MHz, 960 MHz
available cpufreq governors: conservative, userspace, powersave, ondemand, performance
current policy: frequency should be within 480 MHz and 960 MHz.
The governor "ondemand" may decide which speed to use
within this range.
current CPU frequency is 528 MHz (asserted by call to hardware).
cpufreq stats: 144 MHz:0.42%, 312 MHz:0.14%, 528 MHz:89.03%, 720 MHz:4.92%, 864 MHz:0.83%, 912 MHz:0.31%, 960 MHz:4.34% (555)

 

[zador.blood.stained]

Yep, 40.7% iowait. mmcqd in top.

Now please check your sd card as @tkaiser advised you.

 

Difference between jessie and wheezy may be caused by rootfs mount options.

[tkaiser]

It might be, but it's quick enough to exclude from possible reasons extra CPU usage or wrong cpufreq settings. And "top" outputs %wait too.

 

Yes, but the iostat version Linux/Armbian is using fortunately provides /proc/cpustat informations too (does not apply to iostat on other OS or even Linux distros): http://pastebin.com/r90mEWFM

 

And what I especially like at iostat is the ability to see 'obvious' limitations (eg. a crappy random I/O implementation on a SD card that shows high sequential transfer speeds) since it not only lists 'throughput' in KB/s but also tps (transactions per second). I would suspect that @FanM will experience fixed and rather low values for both regarding writes.

 

BTW: I had also one TF card that started (thermal?) throttling when being used intensively after approx. 90 seconds. Short tests with my MacBook showed always above 80 MB/s but after a while the thing got terribly slow. That's just another good reason to check each and every flash based media (be it an SD card or an USB thumb drive) at least one time prior to using it. 

[tkaiser]

available cpufreq governors: conservative, userspace, powersave, ondemand, performance

 

This looks like mainline kernel (since 'interactive' disappeared). While I'm still convinced that your SD card is to blame for the slow speed (dpkg enforces synchronous I/O which means all changes are always written immediately to disk -- something that normally does NOT happen since write buffer/cache jumps in and the kernel writes stuff to disk later and re-uses file contents available in the cache/RAM when needed) the following I/O related settings might be worth a look: http://linux-sunxi.org/Cpufreq#The_.22ondemand.22_governor

 

Since I still don't like jessie that much I've no idea how to enforce the io_is_busy setting for example... (this one being responsible for increasing cpufreq with %iowait)

[zador.blood.stained]

@FanM says that on Wheezy performance is subjectively better, so if card is OK, it's possible that something else is reading or writing to it, especially small reads or writes that causes excessive ext4 journalling (guess "iotop" can help with it)

Since I still don't like jessie that much I've no idea how to enforce the io_is_busy setting for example... (this one being responsible for increasing cpufreq with %iowait)

What's the difference with Wheezy here? You still can put commands in /etc/rc.local, you can put this setting in /etc/sysfs.conf, you can write init.d script, systemd unit or udev rule to do it too.

[tkaiser]

@FanM says that on Wheezy performance is subjectively better

 

Yeah, but I don't trust in subjectivity at all. It's always based on a 'sample size' too small and there are so many other factors influencing real or subjective performance that it's not worth to trust in. And based on my experiences (part of our job is to solve IT problems at large scale) often reality will be adjusted by expectations (unknowingly of course! I learned a lot when I bought and read Blink the last time I was in Naples although I missed the beauties of the landscape since I read through the whole book on our train ride).

 

Thx for pointing out the unchanged role of /etc/rc.local in Jessie (I really didn't know it and thought one had to create a service or any other inconvenient mechanism )

[Igor]

Hmmm, yesturday you told me that 17min is to much for upgrade. But today that operation tooks 35min. and you telling me that it is normal and in fact needs 10+ minutes. I am kinda confused

 

I read the post too quickly and haven't realized that you were upgrading the kernel too and ...

 

The issue here is, that it has been months since last images build and we are only updating since. Before, we never had this much old images so upgrade time might become a problem. Still, this is completely normal on a PC or any other distro! First you install, than you issue update which can take hours ... if HW is on the level of Cubietruck.

[FanM]

Friends, hi again!

 

First of all, thanks to everyone for help. Special thanks for f3 and sysstat stuff.

 

Here is my strange and suspicious results.

I have tested that image for next cards this time:

2Gb not SDHC, 4Gb SDHC class 2, 8Gb SDHC class 10, 16Gb class 10 UHS-I (2 brands).

Wrote the same image on all that cards and ran upgrade procedure.

On 16Gb class 10 UHS-I (the one i used before posted the question) with 2 brand names, and on 2 Gb with NO sdhc, that procedure takes 15-17 minutes.

On 4gb and 8gb it takes about 6-7 minutes!!!

Iostat 5 when unpacking deb packets allways show big iowait (about 40%) in all cases. But when running on 4 and 8 gb the Write statistics show speed about 2-3mb. In case with 2 and 16gb it is 500-800kb.

F3 write statistic: 2gb - 3.78, 4gb - 4.3, 8gb - 4.2, 16gb - 6.9 (tested with /var directory)

Reading speed good on all cards.

 

Here is the thing. 16gb class 10 UHS-I with unpacking packets process works the same slow as slowest 2gb card.

 

Do not know what else to do. 

[tkaiser]

Here is the thing. 16gb class 10 UHS-I with unpacking packets process works the same slow as slowest 2gb card.

 

First of all: A20 won't exceed 20 MB/s sequential speeds regardless which card you use. UHS-I doesn't play any role. As a reference: http://linux-sunxi.org/User:Wens#SD.2FMMC

 

Then: There exists a huge amount of fake SD cards (that's where the name f3 originates from: fff --> fight flash fraud). Especially 'quality brands' suffer from this (you will get an insane high amount of faked Kingston cards for example).

 

Then: f3 is meant to check the whole capacity of the SD card. You should create a single full size partition, then run f3write on the full capacity and then run f3read afterwards. This will detect bit flips and fake capacity also. And you get both sequential write and read speeds (that's important when used on a SBC).

 

And also important: The SD card speed classes always just specify sequential write speeds (since that's what has to be guaranteed when you use it with a camera or to capture video). But there's no relationship to 'random I/O'. And the latter plays an important role when an SD card is used for the rootfs. I've seen 'high class' SD cards being capable to write/read 30/35 MB/s and totally sucked when it's about random I/O (writing/reading a bunch of really small files).

[FanM]

All you wrote is totally clear for me. But checking the real capacity and exact speed is not what we need in this case.

As i told before, that cards perfectly works with wheezy. That situation is only with jessie. And as i told you that problem only with UHS-I versions.

Yes, you told that there is a lot of fake cards all around.  I am using original Kingston and A-Data, both UHS-I. Their write and read speed maybe not as should be in theory, but not less than class 4 for sure.

That 4gb card, what i mentioned, is cheapest and slowest and anyway cannot be faster than class10UHS with any quality. 

As i told before both cards from different suppliers gives absolutelly the same results. And UHS-I, by the way, use absolutelly another chip than sdhc cards.

[tkaiser]

But checking the real capacity and exact speed is not what we need in this case.

 

Nope, it's the only thing that matters. Check your cards and check random I/O also (later, using 'iozone -O' -- apt-get install iozone3). There's no need to blame a specific Armbian version for problems caused by hardware.

 

BTW: You can trust in specs but that won't help at all. Also thinking about UHS-I when the host controller isn't capable of that

 

Remark: A20 might be UHS-I capable but then board/drivers have to adjust voltages on the fly and I doubt that has been implemented. Therefore all you get with a normal SBC is 25 MB/s under best conditions. And based on the f3write results your cards seem to perform way below their specs (pretty normal, in case you're interested read from here on)

[FanM]

Thanks friend, i will try that. And i am not blaiming Igor's creation. Big thanks to Igor for that work!

P.S.: how to check I/O random?

[tkaiser]

P.S.: how to check I/O random?

 

I would use iozone -O as already written above. You could also use bonnie/bonnie++ but based on the version you choose you might not check I/O but something completely different (applies to most benchmark/monitoring tools -- always worth a read/laugh: http://slideshare.net/brendangregg/qcon-2015-broken-performance-tools)