FanM Posted January 11, 2016 Posted January 11, 2016 Hi friends! Here is the situation. Trying the armbian jessie 3.4.109 and 4.2.3. When installing a new soft it takes to much time. For example, when running apt-get upgrade after fresh image install, it takes about 17 minutes. Of course it contain the core update. But that upgrade part on jessie takes about 7min, on wheezy about 3min. The same thing when just install some software, like apache2 or mc. It takes to much time (for mc it takes about 1 min on jessie, about 30sec on wheezy). The same with reloading. It takes about 1min. Situation is the same on both jessie 3.4.109 and 4.2.3 Tried the same with Banana PI. The same thing. I use the fastest SD card. What is your expirience? Any comments?
FanM Posted January 11, 2016 Author Posted January 11, 2016 Meanwhile i was trying to find some information on that thing. Found some info about syslog and systemd. Is there any connection between my problem and systemd? Does the systemd writes the log on SD card, but syslog in cache memory?
Igor Posted January 12, 2016 Posted January 12, 2016 17 min is way too much. Some speed decrease could happen if you had terrible bad luck and httpredir choose some mirror that is broken or our servers very in trouble ... but they are working fine. Fastest card. Can you measure r/w speed within Linux. What is your download speed? How many files are downloaded. Can you post a upgrade log?
FanM Posted January 12, 2016 Author Posted January 12, 2016 Igor, hi! R speed is more than 11.5 (can not find faster destination storage) , W speed is 11.5mb/s. I wrote an image jessie 4.2.3. Ran installing mc, ntpdate, and apt-get upgrade Downloading from repository is not a problem, it is fast process everytime. The main thing is that unpacking takes a lot of time. As you can see in log, today that process took 35min. Incredable long time. Preparing to unpack .../linux-headers-next-sunxi_4.81_armhf.deb ... Unpacking linux-headers-next-sunxi (4.81) over (4.5) ... That took about 10 minutes!!! That is all kinda strange. For example, i downloaded a file with size 11M that contain 2519 files. So unpacking that file needs only 5 seconds. Do not know what to do with it. 17 min is way too much. Some speed decrease could happen if you had terrible bad luck and httpredir choose some mirror that is broken or our servers very in trouble ... but they are working fine. Fastest card. Can you measure r/w speed within Linux. What is your download speed? How many files are downloaded. Can you post a upgrade log? apt.tar.gz
Igor Posted January 12, 2016 Posted January 12, 2016 Kernel upgrade can take 10+ minutes, that's somehow normal. It's a lot of files ... there are modules and headers, so it take some time.
FanM Posted January 12, 2016 Author Posted January 12, 2016 Hmmm, yesturday you told me that 17min is to much for upgrade. But today that operation tooks 35min. and you telling me that it is normal and in fact needs 10+ minutes. I am kinda confused
tkaiser Posted January 12, 2016 Posted January 12, 2016 So unpacking that file needs only 5 seconds. Can you try it again, this time the unpack command followed by " && sync"?
zador.blood.stained Posted January 12, 2016 Posted January 12, 2016 @FanM Also try reinstalling kernel headers package (as it has many small files) and while it's installing, take a look at output of "top" command in another tty/SSH session. Values that should tell something: load average (end of first row) %cpu (3rd row) top processes and their CPU usage Post output here in code or spoiler block (1 "page" should be enough). Also check (and post here) output of "cpufreq-info" in case there are issues with cpufreq settings. Edit: Does the systemd writes the log on SD card, but syslog in cache memory? journald on Jessie by default should use only volatile storage (/run/systemd/journal), logs are written on SD card by rsyslog, which reads it from journald. It's a little CPU overhead, but it shouldn't generate extra disk writes.
tkaiser Posted January 12, 2016 Posted January 12, 2016 @FanM Also try reinstalling kernel headers package (as it has many small files) and while it's installing, take a look at output of "top" command in another tty/SSH session. I would suspect we're talking here about a horribly slow SD card. I would better go with 'apt-get install sysstat' and then let an 'iostat 5' run and watch %iowait. You might use '--force-unsafe-io option' with dpkg since it has been adjusted to use synchronous I/O a few years ago. But I would better check the SD card first (using f3 for example or checking both sequential and random I/O)
zador.blood.stained Posted January 12, 2016 Posted January 12, 2016 @tkaiser It might be, but it's quick enough to exclude from possible reasons extra CPU usage or wrong cpufreq settings. And "top" outputs %wait too. from "man top" wa, IO-wait : time waiting for I/O completion
FanM Posted January 12, 2016 Author Posted January 12, 2016 That info took when installing gdebi software. Totally it itakes 11.5min for installing. I get top info twice when it was unpacking deb packets. top - 19:26:21 up 38 min, 2 users, load average: 1.16, 0.82, 0.68Tasks: 110 total, 2 running, 108 sleeping, 0 stopped, 0 zombie%Cpu(s): 5.9 us, 9.8 sy, 0.0 ni, 42.0 id, 42.2 wa, 0.0 hi, 0.0 si, 0.0 sttop - 19:27:52 up 39 min, 2 users, load average: 1.45, 1.00, 0.76Tasks: 110 total, 1 running, 109 sleeping, 0 stopped, 0 zombie%Cpu(s): 11.5 us, 8.4 sy, 0.0 ni, 39.3 id, 40.7 wa, 0.0 hi, 0.0 si, 0.0 st 84 root 20 0 0 0 0 S 20.4 0.0 2:28.70 mmcqd/0 7054 root 20 0 11732 8616 1120 S 14.8 0.4 0:03.78 dpkg-deb 7064 root 20 0 9084 4152 3428 S 8.6 0.2 0:00.98 sshd 5849 root 20 0 32132 27564 23636 S 7.9 1.3 0:39.25 apt-get 5974 root 20 0 13856 11636 1824 D 3.0 0.6 0:20.79 dpkg root@cubietruck:/etc/apt# cpufreq-infocpufrequtils 008: cpufreq-info © Dominik Brodowski 2004-2009Report errors and bugs to cpufreq@vger.kernel.org, please.analyzing CPU 0:driver: cpufreq-dtCPUs which run at the same hardware frequency: 0 1CPUs which need to have their frequency coordinated by software: 0 1maximum transition latency: 244 us.hardware limits: 144 MHz - 960 MHzavailable frequency steps: 144 MHz, 312 MHz, 528 MHz, 720 MHz, 864 MHz, 912 MHz, 960 MHzavailable cpufreq governors: conservative, userspace, powersave, ondemand, performancecurrent policy: frequency should be within 480 MHz and 960 MHz.The governor "ondemand" may decide which speed to usewithin this range.current CPU frequency is 528 MHz (asserted by call to hardware).cpufreq stats: 144 MHz:0.42%, 312 MHz:0.14%, 528 MHz:89.03%, 720 MHz:4.92%, 864 MHz:0.83%, 912 MHz:0.31%, 960 MHz:4.34% (555)analyzing CPU 1:driver: cpufreq-dtCPUs which run at the same hardware frequency: 0 1CPUs which need to have their frequency coordinated by software: 0 1maximum transition latency: 244 us.hardware limits: 144 MHz - 960 MHzavailable frequency steps: 144 MHz, 312 MHz, 528 MHz, 720 MHz, 864 MHz, 912 MHz, 960 MHzavailable cpufreq governors: conservative, userspace, powersave, ondemand, performancecurrent policy: frequency should be within 480 MHz and 960 MHz.The governor "ondemand" may decide which speed to usewithin this range.current CPU frequency is 528 MHz (asserted by call to hardware).cpufreq stats: 144 MHz:0.42%, 312 MHz:0.14%, 528 MHz:89.03%, 720 MHz:4.92%, 864 MHz:0.83%, 912 MHz:0.31%, 960 MHz:4.34% (555)
zador.blood.stained Posted January 12, 2016 Posted January 12, 2016 Yep, 40.7% iowait. mmcqd in top. Now please check your sd card as @tkaiser advised you. Difference between jessie and wheezy may be caused by rootfs mount options.
tkaiser Posted January 12, 2016 Posted January 12, 2016 It might be, but it's quick enough to exclude from possible reasons extra CPU usage or wrong cpufreq settings. And "top" outputs %wait too. Yes, but the iostat version Linux/Armbian is using fortunately provides /proc/cpustat informations too (does not apply to iostat on other OS or even Linux distros): http://pastebin.com/r90mEWFM And what I especially like at iostat is the ability to see 'obvious' limitations (eg. a crappy random I/O implementation on a SD card that shows high sequential transfer speeds) since it not only lists 'throughput' in KB/s but also tps (transactions per second). I would suspect that @FanM will experience fixed and rather low values for both regarding writes. BTW: I had also one TF card that started (thermal?) throttling when being used intensively after approx. 90 seconds. Short tests with my MacBook showed always above 80 MB/s but after a while the thing got terribly slow. That's just another good reason to check each and every flash based media (be it an SD card or an USB thumb drive) at least one time prior to using it.
tkaiser Posted January 12, 2016 Posted January 12, 2016 available cpufreq governors: conservative, userspace, powersave, ondemand, performance This looks like mainline kernel (since 'interactive' disappeared). While I'm still convinced that your SD card is to blame for the slow speed (dpkg enforces synchronous I/O which means all changes are always written immediately to disk -- something that normally does NOT happen since write buffer/cache jumps in and the kernel writes stuff to disk later and re-uses file contents available in the cache/RAM when needed) the following I/O related settings might be worth a look: http://linux-sunxi.org/Cpufreq#The_.22ondemand.22_governor Since I still don't like jessie that much I've no idea how to enforce the io_is_busy setting for example... (this one being responsible for increasing cpufreq with %iowait)
zador.blood.stained Posted January 12, 2016 Posted January 12, 2016 @FanM says that on Wheezy performance is subjectively better, so if card is OK, it's possible that something else is reading or writing to it, especially small reads or writes that causes excessive ext4 journalling (guess "iotop" can help with it) Since I still don't like jessie that much I've no idea how to enforce the io_is_busy setting for example... (this one being responsible for increasing cpufreq with %iowait) What's the difference with Wheezy here? You still can put commands in /etc/rc.local, you can put this setting in /etc/sysfs.conf, you can write init.d script, systemd unit or udev rule to do it too.
tkaiser Posted January 12, 2016 Posted January 12, 2016 @FanM says that on Wheezy performance is subjectively better Yeah, but I don't trust in subjectivity at all. It's always based on a 'sample size' too small and there are so many other factors influencing real or subjective performance that it's not worth to trust in. And based on my experiences (part of our job is to solve IT problems at large scale) often reality will be adjusted by expectations (unknowingly of course! I learned a lot when I bought and read Blink the last time I was in Naples although I missed the beauties of the landscape since I read through the whole book on our train ride). Thx for pointing out the unchanged role of /etc/rc.local in Jessie (I really didn't know it and thought one had to create a service or any other inconvenient mechanism )
Igor Posted January 12, 2016 Posted January 12, 2016 Hmmm, yesturday you told me that 17min is to much for upgrade. But today that operation tooks 35min. and you telling me that it is normal and in fact needs 10+ minutes. I am kinda confused I read the post too quickly and haven't realized that you were upgrading the kernel too and ... The issue here is, that it has been months since last images build and we are only updating since. Before, we never had this much old images so upgrade time might become a problem. Still, this is completely normal on a PC or any other distro! First you install, than you issue update which can take hours ... if HW is on the level of Cubietruck.
FanM Posted January 12, 2016 Author Posted January 12, 2016 Friends, hi again! First of all, thanks to everyone for help. Special thanks for f3 and sysstat stuff. Here is my strange and suspicious results. I have tested that image for next cards this time: 2Gb not SDHC, 4Gb SDHC class 2, 8Gb SDHC class 10, 16Gb class 10 UHS-I (2 brands). Wrote the same image on all that cards and ran upgrade procedure. On 16Gb class 10 UHS-I (the one i used before posted the question) with 2 brand names, and on 2 Gb with NO sdhc, that procedure takes 15-17 minutes. On 4gb and 8gb it takes about 6-7 minutes!!! Iostat 5 when unpacking deb packets allways show big iowait (about 40%) in all cases. But when running on 4 and 8 gb the Write statistics show speed about 2-3mb. In case with 2 and 16gb it is 500-800kb. F3 write statistic: 2gb - 3.78, 4gb - 4.3, 8gb - 4.2, 16gb - 6.9 (tested with /var directory) Reading speed good on all cards. Here is the thing. 16gb class 10 UHS-I with unpacking packets process works the same slow as slowest 2gb card. Do not know what else to do.
tkaiser Posted January 12, 2016 Posted January 12, 2016 Here is the thing. 16gb class 10 UHS-I with unpacking packets process works the same slow as slowest 2gb card. First of all: A20 won't exceed 20 MB/s sequential speeds regardless which card you use. UHS-I doesn't play any role. As a reference: http://linux-sunxi.org/User:Wens#SD.2FMMC Then: There exists a huge amount of fake SD cards (that's where the name f3 originates from: fff --> fight flash fraud). Especially 'quality brands' suffer from this (you will get an insane high amount of faked Kingston cards for example). Then: f3 is meant to check the whole capacity of the SD card. You should create a single full size partition, then run f3write on the full capacity and then run f3read afterwards. This will detect bit flips and fake capacity also. And you get both sequential write and read speeds (that's important when used on a SBC). And also important: The SD card speed classes always just specify sequential write speeds (since that's what has to be guaranteed when you use it with a camera or to capture video). But there's no relationship to 'random I/O'. And the latter plays an important role when an SD card is used for the rootfs. I've seen 'high class' SD cards being capable to write/read 30/35 MB/s and totally sucked when it's about random I/O (writing/reading a bunch of really small files).
FanM Posted January 12, 2016 Author Posted January 12, 2016 All you wrote is totally clear for me. But checking the real capacity and exact speed is not what we need in this case. As i told before, that cards perfectly works with wheezy. That situation is only with jessie. And as i told you that problem only with UHS-I versions. Yes, you told that there is a lot of fake cards all around. I am using original Kingston and A-Data, both UHS-I. Their write and read speed maybe not as should be in theory, but not less than class 4 for sure. That 4gb card, what i mentioned, is cheapest and slowest and anyway cannot be faster than class10UHS with any quality. As i told before both cards from different suppliers gives absolutelly the same results. And UHS-I, by the way, use absolutelly another chip than sdhc cards.
tkaiser Posted January 12, 2016 Posted January 12, 2016 But checking the real capacity and exact speed is not what we need in this case. Nope, it's the only thing that matters. Check your cards and check random I/O also (later, using 'iozone -O' -- apt-get install iozone3). There's no need to blame a specific Armbian version for problems caused by hardware. BTW: You can trust in specs but that won't help at all. Also thinking about UHS-I when the host controller isn't capable of that Remark: A20 might be UHS-I capable but then board/drivers have to adjust voltages on the fly and I doubt that has been implemented. Therefore all you get with a normal SBC is 25 MB/s under best conditions. And based on the f3write results your cards seem to perform way below their specs (pretty normal, in case you're interested read from here on)
FanM Posted January 12, 2016 Author Posted January 12, 2016 Thanks friend, i will try that. And i am not blaiming Igor's creation. Big thanks to Igor for that work! P.S.: how to check I/O random?
tkaiser Posted January 13, 2016 Posted January 13, 2016 P.S.: how to check I/O random? I would use iozone -O as already written above. You could also use bonnie/bonnie++ but based on the version you choose you might not check I/O but something completely different (applies to most benchmark/monitoring tools -- always worth a read/laugh: http://slideshare.net/brendangregg/qcon-2015-broken-performance-tools)
Recommended Posts