Tido Posted March 25, 2016 Posted March 25, 2016 Hi, I have heard in a podcast about: fatrace “file access traceâ€, not “fat race†fatrace: report system wide file access events Part of our efforts to reduce power consumption is to identify processes which keep waking up the disk even when the computer is idle. This already resulted in a few bug reports (and some fixes, too), but we only really just began with this. I cannot find it in the repositories of my: Armbian 4.2 3.4.108-banana apt-cache search fatrace apt-cache show fatrace I was wondering if only I am interested in it or if it would be interesting to add it to the repository ? Cheers Tido
tkaiser Posted March 25, 2016 Posted March 25, 2016 You're not using Armbian when it's about repository contents but either Wheezy, Jessie or Ubuntu Trusty. I would suspect Wheezy, right? https://packages.debian.org/search?keywords=fatrace Why don't you take the deb from here and try to install it using 'dpkg -i'?
Tido Posted March 25, 2016 Author Posted March 25, 2016 Well, as I read through the article form this guy, I came across, that it might not be available in Debian 7 Wheezy. So I thought as the 3.4 Kernel is quite important in the ARMHF, it may be worth to add it to the 3.4 repo of Armbian. I tried your suggestion and installed it, it did not resolve the dependency or couldn't because not available. root@lamobo:/home# fatraceCannot initialize fanotify: Function not implemented
tkaiser Posted March 25, 2016 Posted March 25, 2016 Try to use the words you read when calling a failing command for feeding search engines. It's a missing kernel thingie. And regarding '3.4 repo of Armbian' you should be aware that this doesn't exist. We use the software from Debian/Ubuntu repositories and just a few Armbian specific packages are provided by apt.armbian.com. Next step: Build an own kernel providing the necessary features to play with fat rat (for whatever reasons -- you noticed that ARM is not Intel and that you can avoid the whole issue by choosing wisely where to put the rootfs on?).
Tido Posted March 25, 2016 Author Posted March 25, 2016 of course I threw it into http://www.google.com/ncr but with "debian package fanotify" and there were no packages. Well, if you say it is a kernel thingy I understand now why. Fatrace is not fatrat obviously, so I ignore your comment about Intel.
tkaiser Posted March 26, 2016 Posted March 26, 2016 I ignore your comment about Intel. Great! If you wouldn't ignore it, turn on your brain and think about the differences between ARM boards (SD card or NAND/eMMC to run the OS from) and Intel (HDD/SSD -- dealing with unwanted wake-up events for these types of devices is what fatrace was made for) then you would be able to understand how to do it correctly: Set up everything right and don't give a shit about fatrace at all. If the rootfs remains on SD card or eMMC and a connected HDD has been configured accordingly (using hdparm or using hdd-spindown.sh with disks that ignore hdparm settings) then there's no need at all to trace file accesses. But maybe that's too easy?
Tido Posted March 26, 2016 Author Posted March 26, 2016 As you know, I have an R1, the rootfs on the HDD (currently). In theorie I can also keep the OS on the SDcard and only have Data on the HDD. As I heard about that tool I thought it was a good idea to find out which of the files are often accessed and optimized the system accordingly. Probably there are other tools, but as I read through the comments it sounded like a really good idea. It also comes with filter-functions for the log file. Btw, if you have Debian 8 Jessie, it should be in those repositories. To make things worse, I have: LAMP Textpattern (cms) Ampache (Music streaming) Firewall CSF Seafile Transmission-Daemon (will not run all the time) OpenElec (if I ever get HW acceleration running) In this respect it might be not easy to get this thing sleeping - therefore I was looking for such a software like fatrace.
tkaiser Posted March 26, 2016 Posted March 26, 2016 As you know, I have an R1, the rootfs on the HDD (currently). As you know I neither care about the mistakes you made nor is there a need to do it wrong. You get a 64GB Samsung EVO TF card for less than 20 bucks today (sequential performance exceeding A20's SDIO bandwidth and showing also superiour random I/O performance when choosing this capacity) so keep on spinning your HDD, start to waste your time analysing disk access patterns and refrain from doing it more intelligently!
Tido Posted March 28, 2016 Author Posted March 28, 2016 Absolutely, but because You think it was a mistake, doesn't mean I stop using it It is not about the money. It is about power-consumption, heat and last but not least tinkering
zador.blood.stained Posted March 28, 2016 Posted March 28, 2016 I don't want to argue with either of you, but I enabled fanotify to sun4i and sun7i legacy kernels, so it should be available in new kernel release. 1
tkaiser Posted March 28, 2016 Posted March 28, 2016 You think it was a mistake, doesn't mean I stop using it [...] It is about power-consumption OMFG! You still don't get it, right? I'm talking about the very same: Power consumption. And if you would do it right (rootfs on a high capacity SD card and only data that will be used when really needed on HDD) the whole fatrace thing would be pointless since you already solved the problem you're trying to analyse. Is it really that hard to understand that conventional HDDs and flash based media like SD cards or SSD are different? And that it's that simple to benefit from flash advantages on ARM boards since it's already the default to run the OS off an highly energy efficient SD card? You introduced your problem manually (the 'mistake' I referred to) by moving the rootfs to HDD which is something I would never do unless the device has high sequential storage bandwidth requirements 24/7. By doing it intelligently (leaving rootfs on the SD card and using a card of approriate size) you can reduce HDD spin-up times to a minimum and especially prevent these unwanted 'wake up' events.
tkaiser Posted March 28, 2016 Posted March 28, 2016 I don't want to argue with either of you, but I enabled fanotify to sun4i and sun7i legacy kernels, so it should be available in new kernel release. Thx, but why restricting it to these 2? Being able to use fanotify can help in some situations. Same applies to TASKSTATS necessary for iotop for example that should IMO look like with sun7i/sun8i in every kernel config: macbookpro-tk:config tk$ grep CONFIG_TASKSTATS linux-*.config linux-cubox-default.config:CONFIG_TASKSTATS=y linux-cubox-dev.config:CONFIG_TASKSTATS=y linux-cubox-next.config:CONFIG_TASKSTATS=y linux-marvell-default.config:# CONFIG_TASKSTATS is not set linux-marvell-dev.config:# CONFIG_TASKSTATS is not set linux-marvell-next.config:# CONFIG_TASKSTATS is not set linux-neo-default.config:# CONFIG_TASKSTATS is not set linux-odroidc1-default.config:CONFIG_TASKSTATS=y linux-odroidc2-default.config:# CONFIG_TASKSTATS is not set linux-odroidxu4-default.config:CONFIG_TASKSTATS=y linux-odroidxu4-next.config:CONFIG_TASKSTATS=y linux-s500-default.config:# CONFIG_TASKSTATS is not set linux-s500-next.config:# CONFIG_TASKSTATS is not set linux-sun4i-default.config:CONFIG_TASKSTATS=y linux-sun5i-default.config:CONFIG_TASKSTATS=y linux-sun7i-default.config:CONFIG_TASKSTATS=y linux-sun8i-default.config:CONFIG_TASKSTATS=y linux-sunxi-dac.config:CONFIG_TASKSTATS=y linux-sunxi-dev.config:CONFIG_TASKSTATS=y linux-sunxi-next.config:CONFIG_TASKSTATS=y linux-toradex-default.config:# CONFIG_TASKSTATS is not set linux-udoo-default.config:# CONFIG_TASKSTATS is not set linux-udoo-neo-default.config:# CONFIG_TASKSTATS is not set linux-udoo-next.config:# CONFIG_TASKSTATS is not set macbookpro-tk:config tk$ grep FANOTIFY linux-*.config linux-cubox-default.config:CONFIG_FANOTIFY=y linux-cubox-default.config:CONFIG_FANOTIFY_ACCESS_PERMISSIONS=y linux-cubox-dev.config:CONFIG_FANOTIFY=y linux-cubox-dev.config:CONFIG_FANOTIFY_ACCESS_PERMISSIONS=y linux-cubox-next.config:CONFIG_FANOTIFY=y linux-marvell-default.config:# CONFIG_FANOTIFY is not set linux-marvell-dev.config:CONFIG_FANOTIFY=y linux-marvell-next.config:CONFIG_FANOTIFY=y linux-neo-default.config:# CONFIG_FANOTIFY is not set linux-odroidc1-default.config:# CONFIG_FANOTIFY is not set linux-odroidc2-default.config:# CONFIG_FANOTIFY is not set linux-odroidxu4-default.config:CONFIG_FANOTIFY=y linux-odroidxu4-next.config:CONFIG_FANOTIFY=y linux-odroidxu4-next.config:# CONFIG_FANOTIFY_ACCESS_PERMISSIONS is not set linux-s500-default.config:# CONFIG_FANOTIFY is not set linux-s500-next.config:# CONFIG_FANOTIFY is not set linux-sun4i-default.config:# CONFIG_FANOTIFY is not set linux-sun5i-default.config:# CONFIG_FANOTIFY is not set linux-sun7i-default.config:# CONFIG_FANOTIFY is not set linux-sun8i-default.config:CONFIG_FANOTIFY=y linux-sunxi-dac.config:CONFIG_FANOTIFY=y linux-sunxi-dev.config:CONFIG_FANOTIFY=y linux-sunxi-next.config:CONFIG_FANOTIFY=y linux-toradex-default.config:# CONFIG_FANOTIFY is not set linux-udoo-default.config:# CONFIG_FANOTIFY is not set linux-udoo-neo-default.config:# CONFIG_FANOTIFY is not set linux-udoo-next.config:# CONFIG_FANOTIFY is not set
zador.blood.stained Posted March 28, 2016 Posted March 28, 2016 Thx, but why restricting it to these 2? Being able to use fanotify can help in some situations. Same applies to TASKSTATS necessary for iotop for example that should IMO look like with sun7i/sun8i in every kernel config: I quickly enabled this for kernel sources that are present on my build host (and to solve @Tido's problem). Editing kernel configs manually is not a good idea anyway, and it's hard to keep track of all features for all kernel branches.
tkaiser Posted March 28, 2016 Posted March 28, 2016 Editing kernel configs manually is not a good idea anyway, and it's hard to keep track of all features for all kernel branches. Hmm... regarding these monitoring features it's obvious that settings differ between different kernel branches by accident and not by design. We should start to enable this one by one when touching the kernel config of one of the affected boards in the future (at least I will do)
Rui Ribeiro Posted March 30, 2016 Posted March 30, 2016 Fatrace seem interesting....I was using dtrace and later on switched to sysdig, which do provide fatrace functionality and so much more however alas, as far as I aware they cannot work in ARM as the kernel does not provide some debugging functionalities that exist in Intel. Sadly, I do not remember the technical details...
tkaiser Posted March 30, 2016 Posted March 30, 2016 There are tons of tracers available in Linux: http://www.brendangregg.com/blog/2015-07-08/choosing-a-linux-tracer.html The problem is to become familiar with each and everyone and not to misuse them (trying to use fatrace to analyse problems that don't need to exist on ARM boards when set up correctly)
Tido Posted March 30, 2016 Author Posted March 30, 2016 Yeah tell me more about that TK, this is my install To make things worse, I have: LAMP Textpattern (cms) Ampache (Music streaming) Firewall CSF Seafile Transmission-Daemon (will not run all the time) OpenElec (if I ever get HW acceleration running) If I keep /boot and /rootfs on the SDcard, does the system automagically know, that /var/www/ sits in another partition on the HDD ? or in your scenario would you keep all programs on the SDcard and only /media/data-partitions on the HDD?
tkaiser Posted March 30, 2016 Posted March 30, 2016 If I keep /boot and /rootfs on the SDcard, does the system automagically know, that /var/www/ sits in another partition on the HDD ? Why do you ask such moronic questions instead of starting to think about what you're doing?! If you don't want to spend a bit of brain on this stuff then use a spinning disk and waste your time with fatrace. Otherwise learn Linux basics (symlinks for example -- they point from here to there -- or fstab where you specify mount points for partitions), think about using mainline kernel (since you will never get video HW acceleration the way you deal with it! Cedrus/vdpau is already working with legacy kernel and OpenELEC is a dedicated minimal distro that can't be mixed with Armbian. So what are you talking about anyway?!) and use transparent filesystem compression. Again: A Samsung EVO TF card with 64GB costs less than $20 bucks and performance can't be much better compared to more expensive cards (since sequential write/read speeds are limited by the board and since random I/O performance benefits from the large capacity of this card). Since it's SDXC I would suspect cards with 128GB/256GB should also work. If I were you I would analyze the storage requirements and put most if not all on the SD card using btrfs with LZO compression. And only non 24/7 stuff on the HDD (the videos you want to watch from time to time and stuff like that). BTW: /var/www on HDD seems to be a very bad idea. Use a card with high IOPS, put all the stuff there and only static content that is not accessed frequently on a HDD that you mount as /var/www/static-content/ for example. It's Linux so you're free to optimize/adjust everything to your needs! 1
Rui Ribeiro Posted March 30, 2016 Posted March 30, 2016 I got a bit lost tkaiser. From the energy saving point of view, it pays using the SD cards, no point arguing about that. I also understand you talking about reorganising your directories/data around. However, my doubt is, wont it get more performance and stability over time using and SSD disk, for instance on the lamobo that has the native SATA interface?
tkaiser Posted March 30, 2016 Posted March 30, 2016 However, my doubt is, wont it get more performance and stability over time using and SSD disk, for instance on the lamobo that has the native SATA interface? Of course! But it's pretty easy to combine the advantages of both flash storage and spinning disks by choosing carefully where to store which stuff. So either replace the HDD with a huge SSD (best solution but still insanely expensive) or start to think about doing it right. As already said: I would never put the rootfs on a HDD unless the HDD has to be running all the time anyway. By keeping the rootfs on the SD card (and storing there everything where you can't control access to in a reasonable way) and only storing stuff on the HDD that will be accessed from time to time you can allow the HDD to spindown when not needed. But if a webserver/CMS is configured to store the data on a HDD how will you prevent unwanted disk wake-ups at all? A single client accessing the favicon of a web page and... HDD spins up since the web server process stats this file. No need for tracing at kernel level, it's already enough to have a look in the logs of server processes to get the idea. And while you can do some fancy caching stuff to prevent HDD access it's way easier to do storage classification and put things on flash based media that will be accessed randomly/frequently and put only stuff on cheap HDD storage that will be accessed from time to time. We have exactly one Lamobo R1 running at a customer's location. Of course rootfs on SD card and 2 partitions on HDD: the large one stores video streams collected from a bunch of RPi over the day, the other will be mounted by a script once a week to do a rsync backup of the SD card. HDD wakes up at 9:45 when video surveillance starts and goes to sleep at 19:20 every day. Impossible when the rootfs would be on HDD. Using a large SSD would be the storage media of choice but if you can do storage classification intelligently you can use a cheap large HDD together with flash based storage (in this case a rather large SD card) to benefit from the advantages of both technologies without suffering from the drawbacks. BTW: I've been surprised how cheap fast/reliable SD cards got in the meantime. If you take the technical limitations of the SD card implementation of the board in question into account then the most expensive cards aren't worth a try (since all the faster SDIO modes can't be used with cheap boards). Maximum sequential throughput on A20 boards for example is limited to ~16MB/s with legacy kernel anyway, what differs a lot is random I/O. And since random I/O is also more important when using an SD card in a SBC you can choose cards that seem rather slow compared to more expensive ones without loosing performance. The 64GB Samsung EVO I ordered recently is pretty slow when it's about sequential writes/reads (when writing an OS image to it on my MacBook it's just ~20MB/s instead of +80MB/s with the SanDisk Extreme Pros I normally use) but that doesn't matter with most boards since they're slower anyway. But the EVO outperforms most other cards when it's about random I/O since its controller makes use of parallel writes/reads. The 64GB EVO shows way more IOPS than the 16GB EVO for example and any of the more expensive SanDisk Extreme Pro. I'm currently preparing a storage FAQ for Armbian and ordered a few more Samsung cards: 32 GB EVO+, 64GB EVO+ and a 64GB Pro (no Pro+ since I have no board that could make use of the performance) to be able to compare. I would suspect that the 64GB EVO outperforms the 32GB EVO+ in an average SBC since max sequential peformance doesn't matter and 64GB outperforms 32GB when it's about random I/O since parallelisms work more efficiently. We'll see... Starting with Armbian 5.1 we'll also log/monitor SD card data in /var/log/armhwinfo.log and then I will start to ask our users that bought more recent SD cards to contribute to some sort of collective benchmarking (preparing a script that prevents collecting meaningless results by controlling benchmark execution)
RagnerBG Posted March 30, 2016 Posted March 30, 2016 I've never seen SD card for a reasonable price that can get even closer as performance, compared to the slowest present day mechanical disk. And SD cards are not so persistent at constant w/r over time as HDD-s. They drop performance and even semi-brake over time, so i wouldn't count on storage like this, but for rootfs, maybe there is a point, if we don't seek performance and we plan to use it and forget it. But again - why we should do this on board like R1, where there is convenient way of using HDD directly on board (after we deal with power issues of course)? This is one of the main reasons i ordered this board (not that i don't regret now, but still like the idea). I am using rootfs on mechanical disks in OpenWrt routers and Lamobo-R1 with Armbian. On both, disks are off in most of the time and spin very rarely, but more often in Armbian, than OpenWrt, where i have a lot of stuff running, but also using HDD-Idle. In Armbian i have almost nothing except some multimedia, so maybe HDD-Idle is a good idea for there too, but i am not at that part yet. I also have similar configuration as @tkaiser recommends - OpenWrt router with rootfs on USB flash and HDD for storage, i can confirm this is probably better use of HDD.
tkaiser Posted March 30, 2016 Posted March 30, 2016 @RagnerBG: How do you define 'performance' when we're talking about storage? Most people only think about moronic dd 'benchmarks' measuring bullshit. And ignore that a more recent quality SD card outperforms even the fastest mechanical drives when it's about random I/O. That's the important part: What's the use case? If you're recording video then you're interested in high sequential transfer speeds (cheap HDD outperforms SD card). If we're talking about where to store the rootfs or user's homedirs then sequential transfer speeds are pretty irrelevant and it's all about random I/O (cheap SD card when chosen wisely outperforms most/any cheap HDD). As long as the specific use case isn't taken into account all this 'performance' talking is meaningless (especially when only numbers of historical interest are considered -- I don't care that much how SD cards performed a few years ago since the ones I use now are obviously different)
Recommended Posts