NanoPi Neo 2, memory leak in proftpd, even worse if SSL encrypted

[esbeeb]

Armbianmonitor:

http://ix.io/1BEx

(Edit: it's a red herring for me to worry about swap.  Please see a few posts down...)

 

Hello.  My NanoPi Neo 2 only has 512MB RAM.  I'm using it for OpenMediaVault.  It starts swapping very badly (totally paralysing it) when I FTP 60GB worth of tiny files into an attached 500GB SSD SATA drive (attached with the NAS kit).  Yes, ProFTPd is RAM-hungry!

 

I know how to create and work with traditional linux swap files.  I'm a noob to this whole zram thing, which I suspect is more trouble than it's worth.  I would prefer to not use zram, and use a traditional swap file on an attached 6TB 3.5" SATA drive (attached over USB 2.0 with UAS).  It's a backup drive that sits there all day long doing nothing, in fact it goes to sleep all day (and only wakes up in the middle of the night to do a backup).

 

zramctl shows:

NAME       ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram0 lzo          60.3M   4K   78B   12K       4 [SWAP]
/dev/zram1 lzo          60.3M   4K   78B   12K       4 [SWAP]
/dev/zram2 lzo          60.3M   4K   78B   12K       4 [SWAP]
/dev/zram3 lzo          60.3M   4K   78B   12K       4 [SWAP]
 

Those zram device files are on my Sandisk Ultra MicroSD card (which is "A1"), where Armbian runs from.  My lightly-used SATA drive would be better for me, for running a traditional swap file from (due to its just sleeping all day otherwise).

 

Is there some way to tell my Armbian "quit with zram, and just use a traditional swap file, as per my /etc/fstab"?

 

BTW: I've set my "vm.swappiness" to 5 in /etc/sysctl.conf

 

[martinayotte]

3 hours ago, esbeeb said:

Is there some way to tell my Armbian "quit with zram

Edit the following file : /etc/default/armbian-zram-config

[tkaiser]

4 hours ago, esbeeb said:

I'm a noob to this whole zram thing, which I suspect is more trouble than it's worth

Sure! That's the reason we switched to zram. For no other reason than generating troubles.

 

4 hours ago, esbeeb said:

Those zram device files are on my Sandisk Ultra MicroSD card

Nope.

 

4 hours ago, esbeeb said:

My lightly-used SATA drive would be better for me, for running a traditional swap file from (due to its just sleeping all day otherwise)

Absolutely not.

 

4 hours ago, esbeeb said:

I've set my "vm.swappiness" to 5 in /etc/sysctl.conf

Bad idea.

 

http://ix.io/1BEx (the stuff everyone should upload but nobody looks at then) shows that you never updated any Armbian packages so you're using an outdated configuration (zram-config package from Ubuntu and vm.swappiness set to a low value which is bad).

 

You should backup your SD card, then upgrade your Armbian installation to get latest packages, set vm.swappiness to 100 and adjust /etc/default/armbian-zram-config for a nice overcommitment of 200%: ZRAM_PERCENTAGE=200, then reboot and try again.

 

Those zram devices are not on your SD card (this would be absolutely stupid, ZRAM is about avoiding swap on slow storage like SD cards or HDDs). The next time you report back please show relevant output from armbianmonitor -u while you transferred a bit over FTP (why that? Why not Samba or anything else more sane?) so that swapping behavior can be observed.

[esbeeb]

I did an apt-get update and apt-get upgrade, and rebooted.  This set the vm.swappiness to 100.  Here is new armbianmonitor -u output, during ftp transfer: http://ix.io/1BJA

 

Note, current version of proftpd-basic installed is 1.3.5b-4, as well as proftpd-mod-vroot, version 0.9.4-1.

 

As I watch htop, here's what I observe, during an FTP upload.  The normal RAM (never mind zram) starts off having only 128M being occupied.  As the FTP transfer progresses, the RAM usage counts up and up and up, over several minutes, in a very linear and gradual way, with proftpd as the top process.  The RAM just keeps slowly filling, at a rate of about 1MB/second, until all the RAM runs out, and then a crash!  It looks like a memory leak somehow in either proftpd-basic or proftpd-mod-vroot!!

 

Activity happens a lot in /var/log/proftpd/vroot.log during the FTP transfer, looking like the following:

 

2019-02-22 09:22:51,284 mod_vroot/0.9.4[2252]: found 0 VRootAliases in directory '/srv/dev-disk-by-id-usb-JMicron_Tech_00000000522A-0-0-part1/share2/somefolder1
2019-02-22 09:22:51,470 mod_vroot/0.9.4[2252]: found 0 VRootAliases in directory '/srv/dev-disk-by-id-usb-JMicron_Tech_00000000522A-0-0-part1/share2/somefolder2
 

I think the zram thing is a red-herring now.  It would clearly all get full, no matter how I fine tune it.  I've changed the title accordingly.

 

Accordingly, I never edited /etc/default/armbian-zram-config.

 

@tkaiser, the reason I use FTP for larger transfers, is that it's faster than SMB.  I get about an extra 10MB/sec using FTP, rather than SMB, over GbE.   I do use SMB by default (in Nemo), for smaller transfers (say, anything less than about 300 MB), and just surfing around file shares, maybe streaming a video or MP3. 

[esbeeb]

I've figured out what causes part of the memory leak.  When SSL is disabled for the FTP server (within Openmediavault, which I had enabled at first), then there will still be a memory leak, but just not as fast.

 

Yes, that means two memory leaks!  ProFTPd just leaks all the faster when you have SSL turned on.

 

With non-SSL encrypted FTP transfers, the memory is still leaking, just much slower.  As the RAM usage slowly creeps up, at a rate of about 1MB every 5 seconds, my 60GB file upload might actually make it across, before all the RAM runs out.

 

Boo, proftpd!!

 

I'll see about opening a bug later...

[esbeeb]

Here's how to re-create the memory leak.  You need to FTP in a large folder, with many tiny files.  The 60 GB folder I'm trying to upload contains about 40k files.  There are also many, many sub-folders, and sub-sub-folders, over 12k.  My NanoPi Neo crashes after uploading less than 13GB of the 60GB (when SSL is enabled on the FTP). 

 

I enabled SSL for FTP using the features provided by OpenMediaVault (within the OpenMediaVault web admin interface).  I similarly used entirely default settings for the FTP server, other than enabling SSL.

 

I believe this could rightfully be called a "denial of service", as in, a security vulnerability.  A malicious user, who had a valid username and password into the FTP server (even if just an anonymous user, if allowed), merely needs to upload a "Calibre Library" folder, containing a bunch of small ebooks, larger than 13GB in size, to cause the server to a freeze.

 

Am I the first to discover this?  Probably not.

[esbeeb]

This might be fixed in proftpd version 1.3.5d.  Note the current version in Armbian is still 1.3.5b.

[Igor]

58 minutes ago, esbeeb said:

This might be fixed in proftpd version 1.3.5d.  Note the current version in Armbian is still 1.3.5b.

If installing packages from https://packages.debian.org/buster/proftpd-basic (+ this app dependencies) solves this problem, we can put them to our repository. Upload here and sent PR: https://github.com/armbian/upload/tree/master/debs

If not, we have nothing else, but wait that Debian people fix this upstream.

[JMCC]

Debian people are very good at backporting security fixes. Why not just make a bug report there?

[tkaiser]

1 hour ago, Igor said:

If installing packages from https://packages.debian.org/buster/proftpd-basic (+ this app dependencies) solves this problem, we can put them to our repository.

 

Unbelievable.

 

2 hours ago, esbeeb said:

Note the current version in Armbian is still 1.3.5b.

 

Nope, Armbian has no such package. Armbian is just a build system producing OS images based on some Debian or Ubuntu variants even if @Igor constantly tries to hide this by advertising Armbian as 'the best OS for SBC'. You're dealing with plain Debian here and if you found a bug you should report it upstream there. If you want a more recent Debian package version you should search backports repo first and if there's nothing try it with apt pinning.

 

@Igorundermining stability of the package system in general by pulling in Buster packages into a Stretch repo is... I miss words... I mean what's the reason to rely on Debian's package system? What's the reason to rely on 'everything outdated as hell' AKA Debian in the first place? Isn't it the promise of 'stable' and the benefits of relying on a strong team of maintainers? It really looks like Armbian is still moving into a direction where anyone interested in STABLE operation ist lost  

[JMCC]

8 minutes ago, tkaiser said:

advertising Armbian as 'the best OS for SBC'

Well, I think that matter is subject to opinion. Actually, Armbian offers full OS images for download, so IMO it is legit to advertise it as an OS, even though the main work of the project is in the kernel and build script and not in maintaining the userspace layer of the OS.

 

10 minutes ago, tkaiser said:

undermining stability of the package system in general by pulling in Buster packages into a Stretch repo

I totally agree with this. If we are constantly complaining about the lack of resources to maintain the project, it is not a good idea to mess with upstream problems, where there are already hundreds of qualified developers working on it. Doing backports ourselves and maintaining them is a huge lot of work.

[Igor]

8 hours ago, tkaiser said:

Unbelievable.

Application is not critical and my proposition is that if he find a working hack to patch this, why not? I am patching Chromium this way, because its a popular app and its a total mess upstream. It doesn't work in STABLE build at all. In general, many hacks are done dirty and you don't complain because you don't know about. Debian is not automotive grade Linux. At leas you and me know that, while most of the people out there have no clue.

Armbian implicates its something to do with ARM and something to do with Debian(Ubuntu) and we are trapped inside this. Its a build system in its core, yes. What that builds system does? It makes board support packages (u-boot, kernel and board specifics hacks) and bind them with a Debian like system to a boot-able image ... which gets distributed. We define package base, we change some settings and we fix some user land bugs. This was the initial idea, this idea was shaped trough the time and will be changed, adjusted in the future.

 

Why shell a few Armbian folks, which are anyway busy with more important and very specific stuff, report and fix generic architecture agnostic upstream bugs? Should I deal with a bug that is found by 3rd party in proftpd? Do I want? NO. Is that my responsibility? The same as from anyone. Do I have time to deal with that? Absolutely no.
 

8 hours ago, tkaiser said:

undermining stability of the package system

By updating a single app which is broken anyway? Calm down.

 

8 hours ago, tkaiser said:

I mean what's the reason to rely on Debian's package system? What's the reason to rely on 'everything outdated as hell' AKA Debian in the first place?

Nothing specific. Because people are used to this user land perhaps? Perhaps because there is not possible to produce automotive Linux with resources which are available?

Changing to something better and keep maintaining Armbian at the same time? How? Why?

 

8 hours ago, tkaiser said:

It really looks like Armbian is still moving into a direction where anyone interested in STABLE operation ist lost

Debian folks are also doing mistakes (because they are human?) ... but yes, I agree with you - Debian uses outdated package base. Mint? Is RedHat branch or Arch any better? Perhaps Gentoo?

Now what? We are tied to Debian, which has outdated package base. Ubuntu can only be slightly better or worse. We had discussions and ideas in the past to implement or fork some existing Linux from the embedded side. Something very very minimal (Alpine?), which is limited in functions ... and problems. 

Our Linux is limited to ARM platform. Whenever we stay limited only on ARM, development in sense of going on new/own user land path represent a waste of resources if dealing only with ARM. And we open yet another box of related problems.

[esbeeb]

11 hours ago, Igor said:

Calm down.

Agree strongly.  @tkaiser, you're technical expertise is clearly very well developed, and is an asset to this community, but some diplomacy and gentle speech will go a long way towards everyone getting along on this forum.  Getting along is very important if Armbian and Linux in general is to grow as strong as we would all like it to, longer term, especially as it moves rapidly towards a more prime-time audience who want and expect to see a polished act.

 

So let's not over-react here.  Harsh speech doesn't help.

 

I looked for a debian backport of proftpd.  Unfortunately, there isn't one that I could find.  I can live with this problem for now, it's no show stopper.  I still have SMB working decently well in OpenMediaVault, and I can just turn off the FTP server as a workaround.

 

I'm just about to go for a week's vaction, and I'll see what may have transpired here when I return.  Sorry, I don't have the time or energy right now to go through Debian's formal procedure to file a security vulnerability report (which I feel is the best and most proper solution).  If someone else would like to do that, I would really appreciate it.  I also don't have time to send a report upstream to the OpenMediaVault project either, who might not see this thread.

 

Debian is a very conservative distro, but at least they do backport fixes to Security Vulnerabilities.  I think they are the first resort to a solution, which would then trickle downstream to us. 

 

I'm personally glad Armbian is based on Debian, and I probably wouldn't have picked Armbian in the first place (as my go-to distro for non-Raspberry Pi ARM boards), if it weren't Debian-based.  I forgive Armbian from having to do many hacks behind to scenes to make it work, because I know what a hairball the Linux experience tends to be on all the wild and crazy and diverse ARM development boards, no matter which way you try to approach it.

[tkaiser]

On 2/24/2019 at 6:36 AM, esbeeb said:

@tkaiser, you're technical expertise is clearly very well developed, and is an asset to this community, but some diplomacy and gentle speech will go a long way towards everyone getting along on this forum

 

Oh, "this forum"...

 

This forum is pretty much irrelevant for what's important. I pushed contents into this forum for over 3 consecutive years trying to attract foreign readers/developers to these contents and get interested in Armbian to get broader adoption and relevance.

 

My goal was to strengthen a small project (back in 2015) to become relevant since my needs are a stable OS distribution on ARM (I'm a server guy, I'm not interested in fancy shit but stable operation). Unfortunately to no avail. In theory both fancy shit and stable operation are possible at the same time but that's not how it works here.

 

Armbian is still in playground mode. And it won't change anytime soon or at all. If the 'project lead' now even thinks about sabotaging Debian's packaging all is lost. There's no 'checks and balances' into place compared to serious software projects if one person simply can decide to do whatever crazy idea strikes his head.

 

It's a problem of ignorance and you can't argue against it if the affected person simply doesn't give a shit. Look at 

Countless times developers tried to escalate those old and boring problems in a polite way. What happened? It got ignored. In the end this is a single person's project the way it's set up since while all contributing developers always tried to achieve a consensus and conform to (non-existent) rules Igor simply does what he thinks would be the best idea at the very moment. While complaining being overwhelmed he even invests time to make things worse (see the absolutely useless last efforts to change kernel versions for XU4 platform). I'm tired of cleaning up since I can spend my time on more important things.

 

It's not about which OS base to choose but to understand that a project needs rules and defined goals at least if it want to leave playground area and become the basis for 'stable operation'. Unfortunately this is not possible with Armbian. After wasting several days of my life for discussions here with always the same result (Igor doing what he wants to do without communication or even feeling bound to a 'consensus' reached before) it's time to stop.

 

@Tido move my post to the bin as usual!

[Tido]

7 hours ago, tkaiser said:

move my post to the bin as usual!

Thank you for the flowers, but I have never deleted a post from you.  Why do you think so?

[esbeeb]

 

9 hours ago, tkaiser said:

Armbian is still in playground mode.

 

The way I see it, all these ARM development boards always show up on the market in a highly immature state (with respect to software support in linux, and all associated layers above the kernel).  Indeed, they are a playground for Linux developers, and are thus called development boards.

 

I don't see how Armbian, or any other project like Armbian (like say, DietPi) could ever hope to reach "non-playground mode", considering that all boards arriving onto the market take huge efforts, one board at a time, to smooth all the rough edges.  That can be so very time consuming, that it's no surprise that the Armbian project, as a whole, doesn't ripen in all the other ways that we might like.

 

I for one, do not blame Igor, for the horrifically unpolished state (Linux software-wise) that new ARM development boards are "born with" on the market.  Each new board, which this project decides to take seriously, dumps a huge burden onto this community.

 

I, for one, would like to suggest that boards coming from vendors who have track records of creating a lot of development work to get polished, should not be included in Armbian, no matter how sweet the tech specs look, and no matter how low the price is.  Even if just a handful of boards are works-in-progress or supported, from a more benevolent, Linux-friendly vendor, I call that options enough.

 

So in summary, I say, support fewer, more sane boards!  Quit trying to support a wide range of boards that are too large a nightmare to tackle.  Sure, list those "nightmare" boards, but then just say underneath "too big a nightmare for Armbian to develop for".  That will send a message back to the vendors to make their boards, and linux software offerings, more sane.

 

@Igor, which 10 Armbian-supported/WIP boards would you say are the biggest nightmares to support?  Which vendors are the worst for making software development hassles?  Maybe just halt all development efforts on those.  Sure, many people might grumble, but once they get over it, they can fork out $100 or whatever to just buy some newer, better board which is much easier for Armbian to support, and "get back in the game" again.

[esbeeb]

I just emailed Debian security, informing them of the DOS Attack.

[esbeeb]

They had me file a bug report:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=923926

[esbeeb]

There's still no security fix yet for the proftpd currently in Debian stable, and Debian agrees proftpd 1.3.5d upstream does contain the fix.

[dolphs]

hmm what about 1.3.6-4?

https://packages.debian.org/sid/proftpd-basic

 

to be added, eg:

cat sid.repo
deb http://httpredir.debian.org/debian sid main contrib non-free


cat upgrade.sh
#!/bin/sh

cat sid.repo >>/etc/apt/sources.list

apt-get update
apt-get -t sid install -y proftpd-basic

sed -i '$d' /etc/apt/sources.list

 

[JMCC]

Okay, let me strongly state that simply installing packages from a different Debian release, or mixing sources from different releases, is a very, very bad idea. It will make your system unpredictably unstable, and probably ruin it. Even with the proper apt-pinning (which is not even being considered in this thread, except for one mention by @tkaiser above) is not a good idea IMO.

 

If you want to install a more recent version of proftpd or any other package, simply make a backport. Debian makes it extremely easy, and there are different forms of doing it, all of them quick and easy. Like, for example, this one: https://wiki.debian.org/SimpleBackportCreation

 

[esbeeb]

I, for one, will patiently wait for Debian to do that backporting.  Or wait until Buster is released.  Whichever will come first.  I have enough on my plate already.

[tkaiser]

On 3/7/2019 at 8:40 AM, Tido said:

Thank you for the flowers, but I have never deleted a post from you.  Why do you think so?

Maybe you were drunk at the time and can't remember? https://github.com/armbian/build/commit/7f1c5b19cd58f100218c0175f9e81df1b5276003#commitcomment-33848416

 

Moving posts to threads that are inaccessible is the same as deletion (but maybe you're not able to get this). Apropos deletion. Asides that you have not the slightest idea what you babble here about (totally missing the context)

You claim 'it is written in the internet or in one of TKs posts, ahh wait, he deleted everything here'. Care to elaborate what you mean? Here is the list of my 5432 posts so far: https://forum.armbian.com/profile/7-tkaiser/content/ -- you almost singlehandedly stopped me from posting more in this forum since it makes absolutely no sense to post in a place where a dumbass with moderator privileges deletes posts (and either doesn't get what he does or simply lies).

 

In case you want to censor again be aware that this post is already archived: https://archive.fo/8eMcV

[Tido]

3 hours ago, tkaiser said:

singlehandedly stopped me from posting more in this forum since it makes absolutely no sense

when I read that,  it just sprang to my mind: Link

I better follow that advice.

 

[giri@nwrk.biz]

Polluting a Linux Distro Release with foreign packages is always a very very bad idea. It may lead to unforeseen library upgrades which may break  library ABIs and also may completely mess up your installation. This is one of the reasons why Debian freezes package versions on a per release basis. The Debian model is extremely conservative, but it works. My servers are up and running since Wheezy and upgrades worked flawlessly.

BTW: this package sideloading done by the armbian repos got me in trouble two or three times (quite some time ago). By the time this really pissed me off, so I switched hardware and moved to a proper distro.

Nevertheless, if you really want to pull a package from a foreign release here is how to do the safest way:

Set default release by creating following file:

$ nano /etc/apt/apt.conf.d/10defaultRelease

Add this line and save:

APT::Default-Release "stretch";

Now add repo to pull the package from by editing:

$ nano /etc/apt/sources.list

Add following line and save:

deb http://ftp.debian.org/debian/ buster main contrib non-free

Now update package list and install:
 

$ sudo apt update
$ sudo apt install proftpd-basic -t buster

PS: I have no clue about the current state, just stumbled across this thread and totally get the point of some others in here.