armbian-config mayan edms deployment fails

[w0ndersp00n]

So with a few modifications in the script from Mayan I was able to build the image on my C2. Building on the C2 with Class 10 SD Card takes about 50 minutes, so I assume that a Raspberry Pi 3 might need more than 1 hour.

 

However there are some issues which I haven't figured out, since I usually don't write or edit shell scripts.

• The default Installation script pulls mayanedms:latest, however during the build a versioned image is created. This means that the variable DOCKER_MAYAN_IMAGE should be updated for every new version. I don't know if there is some automatic way to do this or if the script needs updating after every release.
• Right now it's not possible to add additional tesseract-ocr languages. They should be added when creating the container, but since that happens in the script the only way to do that is to remove the container after the script finishes and create it again with the right environmental variable.

Another small gripe is the memory usage. On my C2 it uses about 900MiB of memory and on my N2 about 1300MiB, so that's quite a lot.

 

Besides those bullets it pretty much works ok on my Odroid C2 and N2.

 

https://gist.github.com/w0ndersp00n/86cd913616d64eb90674aedf4b6f6d0d

Edited April 9, 2020 by w0ndersp00n
Added gist

[Tido]

3 hours ago, w0ndersp00n said:

takes about 50 minutes

Thank you very much 

 

I have to admit I am a total noob if it comes to Docker. Are there some tools to improve building such an image or tweaks to bring down building time (reuse some parts the next time) ?

 

@Igor, Master of bash-scripts

3 hours ago, w0ndersp00n said:

or if the script needs updating after every release

What is your thought about it?

 

[Igor]

On a long run, software installation should anyway be switched to docker based installation. And drop everything else. But again, if I am alone doing that, I will not even start.

[w0ndersp00n]

3 hours ago, Tido said:

I have to admit I am a total noob if it comes to Docker. Are there some tools to improve building such an image or tweaks to bring down building time (reuse some parts the next time) ?

 

 

Considering the complexity and all the dependencies I doubt it. The buildfile for Mayan EDMS uses Debian Slim and removes a lot of packages during the build in order to keep the resulting image as small as possible. It theoretically would be possible to to switch to Alpine, but considering all dependencies for Mayan I don't think it's worth the trouble.

 

For every new version of Mayan a new image should be build. So that would mean an hour of time from a C2 or RPi3. For anything lower end I wouldn't even consider Mayan EDMS, it's way to heavy.

 

Right now I'm trying to figure out if I can extend the script in order to automatically get the latest version number from here: https://docs.mayan-edms.com/releases/index.html or here: https://pypi.org/project/mayan-edms/

 

If I'm able to do that, then this script will be able to automatically build the latest version and use it to create the container.

 

I've never installed Mayan using Softy, so I don't know if the user can change the Environmental Variables before the script actually starts.

[Tido]

On 4/9/2020 at 10:33 PM, w0ndersp00n said:

For anything lower end I wouldn't even consider Mayan EDMS, it's way to heavy.

Good to know.

This raises the question of turning away from the Docker image and install it natively - in order the reach a bigger audience.

 

And if this install fails anyway, is there a way to track/record the actions of this install to remove it completely - it consists of Ngnix, Python, Redis and Postgres... thinking out loud.

I guess there are some ways to: linux track an install; but just for apt or here.  And for Python I only found this. 

Could the script executing the install running a pipe command to keep the output in a log file,  so it would be possible to reverse the install?

 

Does this make sense?

 

Edit: I needs quite a bit: https://docs.mayan-edms.com/chapters/deploying.html

 

Edited April 10, 2020 by Tido
It is Django, not Ngnix

[w0ndersp00n]

No, installing on bare metal wouldn't lower memory usage. Maybe 1 or 2MiB from the Docker overhead, but isn't worth the trouble of installing Mayan. I'd still recommend a container (Docker or podman). Mayan has high system requirements: https://docs.mayan-edms.com/parts/installation.html#minimum-hardware-requirements

 

Considering this it simply doesn't make sense to install it on lower end ARM devices. It should have at least 1GiB of RAM, but that is the bare minimum. 2GiB is recommended, so a Raspberry Pi 3 would be the bare minimum for running Mayan EDMS. But a Pi 4 or Odroid C2 with 2GB would be a better choice.

[Tido]

25 minutes ago, w0ndersp00n said:

installing on bare metal wouldn't lower memory usage.

agreed, but wouldn't it reduce the time needed for the update procedure... unless you let it do that at 2 o'clock in the morning so one wouldn't get bothered by the down time.

 

[w0ndersp00n]

Yes, I guess updating would be quicker. But I personally wouldn't want to try; if a new version in a container would fail, it's easier to roll back.

 

I also changed the script a little bit. I figured that the version number is stored in the Git repo, so I'm using that in the script. So the current script is able to always build the newest Mayan version and start a container with that version.

[Tido]

7 hours ago, w0ndersp00n said:

I also changed the script a little bit.

I just give it a try on the Rock Pi 4b: INSTALL_DOCKER=true sh get-mayan-edms.sh

 

The script did quite a lot. I think at one point it even updated the whole system but I cannot find and see any code in the script doing that, unless armbian is configured to do unattended update.. well I also found that Thunderbird & OpenOffice is back. So, who knows what else has changed lately.

As you have mentioned before it took almost an hour. Now and then I checked htop, the SoC didn't go above 60° so I guess the busiest part was on the SDcard.

 

On 4/9/2020 at 2:52 PM, w0ndersp00n said:

Right now it's not possible to add additional tesseract-ocr languages.

would this solve it? https://docs.mayan-edms.com/chapters/docker/customizing.html?

 

22 hours ago, w0ndersp00n said:

No, installing on bare metal wouldn't lower memory usage. Maybe 1 or 2MiB from the Docker overhead, but isn't worth the trouble of installing Mayan.

If I were installing WordPress, I had a WordPress-Container, an Apache and a Postgres. These I would have to 'connect' to eachother.  And I could update each container-image independent of the others, right? 

And if so, why is it different for Mayan?

 

 

I wonder why the Memory line is completely filled while it only reports 1.43GB usage:

 

Edited April 10, 2020 by Tido

[w0ndersp00n]

4 hours ago, Tido said:

I just give it a try on the Rock Pi 4b: INSTALL_DOCKER=true sh get-mayan-edms.sh

 

The script did quite a lot. I think at one point it even updated the whole system but I cannot find and see any code in the script doing that, unless armbian is configured to do unattended update.. well I also found that Thunderbird & OpenOffice is back. So, who knows what else has changed lately.

As you have mentioned before it took almost an hour. Now and then I checked htop, the SoC didn't go above 60° so I guess the busiest part was on the SDcard.

 

This script doesn't update the system, it only installs docker when the variable is set. But great to know the build also worked for the Rock Pi!

 

4 hours ago, Tido said:

would this solve it? https://docs.mayan-edms.com/chapters/docker/customizing.html?

 

Maybe... I could add an additional variable where you can enter the tesseract-ocr languages. But if this script would eventually be called from Softy, the question would never be asked, so a default install is performed.

 

4 hours ago, Tido said:

If I were installing WordPress, I had a WordPress-Container, an Apache and a Postgres. These I would have to 'connect' to eachother.  And I could update each container-image independent of the others, right? 

And if so, why is it different for Mayan?

 

The Mayan containers can be updated independently, but this script was made for easy set-up. So it'll remove all the containers and create them again on every build. The used images for Postgres and Redis stays the same, the container just gets a new id.

 

4 hours ago, Tido said:

 

I wonder why the Memory line is completely filled while it only reports 1.43GB usage:

 

I think the yellow line is the cache and is available for other applications when they need it. Mine shows almost the same:

 

[Tido]

6 minutes ago, w0ndersp00n said:

if this script would eventually be called from Softy,

I don't know what you had to tweak on the script and if you sent a merge-request to the Mayan Gitlab, because as it is, it doesn't work.

tesseract-ocr languages ... to improve the script in such a way, only makes sense if there is the chance to improve it in general I guess?  (see next comment)

 

9 minutes ago, w0ndersp00n said:

The Mayan containers can be updated independently, but this script was made for easy set-up.

So, instead of checking what is really necessary the script simply does everything?  Could the script be changed so it only updates the Mayan image and attaches the Postgres & Redis?

Or could the part be copied to create the Mayan-only image and re-attach the Postgres & Redis (like having one script for inital build  and another script for the update)?

 

I am looking for a DMS for a long time. There is another DMS Paperless, similar idea, but smaller: https://paperless.readthedocs.io/en/latest/setup.html#setup-installation-docker  could this be a better solution?

 

[w0ndersp00n]

10 hours ago, Tido said:

I don't know what you had to tweak on the script and if you sent a merge-request to the Mayan Gitlab, because as it is, it doesn't work.

tesseract-ocr languages ... to improve the script in such a way, only makes sense if there is the chance to improve it in general I guess?  (see next comment)

 

The changes made are in no way valuable for upstream. The original script simply pulls the AMD64 image from Docker Hub, which is way quicker and easier than building the image during setup.

 

10 hours ago, Tido said:

So, instead of checking what is really necessary the script simply does everything?  Could the script be changed so it only updates the Mayan image and attaches the Postgres & Redis?

Or could the part be copied to create the Mayan-only image and re-attach the Postgres & Redis (like having one script for inital build  and another script for the update)?

 

Probably creating a second script to only update the Mayan image and container would be a simpler solution. However, recreating the containers for Postgres and Redis barely takes up measurable time on my C2. So it's not worth the effort. I assume higher end boards, such as N2 and Rock Pi 4 wouldn't even notice this part of the script.

 

The more interesting question would be: how to know if a new version was released and trigger a new build? Best practice obviously is not to do that on production systems, but otherwise you'd have to monitor the Mayan releases yourself and then decide when to trigger the new build. Unless you schedule the build.

 

10 hours ago, Tido said:

I am looking for a DMS for a long time. There is another DMS Paperless, similar idea, but smaller: https://paperless.readthedocs.io/en/latest/setup.html#setup-installation-docker  could this be a better solution?

 

 

I've looked at Paperless myself as well. About a year ago the developer announced he wouldn't be very active anymore. I just noticed that the community took over, so I quickly did a test and these are my findings:

• Paperless, just like Mayan, only offers an AMD64 image, so ARM images should be build by yourself
• On my C2, Building Paperless takes about 10 - 15 minutes, compared to the 1 hour from Mayan
• Memory consumption out of the box is about 50 MiB, versus Mayan's 350 MiB on my C2
• Building the image is way more fiddly, and scripting it would require too much effort
  • You need to clone the Git repo
  • You need to edit some config files
  • And from that moment you can start building

My conclusion would be that anyone who would like to use Paperless should build the image with the instructions from Paperless manual. After you've build the image, you probably can build new versions again after refreshing from Git.

 

After playing arount with Paperless, I found it less suitable for me. I also am looking for a DMS, and Mayan offers way more flexibility and more features regarding sorting files, searching, metadata etc. Paperless doesn't even allow for a web upload of a file, which is something that I need.