Multimedia related stuff on Armbian (OpenGL ES, Videodecoding, 'Mali' etc.)

[chwe]

Seems that hardware accelerated Videodecoding & GPU usage is the new 'hot topic' at the moment. A lot of this stuff is discussed in various topics and threads and subforums here. As nice at it is to bring this topics up, it's hard to follow them across the forum and when those questions come up from users you have to link the to a bunch of topics which they probably don't read cause it's simply to much to read (I now that this is a shitty argument, but reading is boring for most people in those days..  ). 

So why does it make sense to open another thread targeting this topic? This should summarize the things that work, no suggestions what could work, should work, may work and hope that it could work.. That's what all those other threads are for...  And it's focused on Armbian not raspian, not android, not what ever OS you use on your board. Focused on Armbian means also, the board has at least wip status and you can download images from our downloadpage.

Theres a lot of ongoing topics which might find their way into this thread as soon as this stuff is ready (e.g. hardware accelerated video encoding on allwinner in mainline).

 

Basics:

For the most more experienced users it's obvious, video decoding doesn't happen on the GPU which means Mali Support != good videodecoding performance (this might come true when those VPU stuff from mali finds its way into the SoCs we support - that will make things even more confusing ). As a side-note for the KODI users, you need also GPU support to bring it up (OpenGL ES 2.0 as described in their wiki). But videodecoding happens on the VPU not CPU nor GPU (software accelerated videodecoding is possible on the CPU, but the CPUs on ARM boards are mostly not powerful enough for software decoding in high-resolution). The GPU is responsible to keep load from the CPU for desktop usage but not for the video stuff (despite some special hacks).

A second topic which might be important, even when hardware accelerated video decoding is possible from userspace, this doesn't mean every application on your Armbian SBC can benefit from it (e.g. browser etc.).  A bit old, but I think this explains things better than I can do:

https://www.cnx-software.com/2013/12/10/most-embedded-gpus-do-not-support-hardware-video-decoding-acceleration-the-vpu-does/

 

Allwinner:

Currently, VPU support is only possible with the Legacy Kernel (3.4.113) and very limited by using desktop builds of Armbian. Behind the scenes Cedrus is responsible for the usage of the VPU. Under Armbian hardware accelerated videodecoding is possible by use of the provided MPV player. To Summarize it:

• Legacy & Desktop images only
• MPV player uses it
• H3 & A20 SoC

As far as I know, nobody tried the new drives for mainline yet on armbian. 

 

Rockchip:

RK provide a lot of stuff to make HW acc. video decoding possible with their 4.4.x legacy kernel. The efforts taken to bring this to armbian can be followed here: https://forum.armbian.com/topic/6506-tutorial-3d-video-acceleration-and-opencl-in-rk3288-boards-with-new-44-default-kernel/

 

At the moment hardware accelerated video decoding is possible with RK3288 SoCs. Supported is

• Hardware acc. video decoding by using gstreamer (up to 4k@30fps HEVC)
• H.264 HW acc. videodecoding in chromium browser (no VP9)
• by using MPV video decoding happens on CPU (up to 1080p@30fps HVEC by 80-90% CPU load) and accelerated display layer through EGL. That makes a huge difference in playback smoothness over the stock unaccelerated MPV.
• openCL (1.1) and openGL ES (3.2) is supported
• Chromium supports webGL
• HW encoding is also possible, in addition to Gstreamer, through a custom FFMPEG (included in the package from the referenced tutorial, although undocumented)

All this stuff is at the moment not provided by a stock Armbian, but @JMCC wrote an installation script for Ubuntu Xenial:

 

 

Exynos 5422

Exynos 5422 which can be found in Odroid XU4 (HC1/HC2 too but there's no HDMI output) has a less powerful VPU than the RK3288 but hardware accelerated video decoding and mali support is possible as discussed here.

• OpenCL (1.1) and openGL ES (3.1),  with excellent performance thanks to the hexa-core Mali T628.
• MPV, by default it works like in RK3288: SW decoding and accelerated EGL display. But you can add the "-hwdec" option to get HW decoding through the v4l interface (H.264 only: tested up to 1080p@30 to work well, 1080p@60 so-so).

• Chromium supports WebGL and accelerated H.264 video decoding. 

• Emby Server

All this stuff is at the moment not provided by a stock Armbian, but @JMCC wrote an installation script:

 

This thread is supposed to be updated over time as soon as things are implemented in Armbian or proper tutorials are available to do it on your own. IMO development discussion for support shouldn't happen here (this should happen on threads dedicated to a specific board/kernel) and as soon as it is confirmed to work write it here and I'll update this post to get the latest news on the starting post which makes it easy for the 'lazy' ones to get an overview of the current status for a specific board/soc/kernel. I might miss part of the things which are currently running and I'll happily update it when someone points me to those threads or writes a short summarization (preferred ) below this post. I suspect that hardware accelerated video encoding will also come up soon, so this might be also added in this thread as soon as proved results are available.

 

Side-note: I'm not sure if we should place this thread here (General chit chat) or in the Research guides & tutorials section. Cause this is more a overview and not a guide it's here, but might change in the future (depends on other opinions). @JMCC I have in mind that such a summarization was on your 'to-do' list, feel free to correct & add things I might missed here. 

 

Side-note II: The thread only represents work on top of 'stock armbian' @balbes150 work which is done for TV boxes is highly underrepresented here.  Feel free to add your opinions and additions directly into my post (as mod you should be able to do it directly). 

 

[JMCC]

Great idea. Here are some additions I would make. About RK3288:

• Custom MPV uses SW for decoding, but uses and accelerated display layer through EGL. That makes a huge difference in playback smoothness over the stock unaccelerated MPV. I'm working on HW accelerated decoding too.
• Chromium also supports WebGL, with very good performance (actually, a little better than Android).
• HW encoding is also possible, in addition to Gstreamer, through a custom FFMPEG (included in the package from the referenced tutorial, although undocumented). I'm planning to document this too

And also, I would add a whole section about XU4 (Exynos 5422), where we have made important progress too (as detailed here). This is a summary of what we have gotten so far:

• OpenCL (1.1) and openGL ES (3.1),  with excellent performance thanks to the hexa-core Mali T628.
• Regarding MPV, by default it works like in RK3288: SW decoding and accelerated EGL display. But you can add the "-hwdec" option to get HW decoding through the v4l interface (H.264 only: tested up to 1080p@30 to work well, 1080p@60 so-so).
• Chromium supports WebGL and accelerated H.264 video decoding. 

[chwe]

'merged'  

 

We might add your custom installation scripts (in case you ever feel patience to write them) here.

[JMCC]

10 hours ago, chwe said:

We might add your custom installation scripts (in case you ever feel patience to write them) here.

I'm not so sure about that. I think the best option would be to have a thread where the OP holds the link to the scripts and the instructions, and gets updated with every new release, so users don't need to search through all the thread for the latest version. And having you update the OP with each bug fix or new release would unnecessarily slow down the workflow and make you work. I think it is better if I start a new thread for that, focused only on providing the scripts and commentaries/support about them, and we keep this one for informational/research purposes.

 

(BTW: I don't think it would be a bad idea to create an Armbian community wiki for "unofficial" info like the one provided in this post, apart from the "official" docs).

[chwe]

19 minutes ago, JMCC said:

(BTW: I don't think it would be a bad idea to create an Armbian community wiki for "unofficial" info like the one provided in this post, apart from the "official" docs).

I had this discussion with @Igor in the past (split docs in armbian as OS and armbian 'the build script'). Personally, I like the Idea of having such a wiki, but someone has to maintain it, someone has to keep it updated and we barely have resources to do that.. The easiest way to do something similar would be that you open a SoC specific thread about hardware accelerated stuff here: https://forum.armbian.com/forum/26-research-guides-tutorials/ where you can edit your own posts with updates or Igor 'hires' you as a mod so you can update/edit my posts as well.. 

I hope that someone does this work in case I've not the time to update this stuff anymore (just ask every mod, they can edit my posts as well).

[Igor]

2 hours ago, chwe said:

The easiest way to do something similar would be that you open a SoC specific thread about hardware accelerated stuff here: https://forum.armbian.com/forum/26-research-guides-tutorials/ where you can edit your own posts with updates or Igor 'hires' you as a mod so you can update/edit my posts as well..

Exactly. It's already problematic - information is scattered around a web page, documentation, Github and forum which is mainly community, self-organized. IMO adding yet another chunk of infrastructure (Wiki) can only make things worse. If we wanted to improve the current situation, we would need to assign a person(s), dedicated editor for keeping documentation well organized.  We can dupe docs as they are now and move them to the web page or to another form or engine ... but its a huge task/challenge.

We are releasing new webpage, which was quite a challenge, in a few weeks and with it, new features and possibilities are being unleashed. Its also a paradigm change - the current website was a single user renderer directly from build script config. With many dirty scripts and plugins. A future website should be a group, community managed with a simple but powerful backend. Most of the work was done to improve that management, previously controlled with configs. I will give rights to anyone who wants to participate as a board(s) maintainer. In a sense to set facts, collect relevant information, decide which download options should be exposed and adjust things, add warnings, etc. This task is not hard work but helps core crew to find some breath. It should be fun and it comes with some responsibility. More when it's out ...

 

Getting access there or rights for editing the documentation is liberal. Like Wiki. Anyone wants to edit textual data, ping me, you get instant commit access.

 

Back to the topic. I think we should keep those how-to's updated but implement features directly into Armbian. When things are matured and when core developers team finds the time. And then, they become irrelevant for the implementation part but still keep the educational role. For that purpose, it should remain in the right place. Subforum "Research and tutorials" is IMO right spot, it's indexed by the main search engine and it will be linked directly from our new web page. They are important.

[JMCC]

Good news about the new webpage.

 

I also have some good news about RK3288 media implementation: I finally got rkmpp accelerated mpv to work. It works really well, better than Gstreamer, although only in fullscreen. It's a real "Armbian premiere", no other distro implements it so far, neither ASUS' nor Rockhip's own. In the next days, I'll try to put that with some other new goodies into a script, and keep it available there until it is possible to integrate it in the main build script.

[JMCC]

Hello, I'm still alive. Busy last week, plus I'm working on making Kodi GBM work for the RK3288 if possible. I'd like to include it in the multimedia package. In the meantime, I posted the fix to make accelerated MPV work, in case someone wants to use it. 

 

[JMCC]

Update: Kodi+RKMPP+GBM finally working. It's been a tough one: since Kodi 18 is still in alpha state,  I started with 17.6, porting @Kwiboo's Libreelec implementation to .deb packages... only to discover, after I finally got it, that it doesn't work well with the set of kernel/libs we are shipping with Armbian, creating artifacts in videos (it is an early implementation, for older versions of Mali and RKMPP). So I had to start again, with Kodi 18.  Maybe too much time and work, but you know, once you set your mind in making something work, it is hard to stop until you get it.

 

Navigating through GUI and playing 4k videos is smooth like warm butter. But, of course, being in alpha state has quite a few glitches (e.g. crashes frequently when trying to open a file from USB stick). Now, time to package it and release it with the rest of the media stuff.

[chwe]

I'll update the post as soon as you post the tutorial.  

[JMCC]

Okay, the script was finished, I was about to publish it... and I got the idea of adding the ARM Compute Library for OpenCL support. Guess whom the suggestion came from: @tkaiser! I think he is starting to love Desktop and GPU stuff... 

[Tido]

Why don't you put it into a 'Spoiler' so others can already test it ?

[JMCC]

18 minutes ago, Tido said:

Why don't you put it into a 'Spoiler' so others can already test it ?

Well, it's not just the script, it is also all the debs that are packaged with it, so I don't want to upload files unnecessarily,  if I am going to upload a new version in a few hours. I have some free time now, so I'll try to get it ready, and if not, then I'll upload it as it is now.

[JMCC]

Finally publised it: 

@chwe : Please feel free to move it to any other subforum if that is not the proper place.

[chwe]

6 minutes ago, JMCC said:

@chwe : Please feel free to move it to any other subforum if that is not the proper place.

Means, you have now time for the RPi cam on the tinker?  since it is all packed in a script, I think you placed it in the right subforum, in case an other mod disagrees they will move it.  

[JMCC]

Exynos 5422 media script released.  The most important new features compared to the steps posted a few months ago in this thread are a FFmpeg 3.4 supporting hardware encoding,  some Kodi with decoding acceleration (in addition to MPV), and the possibility to install in a headless machine with  non-X versions of the libs and programs. Besides, of course, the goodies already included in the RK3288 script, such as Netflix support or OpenCL samples.

[TonyMac32]

I can test this later today.

[chwe]

updated Exynos 5422.. (others would call this topic bump  sorry, can't warn myself)