Testing hardware video decoding (rockchip, allwinner?)

[jock]

!! DEPRECATED !!

Instructions in this thread are oudated and superseded by the new experimental APT repository for hardware video decoding ffmpeg.

Please refer to this thread from now on!

 

Hello, recent upgrades to armbian are regarding kernel 5.15.

I noticed that many v4l2 fixes and enhancements went into this release, so I decided to compile ffmpeg using LibreELEC patched version and mpv over it.

mpv turns out to be statically linked with ffmpeg, so I propose it here for people who is interested in cutting edge kernel and wants to do some tests.

 

This has been tested on Debian Bullseye and Ubuntu Hirsute on following platforms:

• Rockchip RK3228/9 (kernel 5.10, 5.14)
• Rockchip RK3288 (kernel 5.14)
• Rockchip RK3318/28 (kernel 5.15)

 

It should work on allwinner platforms too, but I didn't test it there.

Binaries are built by me on developing boards.

 

The binary for armhf is available here

The binary for arm64 is available here

 

Copy the binary into /usr/local/bin directory of your system (mpv-armhf for 32 bit systems, mpv-arm64 for 64 bit systems):

sudo cp mpv-armhf /usr/local/bin/mpv

 

Install dependencies for Debian Bullseye and Ubuntu Hirsute:

apt install libass9 libbluray2 librubberband2 libsdl2-2.0-0 libva-drm2 libva-wayland2 libva-x11-2 libva2 libvdpau1 libx264-160 libx265-192 libxss1 libxv1 libfdk-aac2

 

I have had issues with dependencies on Debian Buster/Ubuntu Focal, in particular libx264-160 and libx265-192 are not available there.

I Solved the issue downloading the packages from Debian Bullseye web page and manually installing them.

There may be the need for some other dependency depending upon your actual installation.

 

Run mpv in a virtual terminal (videos up to 4K) with this CLI:

mpv --vo=gpu --hwdec=drm --gpu-hwdec-interop=drmprime-drm --drm-draw-plane=overlay --drm-drmprime-video-plane=primary <video.mp4>

 

Mpv can be run in X11 with this other CLI, but due to buffer copying it requires a good CPU - rk3228 and rk3328 won't even play 720p, rk3288 do 720p fine:

 mpv --vo=gpu --hwdec=auto-copy --gpu-context=x11egl --gpu-hwdec-interop=drmprime-drm <video.mp4>

 

This is an experiment and your mileage may vary a lot:

• H.264 codec should be well supported around the boards;
• H.265 has more limited support
• VP8 should be generally supported
• VP9 seems to still require some work.

[Voidbert]

Greetings!

 

I have been trying out hardware video decoding recently, running Debian bullseye, kernel 5.15.53-rk322x and the lima open-source drivers on my RK3229. Here are my conclusions (working only on a virtual terminal):

• Using Debian's default ffmpeg and mpv installation, I am only able to get up to 720p with few frame skips. That is using software decoding (--hwdec=no) and SDL for rendering (--vo=sdl). Trying to use DRM (for either software decoding and / or rendering) will result in worse performance. Atomic modesetting needs to be turned off (--drm-atomic=no) in order for anything to appear on my screen (or else, I just get a gray screen). I found this issue of yours (https://github.com/mpv-player/mpv/issues/8522) but it applies to the LibreELEC build. In my case (repo build), --drm-atomic=no needs to be turned on in order for the video playback to work, but it is of no use, since you get better performance out of SDL. --hwdec=rkmpp was also tried but performance differences weren't observed.
• When you say you built ffmpeg using LibreELEC's patches, are you referring to this? https://github.com/LibreELEC/LibreELEC.tv/tree/0e57a30916a3a88e0d4c595a5525d3c03156fcc6/packages/multimedia/ffmpeg/patches
• Using your version of mpv, everything works as supposed! Some notes: YUV420 video works very well but RGB video doesn't (both hardware and software decoding). I know it isn't common, I tried it because it could possibly be faster if the CPU didn't have to convert colors between formats, but it ended up being slower.

I have tried to provide my help to the Armbian project previously, but what is needed isn't what I know how to provide. Though, I have an idea to help this community: maintaining ffmpeg packages that can take advantage of this hardware acceleration on these chips. This can be done externally of the Armbian project. What do you think of this idea?

[jock]

3 hours ago, Voidbert said:

What do you think of this idea?

I think this is a great idea!

Ideally having a repository for debian and/or ubuntu where latest development ffmpeg from LibreELEC repository is built and kept up-to-date would be a huge helper for multimedia advancement in armbian!

Something already happens, for example, right now with oibaf repository: oibaf ppa provides latest development mesa packages for current ubuntu distros, thus immediately enabling latest features for both lima and panfrost drivers.

[Khánh Ngô]

I tested on Ubuntu Jammy 5.15.80 (using mpv binary jammy in your folder) on H3 chip. It works like a charm. 

1080p (h264) was tested with 4 frame drops (10min video).

 

Using this command:

mpv --vo=gpu --hwdec=drm <video.mp4>

 

 

But, i want to use --script-opts, --ytdl-raw-options, --ytdl-format. I can use it in mpv from apt-get but your mpv can't parse.

Error parsing option ytdl-format (option not found)
Setting commandline option --ytdl-format= failed.

 

Can you compile again for jammy with that feature support, please? 

 

That's great for those who want to listen to music and videos from YouTube like me.

 

Thank you.

[baryon]

Confirmed working in Bookworm (Armbian 23.8.1) with the right dependencies plucked from Jammy.

• libdav1d5_0.9.2-1_armhf.deb
• libjpeg-turbo8_2.1.2-0ubuntu1_armhf.deb
• libx264-163_0.163.3060+git5db6aa6-2build1_armhf.deb

What does it take to build this, @jock? I am very unclear on your build process. I have the notion to try adapt it to the Debian side of things as I'm not a fan of this kind of nonce package addition.

Edited October 18, 2023 by baryon

[robertoj]

Baryon, do you mean this is working in H3, outputting through HDMI or NTSC? with window manager? With X?

What single board computer did you use?

I want to try what you did... did you replace those libraries and run ldconfig?

Then, what MPV command did you use? And, was it the Bookworm provided MPV?

[jock]

On 8/11/2023 at 8:36 PM, Khánh Ngô said:

Can you compile again for jammy with that feature support, please? 

 

That's great for those who want to listen to music and videos from YouTube like me.

Unfortunately, it is not that simple. v4l2 (ie: kernel) was a moving target in those days, and ffmpeg also was a development version from libreelec project, and things have changed a lot in the meanitme. Replicating the exact setup of that build is very difficult because many pieces have to come together.

 

@baryon it requires to align the kernel user-space headers, a patched version of ffmpeg (v4l2-request and drmprime APIs have not been mainlined yet into ffmpeg, unfortunately). mpv is the easiest piece of code to compile and get it ready here, but the final binary will depend upon:

• the kernel version, probably kernels >= 6.1 won't have problems because API finally got frozen, but older kernels may have issues
• the distribution, because mpv uses ffmpeg and ffmpeg is dependant upon a slew of dinamically linked libraries that are often distro-specific (read: libx264 and libx265 to the most); sometimes soft-linking libraries is a workable workaround, but not always

 

I may give a shot and compile on a more recent kernel/distro, but the last time I did there were some regressions that may have not yet fixed. Plus it takes a lot of time because compiling ffmpeg in particular requires a lot of time.

 

[baryon]

Oh no, @robertoj I made an error. I did not mean to mislead. I meant to say that it works on my rk322x-box, with that version. I neglected that datum. I don't use windowing environments on my box, so it was the 

mpv-armhf --vo=gpu --hwdec=drm --gpu-hwdec-interop=drmprime-drm --drm-draw-plane=overlay --drm-drmprime-video-plane=primary <video.mp4>

command. Actually, it needs to be modified because that command gives the warning

[vo/gpu] gpu-hwdec-interop was selected with the legacy name 'drmprime-drm'. Please change it to 'drmprime-overlay' in your config file or command line.

Fortunately, it tells you exactly what to do, and I can affirm that drmprime-overlay works without the warning. H264 1080p60 over HDMI works fine, even coupled to streamlink. H265 did not work fine. I am not equipped to test more highly.

 

Thank you for the detail, @jock. Trying to get things working naively, I managed to get the FFMPEG build with rkmpp support from https://github.com/hbiyik/FFmpeg linked against librga and libmpp, but that requires the old kernel, apparently. It is very difficult to find any exposition as to how these various iterations of the various subsystems work!

Edited October 18, 2023 by baryon
detail

[robertoj]

 i thought someone had already made it work in the H3

 

In the H3, the problem is not the accelerated H264 decoding. The problem is MPV not able to display the frames with the GPU... I need time to get back in developer mode and play with Mesa configurations and rebuild.

[baryon]

I have reproduced jock's build. He described how to do it adequately, but I'll detail my experience, as I filled in a few blanks myself. I am on the 6.1.50-current-rk322x kernel.

1. Build ilmich's LibreELEC branch ( https://github.com/ilmich/LibreELEC.tv ) on my PC
2. After it completes, copy the ffmpeg-4.4.1 build folder to my rk3229
3. Reconfigure the ffmpeg build (see below)
4. make then make install ffmpeg
5. Checkout c5b258b490f55c19a1cf6060bc7cf796d64f71e9 from the main MPV git
6. meson and ninja mpv into existence

The configure invocation I used differs from the LibreELEC build, but nothing big.

 

./configure --cpu=cortex-a7 --arch=arm --enable-static --enable-shared --enable-gpl --enable-version3 --enable-logging --disable-doc --disable-debug --enable-stripping --enable-pic --enable-optimizations --disable-extra-warnings --enable-avdevice --enable-avcodec --enable-avformat --enable-swscale --enable-postproc --enable-avfilter --disable-gnutls --enable-openssl --disable-gray --enable-swscale-alpha --disable-small --enable-dct --enable-fft --enable-mdct --enable-rdft --disable-crystalhd --enable-v4l2_m2m --enable-libdrm --enable-libudev --enable-v4l2-request --enable-libdrm --disable-vdpau --enable-runtime-cpudetect --disable-hardcoded-tables  --enable-hwaccels  --enable-demuxers --enable-parsers --enable-bsfs --enable-filters --disable-avisynth --enable-bzlib --disable-lzma --enable-libdav1d --enable-libspeex  --enable-zlib --enable-asm --disable-altivec --enable-neon --disable-symver

 

I tried the master branch of mpv. It did not work, but the checkout Jock used (and I documented) did. Thanks again for your work, jock.

 

Lowest hanging fruit here is to get rid of the LibreELEC stage and just have a nice ffmpeg git repo someone can clone to save the awkward steps 1 and 2. I hope to report back with a link to such!

[usual user]

1 hour ago, baryon said:

to get rid of the LibreELEC stage and just have a nice ffmpeg git repo someone can clone to save the awkward steps 1 and 2.

To build ffmpeg with the request API, you usually only need the additional commits  that Jernej provides.

[jock]

@baryon actually you don't need to take @ilmich's branch 4.4.1, but you can use the ffmpeg 6.0 coming from libreelec's master branch: the package details, it's source and the patches to apply are self-explained in the package.mk file; compilation flags also are in the package.mk, you just need to assemble them "manually"

 

 

[Fadis Salman]

@jock Please, can you tell how to build ffmpeg 6.0 from libreelec's master branch?

[jock]

Hello, for those still interested in hardware video decoding, I made a new thread introducing an APT repository to make installation easier for Armbian Debian Bookworm and Ubuntu Jammy distros: