Repository for v4l2request hardware video decoding (rockchip, allwinner)

[jock]

Hello, this quick tutorial is to introduce an experimental Debian and Ubuntu APT repository to install ffmpeg and mpv compiled with v4l2request and v4l2drmprime patches developed by Linux kernel, LIbreELEC and Kodi folks to allow hardware video decoding on stateless decoders like those implemented in Rockchip and Allwinner SoCs for h.264, h.265, vp8 and vp9 codecs.

 

The repository introduces a new package ffmpeg-v4l2request that integrates and substitues the base ffmpeg package and its related packages.

Also provides mpv 0.35.1 for Ubuntu Jammy, which has an overrall better support for hardware video decoders.

 

Preconditions:

• Kernel should be 6.1 or more recent
• armhf or arm64 architecture
• Supported operating systems are Debian Bookworm and Ubuntu Jammy
• Rockchip and Allwinner have already been tested, but this should work on other platforms with stateless decoders supported in kernel

 

APT REPOSITORY SETUP

To install the repository, just copy and paste the lines for your operating system in a terminal

 

For Debian Bookworm:

$ sudo wget http://apt.undo.it:7241/apt.undo.it.asc -O /etc/apt/trusted.gpg.d/apt.undo.it.asc
$ echo "deb http://apt.undo.it:7241/debian bookworm main" | sudo tee /etc/apt/sources.list.d/apt.undo.it.list

 

For Ubuntu Jammy:

$ sudo wget http://apt.undo.it:7241/apt.undo.it.asc -O /etc/apt/trusted.gpg.d/apt.undo.it.asc
$ echo "deb http://apt.undo.it:7241/ubuntu jammy main" | sudo tee /etc/apt/sources.list.d/apt.undo.it.list

 

INSTALL FFMPEG AND MPV PACKAGES

$ sudo apt update
$ sudo apt install ffmpeg-v4l2request mpv

 

SETUP MPV CONFIG FILE

$ sudo mkdir -p /etc/mpv
$ echo -e "hwdec=drm\ndrm-drmprime-video-plane=primary\ndrm-draw-plane=overlay" | sudo tee /etc/mpv/mpv.conf

 

You can now play your videos using mpv and they should run with hardware decoding if supported, either in virtual terminals or in X11/Wayland windows!

Enjoy!

 

Notes:

• your mileage may vary a lot: the more recent is the kernel version, the better is support (you may need edge kernel)
• bug: when rendered in X11/Wayland window, video may show scattered tiles during frames
• bug: Lima driver (Mali 400/450) shows a red/pink tint when video is played in X11/Wayland (see https://github.com/mpv-player/mpv/issues/12968) (workaround below: https://forum.armbian.com/topic/32449-repository-for-v4l2request-hardware-video-decoding-rockchip-allwinner/?do=findComment&comment=177968)
• Panfrost driver should work flawlessy
• 10 bit HEVC are generally supported on all Rockchip devices (rk322x, rk3288, rk33x8, rk3399), but Allwinner H3 have no hardware support for that

 

 

 

[jock]

Post reserved

[baryon]

Ahahaha! Nice! You beat me to it. This morning, I forked the debian-ffmpeg repo to my github and was just dropping by to see what the fella on the other thread was running so I could make a patchset branch for him and make it very simple for him. I just got the 6.0.1 ffmpeg patchset working this weekend. This is even better.

[Werner]

Just tried this on the Station P1 with kernel 6.6.7 (including https://github.com/armbian/build/pull/6051) and Bookworm. x264 works but hevc does not.  Tried a random movie encoded in x265

[ffmpeg/video] hevc: v4l2_request_probe_video_device: select capture format failed
[ffmpeg/video] hevc: Failed setup for format drm_prime: hwaccel initialisation returned error.

ffprobe of the same file says

Stream #0:0: Video hevc (Main 10), yuv420p10le(tv, bt709), 3840x2160 [SAR 1:1 DAR 16:9] ....

 

On a different one, also x265 I get

 

[ffmpeg/video] hevc: ff_v4l2_request_query_control: query control failed, Invalid argument (22)

 

Probably just doing something wrong...

[jock]

@Werner

mpv or ffmpeg may show some "failed" messages, but they appear when they probe v4l2 devices for features and codec support; in case the combo does not work, they report an error and pass to probe the next v4l2 device until they find something valid. You can see similar messages in the screenshots of my PR, but then if you get "Using hardware decoding (drm)" you should be ok.

 

Does the video play with high cpu usage or it does not play at all?

 

 

 

 

[haven]

Regarding this bug:

bug: Lima driver (Mali 400/450) shows a red/pink tint when video is played in X11/Wayland (see https://github.com/mpv-player/mpv/issues/12968)

 

I found that forcing MESA to a different GL core profile can solve the pinkish problem:

export MESA_GL_VERSION_OVERRIDE=3.2COMPAT
export MESA_GLSL_VERSION_OVERRIDE=320

mpv your-video.mkv

 

[jock]

@haven Cool workaround, thanks! Can be also useful to pinpoint the bug/missing feature

[Werner]

Seems to work. Still has a few dropped frames. CPU usage is about 15%.

[haven]

have 3 scenarios:

• perfetct when launching mpv from terminal with gpu-context=drm
• some frame scattering when using mpv windowed in X11
• too many frame dropped with mpv -fs in X11
• perfect with mpv -fs in Xwayland

 

I don't like wayland too much, and I would prefer X11, but seems to have no choice.

Edited December 31, 2023 by haven

[ning]

do you try mpv -vo=gpu-next? I got system reset after seconds on tinker-rk3288.

 

root caused to not enough power, I change power supply to 5v3a USB changer, the issue gone.

[jock]

9 hours ago, ning said:

do you try mpv -vo=gpu-next? I got system reset after seconds on tinker-rk3288.

vo=gpu-next fails on my setups, but the rk3288 tinkerboard-s was one of my test beds and it did not show any reset issue. Also other boards did not show any particular stability issues.

[ning]

14 hours ago, jock said:

vo=gpu-next fails on my setups, but the rk3288 tinkerboard-s was one of my test beds and it did not show any reset issue. Also other boards did not show any particular stability issues.

you can try:  kernel6.6.8+ffmpeg6.1+mpv0.37+mesa23.3.2 with v4l2 patches.

 

if use -ov=gpu, still need WR for lima.

 

 

[jock]

2 hours ago, ning said:

you can try:  kernel6.6.8+ffmpeg6.1+mpv0.37+mesa23.3.2 with v4l2 patches.

 

if use -ov=gpu, still need WR for lima.

But that's not a supported setup for ubuntu jammy or debian bookworm with this repository, perhaps it is better to discuss in a separate thread?

[ning]

commit 78285e98f1479ba27d652c9f79329fee5eb1add2
Author: Philip Langdale <philipl@overt.org>
Date:   Thu Jun 22 12:43:02 2023 -0700

    vo: hwdec: prioritise `drmprime` over `drmprime_overlay`
    
    I originally left `drmprime_overlay` as higher priority because
    `drmprime` was new, and because I didn't have any hardware where both
    worked (only one or the other) so I couldn't compare relative
    performance, and if only one worked, the priority didn't matter.
    
    But with time and more usage, we've reached a point where we can say we
    would recommend using `drmprime` in situations where both work, and
    we've also been able to identify hardware where both do indeed work and
    it seems that `drmprime` is more reliable.
    
    So, let's flip them.

diff --git a/video/out/gpu/hwdec.c b/video/out/gpu/hwdec.c
index 1e3edb0d8d..878ac148fb 100644
--- a/video/out/gpu/hwdec.c
+++ b/video/out/gpu/hwdec.c
@@ -74,8 +74,8 @@ const struct ra_hwdec_driver *const ra_hwdec_drivers[] = {
     &ra_hwdec_rpi_overlay,
 #endif
 #if HAVE_DRM
-    &ra_hwdec_drmprime_overlay,
     &ra_hwdec_drmprime,
+    &ra_hwdec_drmprime_overlay,
 #endif
 #if HAVE_ANDROID_MEDIA_NDK
     &ra_hwdec_aimagereader,

just let know, when use mpv 0.36 on H3, need revert this patch to make mpv work.

[jock]

7 hours ago, ning said:

just let know, when use mpv 0.36 on H3, need revert this patch to make mpv work.

Ah ok, thanks for this, I'll keep in mind with newer versions.

 

[ning]

after study,

I find there 2 way can make mpv work good on RK3288

 

1,  mpv.conf
 

Quote

vo=gpu
hwdec=auto
gpu-hwdec-interop=drmprime-overlay

 

2, mpv.conf

Quote

vo=gpu-next
hwdec=auto

with a patch:

diff --git a/filters/f_hwtransfer.c b/filters/f_hwtransfer.c
index 26b1daaedc..17aaa56ef2 100644
--- a/filters/f_hwtransfer.c
+++ b/filters/f_hwtransfer.c
@@ -348,7 +348,8 @@ static bool probe_formats(struct mp_filter *f, int hw_imgfmt, bool use_conversio
         if (!cstr) {
             MP_VERBOSE(f, "hwdec '%s' does not report hwframe constraints. "
                           "Using static metadata.\n", cur->driver_name);
-            cstr = build_static_constraints(cur);
+            //cstr = build_static_constraints(cur);
+           continue;
         }
         bool found = false;
         for (int i = 0; cstr->valid_hw_formats &&

 

recomend is 1.

[Sean Perez]

Ive been trying to figure something out with the orangepi zero2w, specifically for video decoding.
Ive gotten the 3d engine working and benching fine with opengl, and I installed this when you first posted it for the hardware video acceleration, but what I been trying to figure out in general for the GPU and or the video, is the CMA memory... I think this might be the limiting factor in all this, and is a big oversight on alot of this video stuff. Im not running armbian at the moment but on the orangepi os running 6.1 debian

I run this

cat /proc/meminfo | grep -i cma

CmaTotal:         131072 kB

CmaFree:          117696 kB

(128 mB)

so depending on the different kernel / SoC builds debian, armbian, custom stuff... everyone probably has something different.
And from what I have read in other posts here and within the Raspberry Pi community, the CMA allocation seems to be the most important part for video and 3d hardware acceleration, since its not in the "user space" it technically not have access to the whole RAM only the CMA (128mB in my case).
I think 128 is enough but if your running some thing crazy it could need more as in other post in this forums that people have pushed 1GB ram board to a max of 512mB~

Ive read that the CMA can be in the device tree which seems not to be the case for the orangepi zero2w debian 6.1, so im trying to figure out where else I can change that, I figure I post this here, maybe someone has more insight or has tried changing the CMA for this driver and has verified results?

[jock]

@Sean Perez

Sorry for the very late answer; I'm not sure the orange pi debian contains the proper patches or the proper device tree things; I strongly suggest you to use on of the latest armbian images.

 

I tested it on Allwinner H3 with 1gb of RAM and it worked well, but don't know the status of the hardware video decoders of H618. Probably it is better to use a kernel 6.6 and ask support in the allwinner forums just to know what kind of hardware decoders have.

 

Don't know if CMA is an issue, 128mb should be more than enough to decode full-hd content. rockchip has no need for large CMA buffers since hardware decoders have their own MMUs. If allwinner has a similar design should even not need CMA buffers. Raspberry Pi (1, 2, 3 and probably 4) GPU/VPU instead has no MMU, so it requires CMA for video decoding and 3d graphics.

[plainlethal]

Hi,

 

Interesting repo

Its capable of h264 hw encoding?

[plainlethal]

This not working for me
Which parameter you pass to ffmpeg?

[plainlethal]

This seems to be discontinued.

it was fun while it lasted 🙂

[Werner]

21 hours ago, plainlethal said:

This not working for me

Elaborate

[jock]

On 2/14/2024 at 6:08 PM, plainlethal said:

Its capable of h264 hw encoding?

No.

[anagram0496]

Hi, thanks for the efforts with this. Just tried this on my MXQ Pro V72 (3228A) with latest Armbian w/ XFCE. Unfortunately, any playback on mpv within xfce did not have good performance.

 

In terms of hardware decode, only h264 8-bit could run w/ hw decode. Meanwhile 10-bit H264 and HEVC did not run with HW decode at all. All files were tested at 1080p.

 

Playback performance was only good when run in virtual terminal, but on XFCE, even 8bit H264 stuttered, not to mention the pink/red color issue (which was fixed with the additional mesa flags).

 

I thought it was a throttling issue, since xfce sensors report 80C, so I added a fan blowing from under the tv box and the temps are now within 55-70C. However that did not fix the issue. Is there anything else I could check or try? Thanks.

[sakesm]

Hello, everyone.

@jock Thank you very much for the tutorial that allowed me to experience video acceleration in h618.

Then I would like to know where the V4L2 patch is approximately, interested in this

[jock]

On 5/10/2024 at 11:07 AM, anagram0496 said:

I thought it was a throttling issue, since xfce sensors report 80C, so I added a fan blowing from under the tv box and the temps are now within 55-70C. However that did not fix the issue. Is there anything else I could check or try? Thanks.

 

You should first try to disable xfce compositing, which is very taxing and pretty useless.

Without composition, videos should be much smoother, but take note that rk3228 is very limited in terms of resources so windowed playback is acceptable perhaps up to 1080p25/1080p30, anything above won't work well and even 1080p may have some smoothness issues. 8bit/10bit should not matter though.

[jock]

8 hours ago, sakesm said:

Then I would like to know where the V4L2 patch is approximately, interested in this

Most of the patches are in the kernel mailing lists. ffmpeg patches have been taken from mainline LibreELEC project.

[resi_raider]

Kinda late to the party here, I'm trying to download the apt repo but my connection keeps timing out. Is the repo still up?

[jock]

On 6/28/2024 at 4:23 PM, resi_raider said:

Kinda late to the party here, I'm trying to download the apt repo but my connection keeps timing out. Is the repo still up?

Yes, it is

[alejor]

Hi, I tried this with armbian bookworm and orange pi zero 3 worked flawlessly with h264 hwdec. Now... Armbian image are Ubuntu Noble 24.04, can you please create or recompile packages for that 24.04 distro? or give some instructions to create those packages? thank you!