Mainline VPU


Myy
 Share

17 17

Recommended Posts

Armbian is a community driven open source project. Do you like to contribute your code?

3 hours ago, rubenvb said:

only the "outdated" (mismatching with kernels >=5.11) in the LibreElec repos.

Yeah, LE currently uses 5.10 kernel because we are preparing stable release (no ETA, but hopefully soon). Only after that Linux will be updated to whatever current stable release will be at the time. However, v4l2 request api ffmpeg patches for Linux 5.13 can be found here. Note that they are untested and you still need corresponding kernel patches too.

Link to post
Share on other sites

On 7/6/2021 at 5:04 PM, jernej said:

However, v4l2 request api ffmpeg patches for Linux 5.13 can be found here. Note that they are untested and you still need corresponding kernel patches too.

I just tested this with vanilla Linux 5.13.7 and at least H264 playback with DRM Prime Direct To Plane rendering in Kodi 19.1 works as expected.

HEVC content still just doesn't render at all, but maybe I need some extra kernel patches for that still.

Thanks for the link to the up to date FFmpeg patches, you're a lifesaver!

Link to post
Share on other sites

20 hours ago, rubenvb said:

HEVC content still just doesn't render at all, but maybe I need some extra kernel patches for that still.

Pick patches 14-23 here: https://github.com/jernejsk/LibreELEC.tv/tree/linux-5.13/projects/Allwinner/patches/linux

 

With that, HEVC should work without problem. Official HEVC API still needs some work.

 

20 hours ago, rubenvb said:

Thanks for the link to the up to date FFmpeg patches, you're a lifesaver!

Not a problem, I usually update them with every stable kernel release, just to check on progress.

Link to post
Share on other sites

On 8/6/2021 at 2:18 AM, rubenvb said:

I just tested this with vanilla Linux 5.13.7 and at least H264 playback with DRM Prime Direct To Plane rendering in Kodi 19.1 works as expected.

HEVC content still just doesn't render at all, but maybe I need some extra kernel patches for that still.

Thanks for the link to the up to date FFmpeg patches, you're a lifesaver!

I am trying to compile FFmpeg on Armbian with 5.13.12  kernel. I am getting compilation errors. maybe I did configuration wrong. what configure parameters you use?

Link to post
Share on other sites

I am building FFmpeg from here on a Orange Pi 4 with Armbian Hirsute edge 5.13.12 kernel. I have configured FFmpeg with following parameters.

 

./configure --enable-libdrm --enable-v4l2-request --enable-libudev --enable-static --disable-shared

 

I am getting these errors on compile.

 

AR	libavdevice/libavdevice.a
AR	libavfilter/libavfilter.a
AR	libavformat/libavformat.a
CC	libavcodec/v4l2_request_mpeg2.o
CC	libavcodec/vaapi_encode.o
CC	libavcodec/vaapi_encode_h264.o
CC	libavcodec/vaapi_encode_h265.o
libavcodec/v4l2_request_mpeg2.c:24:37: error: field ‘sequence’ has incomplete type
   24 |     struct v4l2_ctrl_mpeg2_sequence sequence;
      |                                     ^~~~~~~~
libavcodec/v4l2_request_mpeg2.c:25:36: error: field ‘picture’ has incomplete type
   25 |     struct v4l2_ctrl_mpeg2_picture picture;
      |                                    ^~~~~~~
libavcodec/v4l2_request_mpeg2.c:26:41: error: field ‘quantisation’ has incomplete type
   26 |     struct v4l2_ctrl_mpeg2_quantisation quantisation;
      |                                         ^~~~~~~~~~~~
libavcodec/v4l2_request_mpeg2.c: In function ‘v4l2_request_mpeg2_start_frame’:
libavcodec/v4l2_request_mpeg2.c:39:10: error: ‘struct v4l2_ctrl_mpeg2_sequence’ has no member named ‘horizontal_size’
   39 |         .horizontal_size = s->width,
      |          ^~~~~~~~~~~~~~~
libavcodec/v4l2_request_mpeg2.c:39:28: warning: excess elements in struct initializer
   39 |         .horizontal_size = s->width,
      |                            ^
libavcodec/v4l2_request_mpeg2.c:39:28: note: (near initialization for ‘(anonymous)’)
libavcodec/v4l2_request_mpeg2.c:40:10: error: ‘struct v4l2_ctrl_mpeg2_sequence’ has no member named ‘vertical_size’
   40 |         .vertical_size = s->height,
      |          ^~~~~~~~~~~~~
libavcodec/v4l2_request_mpeg2.c:40:26: warning: excess elements in struct initializer
   40 |         .vertical_size = s->height,
      |                          ^
libavcodec/v4l2_request_mpeg2.c:40:26: note: (near initialization for ‘(anonymous)’)
libavcodec/v4l2_request_mpeg2.c:41:10: error: ‘struct v4l2_ctrl_mpeg2_sequence’ has no member named ‘vbv_buffer_size’
   41 |         .vbv_buffer_size = req->output.size,
      |          ^~~~~~~~~~~~~~~
libavcodec/v4l2_request_mpeg2.c:41:28: warning: excess elements in struct initializer
   41 |         .vbv_buffer_size = req->output.size,
      |                            ^~~
libavcodec/v4l2_request_mpeg2.c:41:28: note: (near initialization for ‘(anonymous)’)
libavcodec/v4l2_request_mpeg2.c:44:10: error: ‘struct v4l2_ctrl_mpeg2_sequence’ has no member named ‘profile_and_level_indication’
   44 |         .profile_and_level_indication = 0,
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
libavcodec/v4l2_request_mpeg2.c:44:41: warning: excess elements in struct initializer
   44 |         .profile_and_level_indication = 0,
      |                                         ^
libavcodec/v4l2_request_mpeg2.c:44:41: note: (near initialization for ‘(anonymous)’)
libavcodec/v4l2_request_mpeg2.c:45:10: error: ‘struct v4l2_ctrl_mpeg2_sequence’ has no member named ‘chroma_format’
   45 |         .chroma_format = s->chroma_format,
      |          ^~~~~~~~~~~~~
libavcodec/v4l2_request_mpeg2.c:45:26: warning: excess elements in struct initializer
   45 |         .chroma_format = s->chroma_format,
      |                          ^
libavcodec/v4l2_request_mpeg2.c:45:26: note: (near initialization for ‘(anonymous)’)
libavcodec/v4l2_request_mpeg2.c:37:60: error: invalid use of undefined type ‘struct v4l2_ctrl_mpeg2_sequence’
   37 |     controls->sequence = (struct v4l2_ctrl_mpeg2_sequence) {
      |                                                            ^
libavcodec/v4l2_request_mpeg2.c:49:37: error: ‘V4L2_MPEG2_SEQ_FLAG_PROGRESSIVE’ undeclared (first use in this function); did you mean ‘V4L2_DV_BT_CAP_PROGRESSIVE’?
   49 |         controls->sequence.flags |= V4L2_MPEG2_SEQ_FLAG_PROGRESSIVE;
      |                                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                                     V4L2_DV_BT_CAP_PROGRESSIVE
libavcodec/v4l2_request_mpeg2.c:49:37: note: each undeclared identifier is reported only once for each function it appears in
libavcodec/v4l2_request_mpeg2.c:53:10: error: ‘struct v4l2_ctrl_mpeg2_picture’ has no member named ‘picture_coding_type’
   53 |         .picture_coding_type = s->pict_type,
      |          ^~~~~~~~~~~~~~~~~~~
libavcodec/v4l2_request_mpeg2.c:53:32: warning: excess elements in struct initializer
   53 |         .picture_coding_type = s->pict_type,
      |                                ^
libavcodec/v4l2_request_mpeg2.c:53:32: note: (near initialization for ‘(anonymous)’)
libavcodec/v4l2_request_mpeg2.c:56:10: error: ‘struct v4l2_ctrl_mpeg2_picture’ has no member named ‘f_code’
   56 |         .f_code[0][0] = s->mpeg_f_code[0][0],
      |          ^~~~~~
libavcodec/v4l2_request_mpeg2.c:56:25: warning: excess elements in struct initializer
   56 |         .f_code[0][0] = s->mpeg_f_code[0][0],
      |                         ^
libavcodec/v4l2_request_mpeg2.c:56:25: note: (near initialization for ‘(anonymous)’)
libavcodec/v4l2_request_mpeg2.c:57:10: error: ‘struct v4l2_ctrl_mpeg2_picture’ has no member named ‘f_code’
   57 |         .f_code[0][1] = s->mpeg_f_code[0][1],
      |          ^~~~~~
libavcodec/v4l2_request_mpeg2.c:57:25: warning: excess elements in struct initializer
   57 |         .f_code[0][1] = s->mpeg_f_code[0][1],
      |                         ^
libavcodec/v4l2_request_mpeg2.c:57:25: note: (near initialization for ‘(anonymous)’)
libavcodec/v4l2_request_mpeg2.c:58:10: error: ‘struct v4l2_ctrl_mpeg2_picture’ has no member named ‘f_code’
   58 |         .f_code[1][0] = s->mpeg_f_code[1][0],
      |          ^~~~~~
libavcodec/v4l2_request_mpeg2.c:58:25: warning: excess elements in struct initializer
   58 |         .f_code[1][0] = s->mpeg_f_code[1][0],
      |                         ^
libavcodec/v4l2_request_mpeg2.c:58:25: note: (near initialization for ‘(anonymous)’)
libavcodec/v4l2_request_mpeg2.c:59:10: error: ‘struct v4l2_ctrl_mpeg2_picture’ has no member named ‘f_code’
   59 |         .f_code[1][1] = s->mpeg_f_code[1][1],
      |          ^~~~~~
libavcodec/v4l2_request_mpeg2.c:59:25: warning: excess elements in struct initializer
   59 |         .f_code[1][1] = s->mpeg_f_code[1][1],
      |                         ^
libavcodec/v4l2_request_mpeg2.c:59:25: note: (near initialization for ‘(anonymous)’)
libavcodec/v4l2_request_mpeg2.c:60:10: error: ‘struct v4l2_ctrl_mpeg2_picture’ has no member named ‘picture_structure’
   60 |         .picture_structure = s->picture_structure,
      |          ^~~~~~~~~~~~~~~~~
libavcodec/v4l2_request_mpeg2.c:60:30: warning: excess elements in struct initializer
   60 |         .picture_structure = s->picture_structure,
      |                              ^
libavcodec/v4l2_request_mpeg2.c:60:30: note: (near initialization for ‘(anonymous)’)
libavcodec/v4l2_request_mpeg2.c:61:10: error: ‘struct v4l2_ctrl_mpeg2_picture’ has no member named ‘intra_dc_precision’
   61 |         .intra_dc_precision = s->intra_dc_precision,
      |          ^~~~~~~~~~~~~~~~~~
libavcodec/v4l2_request_mpeg2.c:61:31: warning: excess elements in struct initializer
   61 |         .intra_dc_precision = s->intra_dc_precision,
      |                               ^
libavcodec/v4l2_request_mpeg2.c:61:31: note: (near initialization for ‘(anonymous)’)
libavcodec/v4l2_request_mpeg2.c:51:58: error: invalid use of undefined type ‘struct v4l2_ctrl_mpeg2_picture’
   51 |     controls->picture = (struct v4l2_ctrl_mpeg2_picture) {
      |                                                          ^
libavcodec/v4l2_request_mpeg2.c:65:36: error: ‘V4L2_MPEG2_PIC_FLAG_TOP_FIELD_FIRST’ undeclared (first use in this function)
   65 |         controls->picture.flags |= V4L2_MPEG2_PIC_FLAG_TOP_FIELD_FIRST;
      |                                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
libavcodec/v4l2_request_mpeg2.c:68:36: error: ‘V4L2_MPEG2_PIC_FLAG_FRAME_PRED_DCT’ undeclared (first use in this function)
   68 |         controls->picture.flags |= V4L2_MPEG2_PIC_FLAG_FRAME_PRED_DCT;
      |                                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
libavcodec/v4l2_request_mpeg2.c:71:36: error: ‘V4L2_MPEG2_PIC_FLAG_CONCEALMENT_MV’ undeclared (first use in this function)
   71 |         controls->picture.flags |= V4L2_MPEG2_PIC_FLAG_CONCEALMENT_MV;
      |                                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
libavcodec/v4l2_request_mpeg2.c:74:36: error: ‘V4L2_MPEG2_PIC_FLAG_INTRA_VLC’ undeclared (first use in this function)
   74 |         controls->picture.flags |= V4L2_MPEG2_PIC_FLAG_INTRA_VLC;
      |                                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
libavcodec/v4l2_request_mpeg2.c:77:36: error: ‘V4L2_MPEG2_PIC_FLAG_Q_SCALE_TYPE’ undeclared (first use in this function)
   77 |         controls->picture.flags |= V4L2_MPEG2_PIC_FLAG_Q_SCALE_TYPE;
      |                                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
libavcodec/v4l2_request_mpeg2.c:80:36: error: ‘V4L2_MPEG2_PIC_FLAG_ALT_SCAN’ undeclared (first use in this function)
   80 |         controls->picture.flags |= V4L2_MPEG2_PIC_FLAG_ALT_SCAN;
      |                                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
libavcodec/v4l2_request_mpeg2.c:83:36: error: ‘V4L2_MPEG2_PIC_FLAG_REPEAT_FIRST’ undeclared (first use in this function)
   83 |         controls->picture.flags |= V4L2_MPEG2_PIC_FLAG_REPEAT_FIRST;
      |                                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
libavcodec/v4l2_request_mpeg2.c:86:36: error: ‘V4L2_MPEG2_PIC_FLAG_PROGRESSIVE’ undeclared (first use in this function); did you mean ‘V4L2_DV_BT_CAP_PROGRESSIVE’?
   86 |         controls->picture.flags |= V4L2_MPEG2_PIC_FLAG_PROGRESSIVE;
      |                                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                                    V4L2_DV_BT_CAP_PROGRESSIVE
libavcodec/v4l2_request_mpeg2.c: In function ‘v4l2_request_mpeg2_end_frame’:
libavcodec/v4l2_request_mpeg2.c:121:19: error: ‘V4L2_CID_STATELESS_MPEG2_SEQUENCE’ undeclared (first use in this function); did you mean ‘V4L2_CID_STATELESS_VP8_FRAME’?
  121 |             .id = V4L2_CID_STATELESS_MPEG2_SEQUENCE,
      |                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                   V4L2_CID_STATELESS_VP8_FRAME
libavcodec/v4l2_request_mpeg2.c:126:19: error: ‘V4L2_CID_STATELESS_MPEG2_PICTURE’ undeclared (first use in this function); did you mean ‘V4L2_CID_STATELESS_VP8_FRAME’?
  126 |             .id = V4L2_CID_STATELESS_MPEG2_PICTURE,
      |                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                   V4L2_CID_STATELESS_VP8_FRAME
libavcodec/v4l2_request_mpeg2.c:131:19: error: ‘V4L2_CID_STATELESS_MPEG2_QUANTISATION’ undeclared (first use in this function)
  131 |             .id = V4L2_CID_STATELESS_MPEG2_QUANTISATION,
      |                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
make: *** [ffbuild/common.mak:59: libavcodec/v4l2_request_mpeg2.o] Error 1
make: *** Waiting for unfinished jobs....

 

Link to post
Share on other sites

FFmpeg build fine with v4l2-request-hwaccel-4.3.1-new branch. The media file bbb_sunflower_1080p_30fps_normal.mp4 decoding fine at 90fps with one A72 core at 100% uses.

 

I have tried building MPV player with it but it doesn't look like using the VPU. A72 cores are at 30-40% and A53 are around 15-20%. How can I check if it actually using VPU?

 

MPV playback log file.

 

mpv_sucks.log

Link to post
Share on other sites

21 hours ago, jernej said:

You are using sources meant to be used on 5.14.

A few days ago I also tried to build a ffmpeg with v4l2-request-hwaccel. Because my distribution already uses 4.4, I used these patches. Initially, I got similar error messages as @kprasadvnsi, but the 5.14 kernel header hint was the missing link. I also have a flawlessly compiled ffmpeg package now, but I also don't know if additional patches are needed for other mainline components or how to check the v4l2-request-hwaccel functionality.

Link to post
Share on other sites

Test command would be something like:

ffmpeg -loglevel debug -hwaccel drm -hwaccel_output_format drm_prime -i video.mkv -f null -

 

I think it should be pretty self evident from the log if request api is used or not.

 

Notes:

1. If you don't use LE kernel patches, decoding interlaced H264 content won't work on RK. HEVC also won't work without kernel patches.

2. mpv most probably needs patching. At least it did in the past.

Link to post
Share on other sites

37 minutes ago, jernej said:

Test command would be something like:

ffmpeg -loglevel debug -hwaccel drm -hwaccel_output_format drm_prime -i video.mkv -f null -

 

I think it should be pretty self evident from the log if request api is used or not.

 

Notes:

1. If you don't use LE kernel patches, decoding interlaced H264 content won't work on RK. HEVC also won't work without kernel patches.

2. mpv most probably needs patching. At least it did in the past.

 

I am getting much better performance using your test commands. I am getting 235 fps at 5% average CPU load.

Where should I look for patching in the mpv to get this working?

Link to post
Share on other sites

I don't know what I really need to look for, but since "v4l2_request" in ffmpeg.log is mentioned several times, I think "v4l2-request-hwaccel" is present.

1 hour ago, jernej said:

If you don't use LE kernel patches, decoding interlaced H264 content won't work on RK. HEVC also won't work without kernel patches.

Guess now I have to figure where LE is hideing the current 5.14.0 kernel patches.

Link to post
Share on other sites

1 hour ago, kprasadvnsi said:

I am getting much better performance using your test commands. I am getting 235 fps at 5% average CPU load.

Note that this is pure decoding speed (which can be optimized further). Rendering may add additional overhead.

 

1 hour ago, kprasadvnsi said:

Where should I look for patching in the mpv to get this working?

Last time I heard, this patch was needed: https://github.com/Kwiboo/mpv/commit/91f2b090eef12f48cf1d9e9157e5eefe6a60be6a If this doesn't work, then I don't know.

 

Note that ffmpeg request api support is still experimental. It's quiet likely patches will be rewritten before they will be submitted to ffmpeg mailing list again. That's why there is no official support for it in any player. You are very lucky that somebody in the past actually looked into mpv.

 

1 hour ago, usual user said:

Guess now I have to figure where LE is hideing the current 5.14.0 kernel patches.

LE switched to 5.14 last week, so patches are already in. RK patches are here: https://github.com/LibreELEC/LibreELEC.tv/tree/master/projects/Rockchip/patches/linux/default I suggest taking at least v4l patches.

Link to post
Share on other sites

Quote

Last time I heard, this patch was needed: https://github.com/Kwiboo/mpv/commit/91f2b090eef12f48cf1d9e9157e5eefe6a60be6a If this doesn't work, then I don't know.

 

These changes are already merged in upstream mpv. The patch was for cedrus driver and tested on a Allwinner H3 device with following parameters.

--vo=gpu --gpu-context=drm --hwdec=drm

 

Using these parameters with my mpv build it gives the following error.

kp@orangepi4:~/trails/mpv-build/mpv/build$ ./mpv --log-file=mpv_sucks.log --vo=gpu --gpu-context=drm --hwdec=drm  bbb_sunflower_1080p_30fps_normal.mp4 
Error parsing option gpu-context (option parameter could not be parsed)
Setting commandline option --gpu-context=drm failed.
Exiting... (Fatal error)

 

The only option that didn't throw error is --gpu-context=x11egl 

Link to post
Share on other sites

9 hours ago, kprasadvnsi said:

These changes are already merged in upstream mpv. The patch was for cedrus driver

Ah, good to know. BTW, that's not only for Cedrus but for rkvdec and Hantro drivers too. They all provide same interface.

9 hours ago, kprasadvnsi said:

Using these parameters with my mpv build it gives the following error.

It would be nice to see that more detailed log.

Link to post
Share on other sites

My kernel has now 0010, 0011, 0020, 1000, 1001, 2000 and 2001 LE patch applied.
I don't see any significant change in behavior, so I can't say if the ffmpeg-driven video pipeline realy uses hardware acceleration.

The mpv command of @kprasadvnsi is playing for me. In the mpv.log the Codec list mentions, "h264_v4l2m2m (h264) - V4L2 mem2mem H.264 decoder wrapper", but I don't know if this is used. Neither I don't know how to select which decoder (hantro, rkvdec) will be used, it's the first time I use mpv.

Link to post
Share on other sites

Everything well with h264 and h265 on my side, with both hantro and rkvdec drivers on kernel 5.14 on rockchip RK322x chip (roughly same performance as H3) and latest LE patches applied.

 

One thing that always makes me have headaches are the userland kernel headers, which are not the same kernel headers used to compile modules and provided by armbian as a deb package.

 

The userland kernel headers on Armbian comes from the distribution (debian or ubuntu) and match the distribution kernel version, not the really installed kernel version.

Mismatch may cause compiling troubles (missing or duplicate structs or costants) or even subtle errors during runtime (unable to initialize things...)

 

To make the userland kernel headers, compile the kernel on the board (with LE patches applied) and then running

 

make headers_install INSTALL_HDR_PATH=/usr

 

 will install them in the proper place

 

Just after doing it is possible to go further compiling and installing ffmpeg and then compiling and installing mpv.

 

This is what I use to compile and install ffmpeg:

 

./configure --arch=armhf --extra-libs='-Lpthread -Lm' --disable-debug --enable-stripping --enable-gpl --enable-nonfree --enable-logging --disable-doc --enable-pic --enable-optimizations --enable-ffmpeg --enable-avdevice --enable-avcodec --enable-avformat --enable-swscale --enable-postproc --enable-avfilter --disable-devices --enable-pthreads --disable-w32threads --enable-network --disable-gnutls --disable-gray --enable-swscale-alpha --disable-small --enable-dct --enable-fft --enable-mdct --enable-rdft --disable-crystalhd --disable-vaapi --disable-vdpau --enable-libdrm --enable-version3 --enable-runtime-cpudetect --enable-hardcoded-tables --enable-encoder=ac3 --enable-encoder=aac --enable-encoder=wmav2 --enable-encoder=mjpeg --enable-encoder=png --enable-hwaccels --enable-muxer=spdif --enable-muxer=adts --enable-muxer=asf --enable-muxer=ipod --enable-muxer=mpegts --enable-demuxers --enable-parsers --enable-bsfs --enable-filters --enable-bzlib --enable-libspeex --enable-libx264 --enable-libx265 --enable-zlib --enable-asm --enable-neon --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libvpx --enable-openssl --enable-libv4l2 --enable-v4l2-request --enable-libudev

make -j4

sudo make install

 

This is what I use to compile and install mpv:

 

./waf configure --enable-egl-drm

 

And finally the mpv command line is (run on pure virtual terminal, no X11 at all):

 

mpv -v --vo=gpu --hwdec=drm --gpu-context=drm --gpu-hwdec-interop=drmprime-drm --drm-draw-plane=overlay --drm-drmprime-video-plane=primary bbb_sunflower_1080p_30fps_normal.mp4 

 

the --drm-draw-plane and --drm-drmprime-video-plane arguments are very imporant for my setup, probably because rk322x has no YUV capabilities for overlay plane and MPV refuses to show anything. Also they make the video and draw planes order correct (video below, GUI on top).

 

 

Link to post
Share on other sites

Thank you @jock for your detailed explanation. Since I use a different distribution, your instructions don't apply to my environment, but I know how to do it for myself. I just need to know what to do and @jernej has already pointed this out (needed patches and build requirements).

22 hours ago, jock said:

One thing that always makes me have headaches are the userland kernel headers, which are not the same kernel headers used to compile modules

I haven't considered this aspect so far, as it hasn't been mentioned that LE patched headers are required. I just applied all the LE kernel patches and just omitted obviously not necessary ones. Because ffmpeg build flawless with the native headers, I moved on. I have installed round about twenty kernels at the same time and it will be decided at boot which one to use. Therefore, I only move in new headers when necessary.

gst-play-1.0, ffplay and mpv now play bbb_sunflower_1080p_30fps_normal.mp4 with similar achievement. For gst-play-1.0 I know how to visualize the composed video pipeline. And due to the v4l2slh264dec (V4L2 Stateless H.264 Video Decoder) component in gst-play-pipiline.pdf I know for sure that hardware acceleration is used. So I guess the ffmpeg video pipeline is also using hardware acceleration.

gst-play-1.0 plays bbb_sunflower_2160p_30fps_normal.mp4 with same CPU utilization.

For ffplay and mpv CPU utilization ramps up significantly and frame drops are reported. The video also plays jerky. I can't say which components of the ffmpeg video pipeline is the limiting factor here.

Because the kernel patches also benefit the gstreamer framework, I will probably continue to use them. Let's see what explodes when 5.15.0-rc1 is available next week.

Link to post
Share on other sites

5 hours ago, usual user said:

I haven't considered this aspect so far, as it hasn't been mentioned that LE patched headers are required.

ffmpeg patches use only public, stable headers - at 5.14 that means MPEG2, VP8 and H264. For HEVC, ffmpeg patches contain their own copy of header file. That's why it's important to have matching kernel headers (for stable codecs) and LE patches (for unstable/non-public API and other improvements - if you care).

 

5 hours ago, usual user said:

For ffplay and mpv CPU utilization ramps up significantly and frame drops are reported.

AFAIK ffplay doesn't use HW acceleration. At least I never managed to force it. That's why I gave ffmpeg based test command. I can't help you with mpv...

5 hours ago, usual user said:

Let's see what explodes when 5.15.0-rc1 is available next week.

Only HEVC patches must be realigned, other codecs has been declared stable and thus API will not change. However, drivers implementing that API were improved in some areas.

Link to post
Share on other sites

Thank you @jernej for your detailed background information.

3 hours ago, jernej said:

AFAIK ffplay doesn't use HW acceleration. At least I never managed to force it. That's why I gave ffmpeg based test command. I can't help you with mpv...

In the past I didn't care much about ffmpeg driven video pipelines. I made several attempts, but because of the sparse documentation and not even the basic support in mainline, I gave up pretty quickly. I do this out of curiosity and for early use of future maninline support.
So as long as I can't find information on how to use it in daily use cases, I'll ignore it again. At least I have now some kernel patches that benefit gstreamer framework that I can track till they land in mainline.

Link to post
Share on other sites

1 hour ago, usual user said:

At least I have now some kernel patches that benefit gstreamer framework that I can track till they land in mainline.

Current master branch of gst-plugins-bad (what will land in next stable release) should already work pretty well with all stable codecs (MPEG2, H264 and VP8). HEVC patches won't help, because LE kernel patches add missing functionalities in API and those changes are naturally compatible only with ffmpeg patches from LE. But with every kernel release those patches get smaller.

Link to post
Share on other sites

11 hours ago, jernej said:

Current master branch of gst-plugins-bad (what will land in next stable release) should already work pretty well with all stable codecs (MPEG2, H264 and VP8).

I'm currently at gstreamer1-1.19.1, which works for my daily use cases flawless with the distribution package. But as soon as a deficiency should show up, it would be no big deal for me to rebuild with current master or pull in in-flight patches. But till then I will let the improvements trickle in via the regular updates.

Out of curiosity, I just started two instances of gst-play-1.0 with different videos at the same time, and both videos play flawlessly at the same time. OK, the mixed sound is a bit confusing.

Link to post
Share on other sites

On 9/11/2021 at 11:25 AM, usual user said:

Let's see what explodes when 5.15.0-rc1 is available next week.

Moved on to 5.15.0-rc1. But applying the LE patches makes the kernel unbootable for rockchip. I had now omitted 2001 also, as it no longer applies cleanly, and of course the cherry-picking 0010, which have landed.
Guess I have to take a closer look at which patches are really important to me and see if they are the cause.

With only linux-0011-v4l2-from-list, linux-1001-v4l2-rockchip and 2000-v4l-wip-rkvdec-vp9 applied the kernel keeps working.

Although the drm patches seem interesting, I'll leave them out for now.

Edited by usual user
additional information
Link to post
Share on other sites

7 hours ago, usual user said:

Moved on to 5.15.0-rc1. But applying the LE patches makes the kernel unbootable for rockchip. I had now omitted 2001 also, as it no longer applies cleanly, and of course the cherry-picking 0010, which have landed.
Guess I have to take a closer look at which patches are really important to me and see if they are the cause.

With only linux-0011-v4l2-from-list, linux-1001-v4l2-rockchip and 2000-v4l-wip-rkvdec-vp9 applied the kernel keeps working.

Although the drm patches seem interesting, I'll leave them out for now.

I'm in the middle of exploring how LE patches behave with rockchip 32 bit (rk3288 actually) and kernel 5.14.

I already did the process some weeks ago for kernel 5.10 and on my rk3288 tv box it worked so far (hantro and rkvdec both were working flawlessy).

 

If you're interested I can point the repository to you when I'm done.

 

Link to post
Share on other sites

1 hour ago, jock said:

If you're interested I can point the repository to you when I'm done.

Thank you for the offer @jock.
But 5.14.0 wise I had all VPU relevant LE patches and even the drm ones applied.
With gstreamer framework it is working, even for bbb_sunflower_2160p_30fps_normal.mp4 without any problem. It is mpv that seems not be able to cope with the hight data rate. Useing --hwdec=drm-copy makes the CPU utilization ramp up go away and the video plays smoothly, but to slow and the sound get out of sync. I don't think I am missing anything kernel wise.
Just tried to drop 2000-v4l-wip-rkvdec-vp9 in favour of this more recent patchset.

Resolving a merge conflict to apply the LE patches on top was easy:

Spoiler
--- linux-0011-v4l2-from-list.patch.orig<------>2021-09-08 20:04:26.000000000 +0200
+++ linux-0011-v4l2-from-list.patch<--->2021-09-15 19:00:00.000000000 +0200
@@ -756,9 +756,9 @@ index d31344c4acaa..d068383aeea8 100644
  <----><------><------>return -EINVAL;
..
  <----><------>if (ctx->valid_fmt && ctx->valid_fmt != rkvdec_valid_fmt(ctx, ctrl))
-@@ -155,6 +151,9 @@ static const struct rkvdec_ctrls rkvdec_h264_ctrls = {
+@@ -101,6 +151,9 @@ static const struct rkvdec_ctrls rkvdec_    
..
- static const u32 rkvdec_h264_decoded_fmts[] = {
+ static const u32 rkvdec_h264_vp9_decoded_fmts[] = {
  <---->V4L2_PIX_FMT_NV12,
 +<---->V4L2_PIX_FMT_NV15,
 +<---->V4L2_PIX_FMT_NV16,

 

But the kernel build drops out with this:

Spoiler
drivers/staging/media/hantro/imx8m_vpu_hw.c:145:27: error: ‘V4L2_PIX_FMT_NV12_4L4’ undeclared here (not in a function); did you mean ‘V4L2_PIX_FMT_NV12M’?
  145 |                 .fourcc = V4L2_PIX_FMT_NV12_4L4,
      |                           ^~~~~~~~~~~~~~~~~~~~~
      |                           V4L2_PIX_FMT_NV12M

 

So I will stay at the status quo of LE patches on 5.15.0-rc1 for now and wait till they rebase.
VPU wise everything from LE is in place and seems to work.

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
 Share

17 17