Mainline VPU


Recommended Posts

I've been following @xmixahlx work on pbp-tools.  I have an rk3399 board we're developing in house (kind of) and I'm tasked with seeing what it takes to get the VPU working.  Our build system won't work with the pbp scripts but I extraced the useful bits (urls, patches, etc.) from them to get a build working with the following components.  I also wrote a BASH test script for doing validation (see link below) up to and including attempting to decode and encode to the NULL device with ffmpeg.  I'm not getting the output I'm expecting and was wondering if I've missed a step.

 

This is what I have now.  This is the order I build them as well.

        1. Linux 5.8.5 w/local config plus pbp patch
            a. url: https://github.com/xmixahlx/pbp-tools/tree/master/resources/linux/5.8
            b. linux-5.8-rc4-hwaccel_20200709.diff
        2. v4l2-utils 1.20.0 (no patches)
        3. libva (git master) with patches
            a. url: https://patch-diff.githubusercontent.com/raw/intel/libva/pull/
            b. 332.patch
            c. 340.patch
            d. patches edited based on https://github.com/xmixahlx/pbp-tools/blob/master/pbp-install-libva
        4. libva-v4l2-request (git master) with patches
            a. url: https://patch-diff.githubusercontent.com/raw/bootlin/libva-v4l2-request/pull/
            b. 30.patch
            c. 32.patch
            d. Why not 28, 29 too? (31 doesn't exist: https://github.com/bootlin/libva-v4l2-request/pulls) - I think I found these may not be needed after all, but can't remember now why I left them out.
        5. libva-utils (git master, no patches)
        6. ffmpeg n4.3.1 with patches
            a. url: https://raw.githubusercontent.com/xmixahlx/pbp-tools/master/resources/ffmpeg/4.3/
            b. ffmpeg-4.3-v4l2request-rkvdec_20200709.diff
        7. v4l2-request-test (git master, no patches)
 

The test script, which I linked at the bottom of this post, is mostly just printing out what my setup looks like and responses to various v4l2, va and ffmpeg commands suggested here, the pbp forum and for ffmpeg.  I run it like this:

 

Quote

./vputest.sh -st "1 2 3 6 7" -l test.log

 

I can also run it without the -t option to enable decode/encode tests but those don't work and I think that's because of what the rest of the tests are telling me.  Most important is that vainfo is not showing any profile or entrypoints. 

One important fact for my build:  drm is not enabled yet.  We have it working on a 4.4.179 kernel and are working on a port to 5.4, but I have to wait for that before I port again to 5.8.5.  Not sure if that's going to be a show stopper for testing the VPU or not.  I thought with the va interface I wouldn't need that.

 

Any tips on what to try next would be appreciated.

 

Here is the output from the test.

VPU TESTS

==== CHECK FOR DRIVERS
Drivers:
        hantro-vpu
        rkvdec
==== LIST DEVICES
Video Devices: /dev/video1 /dev/video2 /dev/video0
Media Devices: /dev/media1 /dev/media0
==== DUMP FORMATS AND CONTROLS
++++ /dev/video1:
|----------------------------------------------------

JPEG Compression Controls

            compression_quality 0x009d0903 (int)    : min=5 max=100 step=1 default=50 value=50
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture Multiplanar

        [0]: 'JPEG' (JFIF JPEG, compressed)
|----------------------------------------------------
++++ /dev/video2:
|----------------------------------------------------
        mpeg_2_slice_parameters 0x009909fa (unknown): type=103 flags=has-payload
   mpeg_2_quantization_matrices 0x009909fb (unknown): type=104 flags=has-payload
               vp8_frame_header 0x009910d0 (unknown): type=301 flags=has-payload
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture Multiplanar

        [0]: 'NV12' (Y/CbCr 4:2:0)
|----------------------------------------------------
++++ /dev/video0:
|----------------------------------------------------

Codec Controls

                     h264_level 0x00990a67 (menu)   : min=0 max=15 default=0 value=0
                   h264_profile 0x00990a6b (menu)   : min=0 max=6 default=2 value=2
                    vp9_profile 0x00990b00 (menu)   : min=0 max=0 default=0 value=0
                   hevc_profile 0x00990b67 (menu)   : min=0 max=2 default=0 value=0
                     hevc_level 0x00990b68 (menu)   : min=0 max=8 default=0 value=0
    h264_sequence_parameter_set 0x00990ce8 (unknown): type=110 flags=has-payload
     h264_picture_parameter_set 0x00990ce9 (unknown): type=111 flags=has-payload
            h264_scaling_matrix 0x00990cea (unknown): type=112 flags=has-payload
          h264_slice_parameters 0x00990ceb (unknown): type=113 flags=has-payload
         h264_decode_parameters 0x00990cec (unknown): type=114 flags=has-payload
               h264_decode_mode 0x00990ced (menu)   : min=1 max=1 default=1 value=1
                h264_start_code 0x00990cee (menu)   : min=1 max=1 default=1 value=1
    hevc_sequence_parameter_set 0x00990cf0 (unknown): type=120 flags=has-payload
     hevc_picture_parameter_set 0x00990cf1 (unknown): type=121 flags=has-payload
          hevc_slice_parameters 0x00990cf2 (unknown): type=122 [16] flags=has-payload
            hevc_scaling_matrix 0x00990cf3 (unknown): type=123 flags=has-payload
               hevc_decode_mode 0x00990cf7 (menu)   : min=1 max=1 default=1 value=1
                hevc_start_code 0x00990cf8 (menu)   : min=1 max=1 default=1 value=1
            vp9_frame_context_0 0x009918a0 (unknown): type=400 flags=has-payload
            vp9_frame_context_1 0x009918a1 (unknown): type=400 flags=has-payload
            vp9_frame_context_2 0x009918a2 (unknown): type=400 flags=has-payload
            vp9_frame_context_3 0x009918a3 (unknown): type=400 flags=has-payload
    vp9_frame_decode_parameters 0x009918a4 (unknown): type=404 flags=has-payload
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture Multiplanar

        [0]: 'NV12' (Y/CbCr 4:2:0)
        [1]: 'NV15' (10-bit Y/CbCr 4:2:0 (Packed))
        [2]: 'NV16' (Y/CbCr 4:2:2)
        [3]: 'NV20' (10-bit Y/CbCr 4:2:2 (Packed))
|----------------------------------------------------
==== SHOW HW ACCELS
ffmpeg version n4.3.1-Kodi Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 7.5.0 (GCC)
  configuration: --prefix=/usr --arch=arm64 --target-os=linux --cross-prefix=aarch64-linux-gnu- --enable-cross-compile --sysr
oot=/home2/qsys3/src/os/../aarch64-linux-gnu/install --pkg-config=aarch64-linux-gnu-pkg-config --enable-shared --disable-doc 
--enable-v4l2-request --enable-libdrm --enable-libudev --enable-pic --enable-avfilter --enable-postproc --enable-pthreads --e
nable-gpl --enable-nonfree --enable-version3 --toolchain=hardened --enable-avfilter --enable-libfontconfig --enable-libfreety
pe --enable-libxml2 --enable-openssl --enable-protocol=file --enable-protocol=pipe --enable-hwaccel=h264_vaapi --enable-hwacc
el=hevc_vaapi --enable-hwaccel=mjpeg_vaapi --enable-hwaccel=mpeg2_vaapi --enable-hwaccel=mpeg4_vaapi --enable-hwaccel=vp9_vaa
pi --enable-muxer=h264 --enable-muxer=mjpeg --enable-muxer=mpeg2video --enable-muxer=mp2 --enable-muxer=mp4 --enable-muxer=mo
v --enable-muxer=null --enable-demuxer=mpegvideo --enable-demuxer=h264 --enable-demuxer=hevc --enable-demuxer=mov --enable-en
coder=mjpeg_vaapi --enable-encoder=vp9_vaapi --enable-encoder=vp8_v4l2m2m --enable-encoder=vp8_vp8_vaapi --enable-encoder=hev
c_vaapi --enable-encoder=hevc_v4l2m2m --enable-encoder=mpeg2_vaapi --enable-encoder=h264_vaapi --enable-encoder=h264_v4l2m2m 
--enable-encoder=wrapped_avframe --enable-encoder=pcm_s16le --enable-decoder=mjpeg --enable-decoder=vp9 --enable-decoder=vp8 
--enable-decoder=vp8_v4l2m2m --enable-decoder=vp9_v4l2m2m --enable-decoder=hevc --enable-decoder=hevc_v4l2m2m --enable-decode
r=h264 --enable-decoder=webp --enable-decoder=mpeg2video --enable-decoder=mpeg2_v4l2m2m --enable-decoder=mpeg4 --enable-decod
er=mpeg4_v4l2m2m --enable-rpath --extra-ldflags=-ludev --libdir=/usr/lib --shlibdir=/usr/lib
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
Hardware acceleration methods:
vaapi
drm
|----------------------------------------------------
==== SHOW VAINFO
++++ /dev/video1:
|----------------------------------------------------
libva info: VA-API version 1.9.0
libva info: User environment variable requested driver 'v4l2_request'
libva info: Trying to open /usr/lib/dri/v4l2_request_drv_video.so
libva info: Found init function __vaDriverInit_1_9
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.9 (libva 2.9.0.pre1)
vainfo: Driver version: v4l2-request
vainfo: Supported profile and entrypoints
|----------------------------------------------------
++++ /dev/video2:
|----------------------------------------------------
libva info: VA-API version 1.9.0
libva info: User environment variable requested driver 'v4l2_request'
libva info: Trying to open /usr/lib/dri/v4l2_request_drv_video.so
libva info: Found init function __vaDriverInit_1_9
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.9 (libva 2.9.0.pre1)
vainfo: Driver version: v4l2-request
vainfo: Supported profile and entrypoints
|----------------------------------------------------
++++ /dev/video0:
|----------------------------------------------------
libva info: VA-API version 1.9.0
libva info: User environment variable requested driver 'v4l2_request'
libva info: Trying to open /usr/lib/dri/v4l2_request_drv_video.so
libva info: Found init function __vaDriverInit_1_9
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.9 (libva 2.9.0.pre1)
vainfo: Driver version: v4l2-request
vainfo: Supported profile and entrypoints
|----------------------------------------------------
Tests completed.

 

Here is the test script.

 

 

Link to post
Share on other sites
Armbian is a community driven open source project. Do you like to contribute your code?

We are developing AI vision app on Orange Pi 4 and Intel Movidius Neural Stick 2. Everything is working great up to FullHD with software decoders. But now we need to decode 4K video stream from RTSP to feed it to OpenCV/OpenVINO.

We could successfully use Kwiboo's ffmpeg with current Armbian 5.8.6 kernel without patches. Decoder is definitely working and we can see 120 fps with `ffmpeg -hwaccel drm -hwaccel_output_format drm_prime -i big1.avi -benchmark -f null -` (big1.avi is 3840 x 2160 h264 video).

However I can't find a way to scale it using RGA. As far as I understand it should be some kind of mem2mem filter but I'm not sure how to do it. The best I could come so far is the command: `ffmpeg -y -hwaccel drm -hwaccel_output_format drm_prime -i big1.avi -an -vf 'hwdownload,format=nv12,scale=1280:-2' -pix_fmt rgb24 -f rawvideo /dev/null` that gives about 15fps and 20% CPU usage.

So my question is how can I scale and convert to RGB888 without 4K decoded frame downloaded to the CPU?

Link to post
Share on other sites

Does anybody have a set of compiled FFMPEG libraries that can be used with the latest Armbian Focal release? I have difficulty compiling the branches in Kwiboo's repository and getting them to work afterwards. A set of patches that works against the release from Focal so that I could recompile the deb with those would work too.

I'm using an ASUS Tinkerboard.

Link to post
Share on other sites
On 7/23/2020 at 8:43 PM, rubenvb said:

Current Kodi master (currently hash 9f6d978f646) with the v4l2-request-hwaccel-4.3-rkvdec ffmpeg branch leads to audio only when starting video playback.

 

I finally figured out what was going wrong with my new Kodi git builds: it seems an option "enable DRM PRIME acceleration" appeared in the settings.

To get video output at all, I need to disable this.

Strange thing is, rebuilding the same working commit on an up to date system (I run ArchLinuxARM) produced the same issue, so I am still in the dark what exactly is happening here.

Installing my old build (and restoring the old versions of various dependencies, as Arch updates everything quite frequently), always worked. But building it again, in the same exact way, gave no video output as described in the post above.

 

Is this due to something not working in Kodi, or with the way Kwiboo's FFMPEG works (i.e. "automagically" using the v4l2-request hwaccel path when outputting to DRM)?

 

Having tried the 5.9-rc1 kernel, it seems something broke the NanoPi M4 SATA hat in that release. So still on the 5.8-rc1. I should really look into a nice way to extract the series of patches Kwiboo maintains (or is most of it already merged into mainline?).

 

But in any case, all is well (except 5.9-rc1 SATA hat, but that's not relevant here), it seemed to be user (my) error of having the DRM PRIME acceleration enabled in the Kodi settings.

 

I can't keep thanking you all enough, especially @Kwiboo!

Link to post
Share on other sites
2 hours ago, rubenvb said:

it seemed to be user (my) error of having the DRM PRIME acceleration enabled in the Kodi settings.

Well, DRM PRIME acceleration is the most efficient one for ARM boards and disabling it means that CPU will use much more resources than it's needed. In fact, v4l2-request hwaccel patches for ffmpeg were developed together with DRM PRIME acceleration in Kodi.

 

Note that DRM PRIME acceleration consists of two parts:

1. DRM PRIME HW decoding

2. DRM PRIME zero-copy rendering

 

First one is absolutely necessary to have it enabled, otherwise VPU won't be used and instead video will be SW decoded. Second one is still very important - it bypasses GPU for rendering and instead it uses DRM planes which is much more efficient. This is important for SoCs with weak GPUs like mali400. However, it will only work with GBM version of Kodi and maybe wayland. Although I never tested it with X11, I'm pretty sure it won't work. So for X11 you have to set DRM PRIME rendering method to GLES (I guess). At this point I'm not sure if X11 and GLES even work together and most ARM GPUs don't support GL, only GLES (I really don't use X11 on ARM boards, so I don't know).

 

What is your SoC? Do you use X11 desktop? What is your CPU utilization during video playback (press "o" during playback)? Normal utilization is about 10% of one core.

Link to post
Share on other sites
Just now, jernej said:

Well, DRM PRIME acceleration is the most efficient one for ARM boards and disabling it means that CPU will use much more resources than it's needed. In fact, v4l2-request hwaccel patches for ffmpeg were developed together with DRM PRIME acceleration in Kodi.

 

Note that DRM PRIME acceleration consists of two parts:

1. DRM PRIME HW decoding

2. DRM PRIME zero-copy rendering

 

First one is absolutely necessary to have it enabled, otherwise VPU won't be used and instead video will be SW decoded. Second one is still very important - it bypasses GPU for rendering and instead it uses DRM planes which is much more efficient. This is important for SoCs with weak GPUs like mali400. However, it will only work with GBM version of Kodi and maybe wayland. Although I never tested it with X11, I'm pretty sure it won't work. So for X11 you have to set DRM PRIME rendering method to GLES (I guess). At this point I'm not sure if X11 and GLES even work together and most ARM GPUs don't support GL, only GLES (I really don't use X11 on ARM boards, so I don't know).

 

What is your SoC? Do you use X11 desktop? What is your CPU utilization during video playback (press "o" during playback)? Normal utilization is about 10% of one core.

You're absolutely correct, I saw what I wanted to see :(. It's just using the CPU decoder.

 

The SoC is NanoPi M4 (RK3399).

I use Kodi with GBM started directly through a systemd service; no desktop, no window manager.

Mesa 20.2.1 Panfrost gets me OpenGL ES 3.0, which is used for the Kodi GUI.

Kwiboo's 5.8-rc1 kernel (built from source on the 27th of June) gives me the driver-side v4l2-request API support through the rkvdec device.

The last functional build of Kodi I have is from commit 9f6d978f646, and was built the 25th of May. Anything later (even if I try to build that same exact commit again, though with the current Kwiboo v4l2-request-hwaccel-4.2.2-rkvdec branch, Kodi used 4.2.2 at the time ), I get the broken behaviour :(. My guess is Kwiboo changed something when rebasing the branch after the 25th of May.

 

For the current kodi-git build, I tried both the EGL and Direct-to-Plane options for the DRM PRIME acceleration.

EGL results in no video output (stays on the Kodi menu) and Direct-to-Plane results in flashing of the busy spinner (and no video output) and in both cases normal audio output.

Both result in the UI drawing over itself (i.e. it's not being cleared before it's drawn over) until I stop playback.

 

Is there any way I can provide you with better info about this?

Here is a log of a H265 file not playing back with FFMPEG and video logging enabled:

https://gist.github.com/rubenvb/9da672e92577fd161e8dc51073ee0eda

I added a second log to that gist of the old functional build I have, so the output can be compared.

Note the nonfunctional output is from Kodi commit 9d256dd15d9 built yesterday, and the functional output is commit 9f6d978f646, built on the 25th of May.

 

If the kernel needs to be updated for this to work again, I'll need to investigate the SATA hat not being detected in 5.9-rc1.

 

Thanks for taking interest!

Link to post
Share on other sites
1 hour ago, jernej said:

AFAIK HEVC is not yet supported in RK driver but some WIP patches exists. Try H264 first.

RIght, but these files work well enough with my old Kodi build.

 

The same thing (no video, busy spinner flashing) happens when I try to play H264 files. I'll see if I can conjure up logs for that, although they won't be much different.

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...