Mainline VPU


Recommended Posts

I've been following @xmixahlx work on pbp-tools.  I have an rk3399 board we're developing in house (kind of) and I'm tasked with seeing what it takes to get the VPU working.  Our build system won't work with the pbp scripts but I extraced the useful bits (urls, patches, etc.) from them to get a build working with the following components.  I also wrote a BASH test script for doing validation (see link below) up to and including attempting to decode and encode to the NULL device with ffmpeg.  I'm not getting the output I'm expecting and was wondering if I've missed a step.

 

This is what I have now.  This is the order I build them as well.

        1. Linux 5.8.5 w/local config plus pbp patch
            a. url: https://github.com/xmixahlx/pbp-tools/tree/master/resources/linux/5.8
            b. linux-5.8-rc4-hwaccel_20200709.diff
        2. v4l2-utils 1.20.0 (no patches)
        3. libva (git master) with patches
            a. url: https://patch-diff.githubusercontent.com/raw/intel/libva/pull/
            b. 332.patch
            c. 340.patch
            d. patches edited based on https://github.com/xmixahlx/pbp-tools/blob/master/pbp-install-libva
        4. libva-v4l2-request (git master) with patches
            a. url: https://patch-diff.githubusercontent.com/raw/bootlin/libva-v4l2-request/pull/
            b. 30.patch
            c. 32.patch
            d. Why not 28, 29 too? (31 doesn't exist: https://github.com/bootlin/libva-v4l2-request/pulls) - I think I found these may not be needed after all, but can't remember now why I left them out.
        5. libva-utils (git master, no patches)
        6. ffmpeg n4.3.1 with patches
            a. url: https://raw.githubusercontent.com/xmixahlx/pbp-tools/master/resources/ffmpeg/4.3/
            b. ffmpeg-4.3-v4l2request-rkvdec_20200709.diff
        7. v4l2-request-test (git master, no patches)
 

The test script, which I linked at the bottom of this post, is mostly just printing out what my setup looks like and responses to various v4l2, va and ffmpeg commands suggested here, the pbp forum and for ffmpeg.  I run it like this:

 

Quote

./vputest.sh -st "1 2 3 6 7" -l test.log

 

I can also run it without the -t option to enable decode/encode tests but those don't work and I think that's because of what the rest of the tests are telling me.  Most important is that vainfo is not showing any profile or entrypoints. 

One important fact for my build:  drm is not enabled yet.  We have it working on a 4.4.179 kernel and are working on a port to 5.4, but I have to wait for that before I port again to 5.8.5.  Not sure if that's going to be a show stopper for testing the VPU or not.  I thought with the va interface I wouldn't need that.

 

Any tips on what to try next would be appreciated.

 

Here is the output from the test.

VPU TESTS

==== CHECK FOR DRIVERS
Drivers:
        hantro-vpu
        rkvdec
==== LIST DEVICES
Video Devices: /dev/video1 /dev/video2 /dev/video0
Media Devices: /dev/media1 /dev/media0
==== DUMP FORMATS AND CONTROLS
++++ /dev/video1:
|----------------------------------------------------

JPEG Compression Controls

            compression_quality 0x009d0903 (int)    : min=5 max=100 step=1 default=50 value=50
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture Multiplanar

        [0]: 'JPEG' (JFIF JPEG, compressed)
|----------------------------------------------------
++++ /dev/video2:
|----------------------------------------------------
        mpeg_2_slice_parameters 0x009909fa (unknown): type=103 flags=has-payload
   mpeg_2_quantization_matrices 0x009909fb (unknown): type=104 flags=has-payload
               vp8_frame_header 0x009910d0 (unknown): type=301 flags=has-payload
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture Multiplanar

        [0]: 'NV12' (Y/CbCr 4:2:0)
|----------------------------------------------------
++++ /dev/video0:
|----------------------------------------------------

Codec Controls

                     h264_level 0x00990a67 (menu)   : min=0 max=15 default=0 value=0
                   h264_profile 0x00990a6b (menu)   : min=0 max=6 default=2 value=2
                    vp9_profile 0x00990b00 (menu)   : min=0 max=0 default=0 value=0
                   hevc_profile 0x00990b67 (menu)   : min=0 max=2 default=0 value=0
                     hevc_level 0x00990b68 (menu)   : min=0 max=8 default=0 value=0
    h264_sequence_parameter_set 0x00990ce8 (unknown): type=110 flags=has-payload
     h264_picture_parameter_set 0x00990ce9 (unknown): type=111 flags=has-payload
            h264_scaling_matrix 0x00990cea (unknown): type=112 flags=has-payload
          h264_slice_parameters 0x00990ceb (unknown): type=113 flags=has-payload
         h264_decode_parameters 0x00990cec (unknown): type=114 flags=has-payload
               h264_decode_mode 0x00990ced (menu)   : min=1 max=1 default=1 value=1
                h264_start_code 0x00990cee (menu)   : min=1 max=1 default=1 value=1
    hevc_sequence_parameter_set 0x00990cf0 (unknown): type=120 flags=has-payload
     hevc_picture_parameter_set 0x00990cf1 (unknown): type=121 flags=has-payload
          hevc_slice_parameters 0x00990cf2 (unknown): type=122 [16] flags=has-payload
            hevc_scaling_matrix 0x00990cf3 (unknown): type=123 flags=has-payload
               hevc_decode_mode 0x00990cf7 (menu)   : min=1 max=1 default=1 value=1
                hevc_start_code 0x00990cf8 (menu)   : min=1 max=1 default=1 value=1
            vp9_frame_context_0 0x009918a0 (unknown): type=400 flags=has-payload
            vp9_frame_context_1 0x009918a1 (unknown): type=400 flags=has-payload
            vp9_frame_context_2 0x009918a2 (unknown): type=400 flags=has-payload
            vp9_frame_context_3 0x009918a3 (unknown): type=400 flags=has-payload
    vp9_frame_decode_parameters 0x009918a4 (unknown): type=404 flags=has-payload
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture Multiplanar

        [0]: 'NV12' (Y/CbCr 4:2:0)
        [1]: 'NV15' (10-bit Y/CbCr 4:2:0 (Packed))
        [2]: 'NV16' (Y/CbCr 4:2:2)
        [3]: 'NV20' (10-bit Y/CbCr 4:2:2 (Packed))
|----------------------------------------------------
==== SHOW HW ACCELS
ffmpeg version n4.3.1-Kodi Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 7.5.0 (GCC)
  configuration: --prefix=/usr --arch=arm64 --target-os=linux --cross-prefix=aarch64-linux-gnu- --enable-cross-compile --sysr
oot=/home2/qsys3/src/os/../aarch64-linux-gnu/install --pkg-config=aarch64-linux-gnu-pkg-config --enable-shared --disable-doc 
--enable-v4l2-request --enable-libdrm --enable-libudev --enable-pic --enable-avfilter --enable-postproc --enable-pthreads --e
nable-gpl --enable-nonfree --enable-version3 --toolchain=hardened --enable-avfilter --enable-libfontconfig --enable-libfreety
pe --enable-libxml2 --enable-openssl --enable-protocol=file --enable-protocol=pipe --enable-hwaccel=h264_vaapi --enable-hwacc
el=hevc_vaapi --enable-hwaccel=mjpeg_vaapi --enable-hwaccel=mpeg2_vaapi --enable-hwaccel=mpeg4_vaapi --enable-hwaccel=vp9_vaa
pi --enable-muxer=h264 --enable-muxer=mjpeg --enable-muxer=mpeg2video --enable-muxer=mp2 --enable-muxer=mp4 --enable-muxer=mo
v --enable-muxer=null --enable-demuxer=mpegvideo --enable-demuxer=h264 --enable-demuxer=hevc --enable-demuxer=mov --enable-en
coder=mjpeg_vaapi --enable-encoder=vp9_vaapi --enable-encoder=vp8_v4l2m2m --enable-encoder=vp8_vp8_vaapi --enable-encoder=hev
c_vaapi --enable-encoder=hevc_v4l2m2m --enable-encoder=mpeg2_vaapi --enable-encoder=h264_vaapi --enable-encoder=h264_v4l2m2m 
--enable-encoder=wrapped_avframe --enable-encoder=pcm_s16le --enable-decoder=mjpeg --enable-decoder=vp9 --enable-decoder=vp8 
--enable-decoder=vp8_v4l2m2m --enable-decoder=vp9_v4l2m2m --enable-decoder=hevc --enable-decoder=hevc_v4l2m2m --enable-decode
r=h264 --enable-decoder=webp --enable-decoder=mpeg2video --enable-decoder=mpeg2_v4l2m2m --enable-decoder=mpeg4 --enable-decod
er=mpeg4_v4l2m2m --enable-rpath --extra-ldflags=-ludev --libdir=/usr/lib --shlibdir=/usr/lib
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
Hardware acceleration methods:
vaapi
drm
|----------------------------------------------------
==== SHOW VAINFO
++++ /dev/video1:
|----------------------------------------------------
libva info: VA-API version 1.9.0
libva info: User environment variable requested driver 'v4l2_request'
libva info: Trying to open /usr/lib/dri/v4l2_request_drv_video.so
libva info: Found init function __vaDriverInit_1_9
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.9 (libva 2.9.0.pre1)
vainfo: Driver version: v4l2-request
vainfo: Supported profile and entrypoints
|----------------------------------------------------
++++ /dev/video2:
|----------------------------------------------------
libva info: VA-API version 1.9.0
libva info: User environment variable requested driver 'v4l2_request'
libva info: Trying to open /usr/lib/dri/v4l2_request_drv_video.so
libva info: Found init function __vaDriverInit_1_9
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.9 (libva 2.9.0.pre1)
vainfo: Driver version: v4l2-request
vainfo: Supported profile and entrypoints
|----------------------------------------------------
++++ /dev/video0:
|----------------------------------------------------
libva info: VA-API version 1.9.0
libva info: User environment variable requested driver 'v4l2_request'
libva info: Trying to open /usr/lib/dri/v4l2_request_drv_video.so
libva info: Found init function __vaDriverInit_1_9
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.9 (libva 2.9.0.pre1)
vainfo: Driver version: v4l2-request
vainfo: Supported profile and entrypoints
|----------------------------------------------------
Tests completed.

 

Here is the test script.

 

 

Link to post
Share on other sites
Donate and support the project!

We are developing AI vision app on Orange Pi 4 and Intel Movidius Neural Stick 2. Everything is working great up to FullHD with software decoders. But now we need to decode 4K video stream from RTSP to feed it to OpenCV/OpenVINO.

We could successfully use Kwiboo's ffmpeg with current Armbian 5.8.6 kernel without patches. Decoder is definitely working and we can see 120 fps with `ffmpeg -hwaccel drm -hwaccel_output_format drm_prime -i big1.avi -benchmark -f null -` (big1.avi is 3840 x 2160 h264 video).

However I can't find a way to scale it using RGA. As far as I understand it should be some kind of mem2mem filter but I'm not sure how to do it. The best I could come so far is the command: `ffmpeg -y -hwaccel drm -hwaccel_output_format drm_prime -i big1.avi -an -vf 'hwdownload,format=nv12,scale=1280:-2' -pix_fmt rgb24 -f rawvideo /dev/null` that gives about 15fps and 20% CPU usage.

So my question is how can I scale and convert to RGB888 without 4K decoded frame downloaded to the CPU?

Link to post
Share on other sites

Does anybody have a set of compiled FFMPEG libraries that can be used with the latest Armbian Focal release? I have difficulty compiling the branches in Kwiboo's repository and getting them to work afterwards. A set of patches that works against the release from Focal so that I could recompile the deb with those would work too.

I'm using an ASUS Tinkerboard.

Link to post
Share on other sites
On 7/23/2020 at 8:43 PM, rubenvb said:

Current Kodi master (currently hash 9f6d978f646) with the v4l2-request-hwaccel-4.3-rkvdec ffmpeg branch leads to audio only when starting video playback.

 

I finally figured out what was going wrong with my new Kodi git builds: it seems an option "enable DRM PRIME acceleration" appeared in the settings.

To get video output at all, I need to disable this.

Strange thing is, rebuilding the same working commit on an up to date system (I run ArchLinuxARM) produced the same issue, so I am still in the dark what exactly is happening here.

Installing my old build (and restoring the old versions of various dependencies, as Arch updates everything quite frequently), always worked. But building it again, in the same exact way, gave no video output as described in the post above.

 

Is this due to something not working in Kodi, or with the way Kwiboo's FFMPEG works (i.e. "automagically" using the v4l2-request hwaccel path when outputting to DRM)?

 

Having tried the 5.9-rc1 kernel, it seems something broke the NanoPi M4 SATA hat in that release. So still on the 5.8-rc1. I should really look into a nice way to extract the series of patches Kwiboo maintains (or is most of it already merged into mainline?).

 

But in any case, all is well (except 5.9-rc1 SATA hat, but that's not relevant here), it seemed to be user (my) error of having the DRM PRIME acceleration enabled in the Kodi settings.

 

I can't keep thanking you all enough, especially @Kwiboo!

Link to post
Share on other sites
2 hours ago, rubenvb said:

it seemed to be user (my) error of having the DRM PRIME acceleration enabled in the Kodi settings.

Well, DRM PRIME acceleration is the most efficient one for ARM boards and disabling it means that CPU will use much more resources than it's needed. In fact, v4l2-request hwaccel patches for ffmpeg were developed together with DRM PRIME acceleration in Kodi.

 

Note that DRM PRIME acceleration consists of two parts:

1. DRM PRIME HW decoding

2. DRM PRIME zero-copy rendering

 

First one is absolutely necessary to have it enabled, otherwise VPU won't be used and instead video will be SW decoded. Second one is still very important - it bypasses GPU for rendering and instead it uses DRM planes which is much more efficient. This is important for SoCs with weak GPUs like mali400. However, it will only work with GBM version of Kodi and maybe wayland. Although I never tested it with X11, I'm pretty sure it won't work. So for X11 you have to set DRM PRIME rendering method to GLES (I guess). At this point I'm not sure if X11 and GLES even work together and most ARM GPUs don't support GL, only GLES (I really don't use X11 on ARM boards, so I don't know).

 

What is your SoC? Do you use X11 desktop? What is your CPU utilization during video playback (press "o" during playback)? Normal utilization is about 10% of one core.

Link to post
Share on other sites
Just now, jernej said:

Well, DRM PRIME acceleration is the most efficient one for ARM boards and disabling it means that CPU will use much more resources than it's needed. In fact, v4l2-request hwaccel patches for ffmpeg were developed together with DRM PRIME acceleration in Kodi.

 

Note that DRM PRIME acceleration consists of two parts:

1. DRM PRIME HW decoding

2. DRM PRIME zero-copy rendering

 

First one is absolutely necessary to have it enabled, otherwise VPU won't be used and instead video will be SW decoded. Second one is still very important - it bypasses GPU for rendering and instead it uses DRM planes which is much more efficient. This is important for SoCs with weak GPUs like mali400. However, it will only work with GBM version of Kodi and maybe wayland. Although I never tested it with X11, I'm pretty sure it won't work. So for X11 you have to set DRM PRIME rendering method to GLES (I guess). At this point I'm not sure if X11 and GLES even work together and most ARM GPUs don't support GL, only GLES (I really don't use X11 on ARM boards, so I don't know).

 

What is your SoC? Do you use X11 desktop? What is your CPU utilization during video playback (press "o" during playback)? Normal utilization is about 10% of one core.

You're absolutely correct, I saw what I wanted to see :(. It's just using the CPU decoder.

 

The SoC is NanoPi M4 (RK3399).

I use Kodi with GBM started directly through a systemd service; no desktop, no window manager.

Mesa 20.2.1 Panfrost gets me OpenGL ES 3.0, which is used for the Kodi GUI.

Kwiboo's 5.8-rc1 kernel (built from source on the 27th of June) gives me the driver-side v4l2-request API support through the rkvdec device.

The last functional build of Kodi I have is from commit 9f6d978f646, and was built the 25th of May. Anything later (even if I try to build that same exact commit again, though with the current Kwiboo v4l2-request-hwaccel-4.2.2-rkvdec branch, Kodi used 4.2.2 at the time ), I get the broken behaviour :(. My guess is Kwiboo changed something when rebasing the branch after the 25th of May.

 

For the current kodi-git build, I tried both the EGL and Direct-to-Plane options for the DRM PRIME acceleration.

EGL results in no video output (stays on the Kodi menu) and Direct-to-Plane results in flashing of the busy spinner (and no video output) and in both cases normal audio output.

Both result in the UI drawing over itself (i.e. it's not being cleared before it's drawn over) until I stop playback.

 

Is there any way I can provide you with better info about this?

Here is a log of a H265 file not playing back with FFMPEG and video logging enabled:

https://gist.github.com/rubenvb/9da672e92577fd161e8dc51073ee0eda

I added a second log to that gist of the old functional build I have, so the output can be compared.

Note the nonfunctional output is from Kodi commit 9d256dd15d9 built yesterday, and the functional output is commit 9f6d978f646, built on the 25th of May.

 

If the kernel needs to be updated for this to work again, I'll need to investigate the SATA hat not being detected in 5.9-rc1.

 

Thanks for taking interest!

Link to post
Share on other sites
1 hour ago, jernej said:

AFAIK HEVC is not yet supported in RK driver but some WIP patches exists. Try H264 first.

RIght, but these files work well enough with my old Kodi build.

 

The same thing (no video, busy spinner flashing) happens when I try to play H264 files. I'll see if I can conjure up logs for that, although they won't be much different.

Link to post
Share on other sites
On 10/24/2020 at 2:29 PM, jernej said:

Well, DRM PRIME acceleration is the most efficient one for ARM boards and disabling it means that CPU will use much more resources than it's needed. In fact, v4l2-request hwaccel patches for ffmpeg were developed together with DRM PRIME acceleration in Kodi.

 

Note that DRM PRIME acceleration consists of two parts:

1. DRM PRIME HW decoding

2. DRM PRIME zero-copy rendering

 

First one is absolutely necessary to have it enabled, otherwise VPU won't be used and instead video will be SW decoded. Second one is still very important - it bypasses GPU for rendering and instead it uses DRM planes which is much more efficient. This is important for SoCs with weak GPUs like mali400. However, it will only work with GBM version of Kodi and maybe wayland. Although I never tested it with X11, I'm pretty sure it won't work. So for X11 you have to set DRM PRIME rendering method to GLES (I guess). At this point I'm not sure if X11 and GLES even work together and most ARM GPUs don't support GL, only GLES (I really don't use X11 on ARM boards, so I don't know).

 

What is your SoC? Do you use X11 desktop? What is your CPU utilization during video playback (press "o" during playback)? Normal utilization is about 10% of one core.

 

DRM rendering works with X11 and GLES/EGL. I've made a video player that renders up to 4K/30fps videos with 10-12% overall CPU usage.  That however worked on ayufan images on RockPro64, I'm having difficulties running it on Armbian.

Link to post
Share on other sites
34 minutes ago, Pencheff said:

DRM rendering works with X11 and GLES/EGL. I've made a video player that renders up to 4K/30fps videos with 10-12% overall CPU usage.  That however worked on ayufan images on RockPro64, I'm having difficulties running it on Armbian.

Are you sure it was on mainline and not legacy? On legacy it does work with the median script from jmcc. First post in RK3399.

Link to post
Share on other sites
22 minutes ago, NicoD said:

the median script from jmcc

The media script is dead, long life to the media integration!

It is one of the new features in Tamandua, but I have not documented it yet. For the time being, suffice to say:

  • Get a Buster-legacy image
  • apt-install --install-recommends media-buster-legacy-rk3399 (or -rk3328, or-tinkerboard for 3288)
Link to post
Share on other sites

I just posted the announcement for the official Armbian Rockchip Legacy Multimedia Framework. As you can see on the announcement, I say that from now on all multimedia work will be done in Mainline.

 

That means integrating GPU and VPU acceleration into the official Armbian repos. The timeline is for next Armbian release, in February, if possible.

 

As of today, I am planning to include Kodi and MPV. So if anybody wants to help, it will be much appreciated. :thumbup:

Link to post
Share on other sites

I'm working on getting VA-API going on Linux 5.10.y on the Openhour Chameleon, a RK3288 TV Box that is closest to the Mqmaker Miqi. This has hantro in the kernel and V4L2 is working fine. I am working with Groovy to have fairly modern libraries available. The main purpose is to enable H264 hardware decoding to make Moonlight-QT able to stream from an Nvidia Gamestreaming desktop effectively.

 

I had a look at the various forks of libva-v4l2-request and either:


1. They don't present any entrypoints, like what @mjhammel experienced:

libva info: VA-API version 1.8.0
libva info: User environment variable requested driver 'v4l2_request'
libva info: Trying to open /usr/lib/arm-linux-gnueabihf/dri/v4l2_request_drv_video.so
libva info: Found init function __vaDriverInit_1_8
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.8 (libva 2.8.0)
vainfo: Driver version: v4l2-request
vainfo: Supported profile and entrypoints

 

2. I get compilation errors. These are for hantro-specific builds like the hantro-h264 branch of ndufresne eg:

 

../src/h264.c: In function ‘h264_va_picture_to_v4l2’:
../src/h264.c:250:12: error: ‘VAPictureParameterBufferH264’ has no member named ‘num_ref_idx_l0_default_active_minus1’
  250 |   VAPicture->num_ref_idx_l0_default_active_minus1;

 

The root of these issues is work by ph5 but I think this may be due to them working with a different version of the libva library which has this member. Can anyone confirm the libva library used in these branches?

 

For vainfo, I expect to see entrypoints like:

 

vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            :	VAEntrypointVLD
      VAProfileMPEG2Main              :	VAEntrypointVLD
      VAProfileH264Main               :	VAEntrypointVLD
      VAProfileH264Main               :	VAEntrypointEncSliceLP
      VAProfileH264High               :	VAEntrypointVLD
      VAProfileH264High               :	VAEntrypointEncSliceLP

 

Has anyone seen these entrypoints with the v4l2-request library before?

 

Link to post
Share on other sites
19 hours ago, Alex van Vucht said:

I'm working on getting VA-API going on Linux 5.10.y on the Openhour Chameleon, a RK3288 TV Box that is closest to the Mqmaker Miqi. This has hantro in the kernel and V4L2 is working fine. I am working with Groovy to have fairly modern libraries available. The main purpose is to enable H264 hardware decoding to make Moonlight-QT able to stream from an Nvidia Gamestreaming desktop effectively.

 

I had a look at the various forks of libva-v4l2-request and ... I get compilation errors. These are for hantro-specific builds like the hantro-h264 branch of ndufresne eg:

 

../src/h264.c: In function ‘h264_va_picture_to_v4l2’:
../src/h264.c:250:12: error: ‘VAPictureParameterBufferH264’ has no member named ‘num_ref_idx_l0_default_active_minus1’
  250 |   VAPicture->num_ref_idx_l0_default_active_minus1;

 

The root of these issues is work by ph5 but I think this may be due to them working with a different version of the libva library which has this member. Can anyone confirm the libva library used in these branches?

 

Answering my own question: https://github.com/intel/libva/pull/332 is enlightening on the situation with the v4l2-request library. As stated in the pull request, gstreamer works with v4l2 now so the pressing need for va-api support is gone, and it looks like the v4l2-request library needs substantial revising to support the VA-API framework.

 

If an Armbian user wishes to push on with va-api on Rockchip and hantro, they will have to apply the patch on that pull request to a libva source, and compile the hantro-h264 ndufresne branch. Even then, entrypoints probably still won't emerge for use by applications: mjhammel followed these steps and applied this patch but still couldn't unlock hantro. The cedrus boards may have more luck, there seems to have been more work on those and v4l2-request in recent months.

 

Common uses for libva include video stream hardware decoding; reports are that the h264ify extension for Vivaldi may work but I'm having stability issues with that browser and the panfrost DRM driver. That's probably just an rk3288 thing though. At least I got Widevine working but the arm64 people are out of luck at present, which rules out paid streaming services.

 

For my own use case, I am pushing on with moonlight-embedded which has a Gstreamer branch. Good luck with VPU acceleration in the 5.x series!

Edited by Alex van Vucht
Link to post
Share on other sites

libva-v4l2-request is not driver specific, it has same issues with Cedrus too. It was just proof of concept/tech-demo. Bootlin doesn't have time to maintain it. Codecs are moving targets for now, first stable codec will be H264 in 5.11, MPEG2 and VP8 will follow soon after, probably in 5.12 already. HEVC is in early stages, so that won't be stable soon.

Link to post
Share on other sites

It seems I may have found the "issue" in my failed trials post-May 2020 on my NanoPi M4 (rk3399)..

I discovered today (finally) that Kodi's internal ffmpeg isn't being configured with "--enable-libudev --enable-v4l2-request" resulting in missing hwaccels and obviously, no functional hardware acceleration. How this worked before in my build of May 2020 is a mystery (maybe ffmpeg from then auto-enabled these options by chance?).

I first tried to use the FFMPEG_EXTRA_FLAGS variable set on the commandline when invoking cmake, but since it is passed to the ffmpeg cmake script unquoted (see the commit adding it here: https://github.com/xbmc/xbmc/commit/cdbc12608b947d4afefc5bba48dc82e1f3514e52#diff-efa915ad129932095ae6f2a6d75e0d261386dd442b44338b4f6c705e25f1658eR241), only the first item ends up in ffmpeg_conf (due to CMake language rules I believe: https://gitlab.kitware.com/cmake/cmake/-/issues/17113#note_294727).

I have now added the arguments manually and confirmed they make it to ffmpeg, and am currently running a linuxtv-rkvdec-work-in-progress from @Kwiboo's linux-rockchip repository, rebased onto a clean v5.10.4.

Failing to notice this is kind of silly of me, and I'm sorry if I wasted anyone's time with my build configuration issues.

 

H264 decoding works great again (with DRMPRIME/DirectToPlane rendering enabled in the Kodi settings).

I do notice that the limited HEVC content that I have, which worked before (kernel 5.8-rc1 and Kodi/ffmpeg from May 2020), now shows a black screen.

 

Additionally, I tried merging in the changes from @Kwiboo's v5.10.1-drm-rockchip branch, and that gives me an option for a 4k resolution, but when playing back a 1080p H264 file, it appears only the top-left corner of the video output, including the black band around the 1920x800 movie, is shown full screen.

Playing back a 4k Big Buck Bunny MP4 file gives me "no signal" on my TV (it seems the actual 4k resolution is one of those @Kwiboo mentioned in the linked IRC conversation.

Is there a better branch to merge in from instead?

Is there any useful information I can provide to help this get fixed?

 

I hope the kernel v4l2 request uapi can move from staging soon (5.11?) so that the ffmpeg changes can be pulled in there.

 

As always: keep up the good work, I, along with many others, really appreciate what you're doing for these chips!

Link to post
Share on other sites

H264 related headers are moved to UAPI in 5.11 and thus considered stable. ffmpeg patches are already on ML but it will take some time before they're merged. HEVC is sadly still in verly early stages and basically useless without heavy patching. Next stable codecs will most likely be MPEG2 and VP8. Note that codecs are driven by developers interest and HEVC is not on top of the list for paying customers, most likely because of patents.

Link to post
Share on other sites

I am attempting to decode h264 rtsp stream from cameras using vpu. For this ffmpeg just needs to decode and write output to a pipe in the same yuv420p pixel format. I have tried various branches from @Kwiboo github on Armbian with 5.10.21 kernel. when I try with for example 

./ffmpeg -loglevel debug -hwaccel drm -i ~/Jellyfish_1080_10s_30MB.mp4 -benchmark -f null -

I get this and it falls back to software decoding

[h264 @ 0xaaaac0c432f0] Format drm_prime chosen by get_format().
[h264 @ 0xaaaac0c432f0] Format drm_prime requires hwaccel initialisation.
[h264 @ 0xaaaac0c432f0] ff_v4l2_request_init: avctx=0xaaaac0c432f0 hw_device_ctx=0xaaaac0c3e2e0 hw_frames_ctx=(nil)
[h264 @ 0xaaaac0c432f0] v4l2_request_probe_media_device: avctx=0xaaaac0c432f0 ctx=0xffffa8098030 path=/dev/media0 driver=hantro-vpu
[h264 @ 0xaaaac0c432f0] v4l2_request_probe_video_device: avctx=0xaaaac0c432f0 ctx=0xffffa8098030 path=/dev/video1 capabilities=69222400
[h264 @ 0xaaaac0c432f0] v4l2_request_try_format: pixelformat 875967059 not supported for type 10
[h264 @ 0xaaaac0c432f0] v4l2_request_probe_video_device: try output format failed
[h264 @ 0xaaaac0c432f0] v4l2_request_probe_video_device: avctx=0xaaaac0c432f0 ctx=0xffffa8098030 path=/dev/video2 capabilities=69222400
[h264 @ 0xaaaac0c432f0] v4l2_request_try_format: pixelformat 875967059 not supported for type 10
[h264 @ 0xaaaac0c432f0] v4l2_request_probe_video_device: try output format failed
[h264 @ 0xaaaac0c432f0] v4l2_request_probe_media_device: avctx=0xaaaac0c432f0 ctx=0xffffa8098030 path=/dev/media1 driver=rkvdec
[h264 @ 0xaaaac0c432f0] v4l2_request_probe_video_device: avctx=0xaaaac0c432f0 ctx=0xffffa8098030 path=/dev/video3 capabilities=69222400
[h264 @ 0xaaaac0c432f0] v4l2_request_probe_video_device: set controls failed, Invalid argument (22)
[h264 @ 0xaaaac0c432f0] Failed setup for format drm_prime: hwaccel initialisation returned error.
[h264 @ 0xaaaac0c432f0] Format drm_prime not usable, retrying get_format() without it.
[h264 @ 0xaaaac0c432f0] Format yuv420p chosen by get_format().

Using strace I see that the Invalid Argument (22) is coming from ioctl on /dev/video[0123] as it cycles through each one. It may be pixel format 875967059 that is the issue here, how can that be resolved, kernel patches? 

I feel I may be missing something fundamental or not understanding something correctly. For example is a window manager and X11 required for any of this? Most of the discussions are about rendering from what I can gather. My requirement is for ffmpeg to decode and output to pipe while saving segments along the way in original form, and also possibly serving up stream via rtmp too in original format. From the pipe the stream is examined for motion, if found passed off to object recognition on a coral usb device. Would I would be better off trying to get this working on the legacy kernel?

edit: removed -hwaccell_output drm_prime from ffmpeg command, its the same error without it.

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...