The VPU driver

[Kwiboo]

1 hour ago, Myy said:

Well, ok, it obviously couldn't work since I used the wrong kernel tree... I'll retry my tests and fix my script this week-end...

Yes, looks like the script used one of the branches, I do not have one branch with all patches, I tend to group patches into multiple branches to make it easier rebasing and review changes.

If you keep the hdmi-sound base as you have and add https://github.com/Kwiboo/linux-rockchip/compare/rockchip-5.x-hdmi-sound...rockchip-5.x-vpu-v2.patch as a patch you should have both branches.

 

1 hour ago, Myy said:

Yeah, these little beasts are fast for MPEG-2... Too bad that the web is all about H264 and VP8

The 1080p H264, VP8 samples I have tested on my tinker board and even my rock64 running libreelec / kodi-gbm seems to play back fine using sw decoding.

There are some libreelec test images at http://kwiboo.libreelec.tv/test/ for testing, built out of https://github.com/Kwiboo/LibreELEC.tv/commits/rockchip-5.x

[Myy]

Okay, I got it working with MPEG-2 ... Well, at least the logs are coherent this time.

 

You just have to be *sure* that FFMPEG *IS* compiled with v4l2_request enabled AND the option is also enabled in the kernel...

 

I'll release the compilation script in a few minutes, along with the patch I applied to Kwiboo's FFMPEG branch, and then extract the minimum set of patch and apply them to mainline kernel in a few days.

 

Note that this only for MPEG-2

H264 will take some time.

I don't know if the RK3288 decoder supported VP8.

 

[Kwiboo]

Great!

 

The VPU in rk3288 (vpu1) should have VP8 support, same as the VPU in rk3328/rk3399 (vpu2)

rk3288 has a second VPU block that can decode hevc

rk3328/rk3399 has a second VPU block (rkvdec) that can decode h264/hevc/vp9 (they have two different vpu blocks that can decode h264)

 

Just to let you know, I have moved to v5.1 now and latest RK VPU patchset v5 on top of linux v5.1 can be found at end of https://github.com/Kwiboo/linux-rockchip/compare/v5.1...rockchip-5.1-v4l2-from-list-v5.1 (along with a lot of probably unneccecery v4l2 patches)

[Myy]

So :

 

Here's the build script : https://github.com/Miouyouyou/RockMyy/tree/VPU-V4L2-5.x

 

The FFMPEG version I used can be retrieved like this :

Quote

git clone --depth 1 https://github.com/Kwiboo/FFmpeg -b v4l2-request-hwaccel

 

Then, you'll have to apply this patch :

https://github.com/Miouyouyou/RockMyy/blob/VPU-V4L2-5.x/0001-Patch-applied-to-compile-against-a-specific-Kwiboo-s.patch
 

Quote

 

wget https://github.com/Miouyouyou/RockMyy/blob/VPU-V4L2-5.x/0001-Patch-applied-to-compile-against-a-specific-Kwiboo-s.patch

git apply *.patch

 

 

Then, you'll to compile FFmpeg... The part I love the best. It will take roughly 1 hour on a Tinkerboard. Just do something else, or find a way to cross-compile :

Quote

./configure --enable-v4l2-request --enable-libdrm --enable-libudev --enable-shared --enable-pic

*ALL* the options are *REQUIRED* if you want to build a shared library, with V4L2 Request support.

 

Then, you need a media player and a MPEG-2 sample.

For the sample, here it is : https://samplemedia.linaro.org/MPEG2/big_buck_bunny_480p_MPEG2_MP2_25fps_1800K.MPG

For the media player, I used MPV (I took the 0.29.1 release). Compile it against your version of FFMPEG and, before installing it, get into the build directory and try it with the following command :

Quote

LD_LIBRARY_PATH=/usr/local/lib ./mpv --vo=gpu --gpu-context=drm --hwdec=drm /root/big_buck_bunny_480p_MPEG2_MP2_25fps_1800K.MPG -v --msg-level=ffmpeg=trace

 

You might want this patch to use the GLES renderer with MPV : https://gist.github.com/Miouyouyou/b9273ee3d949db3e1eb12f6bf99c1101

 

 

[Myy]

6 minutes ago, Kwiboo said:

Great!

 

The VPU in rk3288 (vpu1) should have VP8 support, same as the VPU in rk3328/rk3399 (vpu2)

rk3288 has a second VPU block that can decode hevc

rk3328/rk3399 has a second VPU block (rkvdec) that can decode h264/hevc/vp9 (they have two different vpu blocks that can decode h264)

 

Just to let you know, I have moved to v5.1 now and latest RK VPU patchset v5 on top of linux v5.1 can be found at end of https://github.com/Kwiboo/linux-rockchip/compare/v5.1...rockchip-5.1-v4l2-from-list-v5.1 (along with a lot of probably unneccecery v4l2 patches)

 

Nice ! I'll take a look at this branch and try to extract the minimal set, in order to test it on mainline kernels.

 

Now, I guess that for H264, I'll need to port this : https://github.com/Miouyouyou/Mainline-Rockchip-VPU/blob/dev/based_on/rk3288_vpu_hw_h264d.c ?

Is there test code available for RK3288 or RK3399 ? Or is it too soon ?

 

Meanwhile, @JMCC , do you feel like trying to create a script, that automatically build FFMPEG + MPV with V4L2 Request support ? Or shall I ?

[Myy]

Anyway, I'll get some sleep and, tomorrow, I'll see if I can retest this on RK3399 devices like a NanoPC-T4 (and maybe the Orange Pi 3399... though my last try with the Orange PI and mainline kernels was terrible).

[Kwiboo]

3 minutes ago, Myy said:

Is there test code available for RK3288 or RK3399 ? Or is it too soon ?

 

There are other people working on porting h264/vp8 to mainline driver, e.g. https://github.com/bbrezillon/linux/commits/rk-vpu1-h264, nothing I have tested yet.

[Myy]

Interesting. I'll take a look at their work.

[JMCC]

4 hours ago, Myy said:

Meanwhile, @JMCC , do you feel like trying to create a script, that automatically build FFMPEG + MPV with V4L2 Request support ? Or shall I ?

Does it need to have also RKMPP support, or just V4L2 is enough for the new driver?

[Myy]

V4L2, with the right kernel headers, is enough. In this version RKMPP isn't needed.

 

Now, there's also a libva driver being developed. I didn't test it, though.

It seems to be used in combination with VLC. I'll test it if I can get the H264 decoder working.

[JMCC]

In that case, mpv builds with V4L2 support by default. Just run on the TinkerBoard:

$ sudo apt remove mpv
$ git clone https://github.com/mpv-player/mpv-build.git
$ cd mpv-build
$ ./rebuild -j4
$ sudo ./install

This will install mpv in your local machine. If you also want to build a debian package, you can do the following additional steps:

$ sudo apt-get install devscripts equivs fakeroot
$ mk-build-deps -s sudo -i
$ dpkg-buildpackage -uc -us -b -rfakeroot -j4

The resulting mpv package can later be installed on any machine running the same distro and architecture.

 

In order to run mpv using the v4l2 support, this is the command line that I use:

mpv -hwdec <filename>

 

[JMCC]

The following deb package should work for Armbian Bionic arm64 (RK3328/RK3399):

https://1drv.ms/u/s!Aj0xDhqxX_nqoygqTuH8eoKnP3in

[Myy]

Nice !

 

However, since Kwiboo actually patched FFMPEG to add V4L2 request support, the FFMPEG might actually be neeed to. I'll try to run the standard Debian packaging sequence you pasted.

[Myy]

Instead of doing a proper packaging job, I decided to test the H264 decoder because... Hey, H264 ! If it works, it means we can actually use RK3288 boards to watch YouTube, Netflix and Twitch.tv, boosting productivity by 10x !

 

But, nah... the H264 is still a Work In Progress. While the basics work (Sending the V4L2 request, Getting an image back), I tried with a video that was lying on my PC HDD, their FFMPEG and mpv and... I got some weird blocks interlacing with frames replacing each other...

 

It's not ready. But there's clear progress made.

 

Still, the kernel from bbrazillon can be used for MPEG-2 decoding too, since it integrates Kwiboo's work, so I'll use this base, strip the changes to the bare minimum required by the V4L2 decoders, generate a patch, integrate it into my main build script and let everyone have fun with this.

Then I'll retry with RK3399 boards.

[Myy]

... So, just to see what are the latest LKML gossips, I went to LKML.org. Besides the common people throwing a tantrum because they provided their work under a license and want to rescind it, for whatever reasons, I saw this :

Hardware-accelerated video decoders used through a firmware instead of hardware registers

 

I skimmed that thread, by curiosity. Basically, Paul Kocialkowski and Nicolas Dufresne are talking about current developments in the Video decoders, and how to deal with angry closed blobs coming from nowhere.

But then, Nicolas then threw this, out of nowhere :

Quote

Oh, I might have forgot to mention, we accidentally found that on RK3288, the decoder is in fact an Hantro G1 (the full version), IMX8M has a small version of Hantro G1 (vp8 and h264 only).

... !!?

 

Ok, let's search on the web for "Hantro G1". Maybe I'll find the Reference Manual for this chip.

And I stumbled on this :

https://github.com/linux4sam/gst1-hantro-g1 (Gstreamer implementation)

https://github.com/linux4sam/g1_decoder

 

Oookay... Maybe this doesn't work on RK3288 boards. I'll have to check.

 

Still, they *REALLY* look similar :

https://github.com/linux4sam/g1_decoder/blob/master/software/source/common/8170enum.h#L38

https://github.com/Miouyouyou/Mainline-VPU-V4L2-Edition/blob/master/rk3288_vpu_regs.h#L155

 

ROCKCHIP ! What the FUCK !? Why did you ask Google to develop a driver for your VPU, while the company already had all the documentation about your chip !?

 

By the way, the chip in itself, is able to decode a good lot of formats :

http://www.verisilicon.com/IPPortfolio_14_58_2_HantroG1.html

 

Quote

 

H.264 Baseline, Main and High Profiles, levels 1 – 5.1

H.264 MVC Stereo High Profile

VP8 for WebM, WebRTC and WebP support

MPEG-4 Simple and Advanced Simple Profiles, levels 0 – 5

Sorenson Spark and H.263 Profile 0, levels 10 – 70

WMV9 / VC-1 Simple, Main and Advanced Profile, levels 0 - 3

MPEG-1&2 Main Profile, levels low, med and high

RealVideo 8/9/10

DivX ® 3/4/5/6 support – Home Theatre Profile Qualification

VP6,VP7

AVS Jizhun Profile

AVS+ (AVS1-P16 Level 6.0 and 6.2) 4:2:0 color format 

JPEG and MJPEG support

 

 

 

[Kwiboo]

Yep, the vpu1 is a Hantro G1 (or at least based on it). G1 is also used in IMX8 and details are in the IMX8M reference documentation.

If you look for rk PX3 trm you also have some docs on the vpu1.

 

Vpu2 is very similar to vpu1 but with regs in different positions.

 

And imx8 g1 decoder code at https://github.com/CliveLau1990/imx-vpu-hantro/tree/master/decoder_sw/software/source is interesting.

[Myy]

8 hours ago, Kwiboo said:

Yep, the vpu1 is a Hantro G1 (or at least based on it). G1 is also used in IMX8 and details are in the IMX8M reference documentation.

...

May I know since when people working on RK3288 VPU knew this ?

Did someone try to port the iMX VPU and found "Hey... this looks similar !" ?

 

And, I know it's bad. If you got a mail from iMX, I'll remove the PDF @Igor.

However, for the record, here's the VPU registers documentation :

 

out.pdf

[Myy]

Here's a first draft of the patches I pulled from bbrezillon tree : https://github.com/Miouyouyou/RockMyy/tree/SplittingPatchesForVPU/patches/kernel/v5.1/VPU

These are applied against mainline v5.1-rc5 kernels, and have been tested with Kwiboo's FFMPEG and a standard MPV.

 

I'll try to test them, and adapt them, against v5.1 releases tomorrow. Then I'll re-arrange them and do a release of RockMyy.

 

I'll then generate a FFmpeg package with Kwiboo's patch, using Kwiboo's tree, since bbrezillon's one doesn't integrate udev /dev/video node detection.

 

And *then* I'll give the whole thing to test to @TonyMac32, who loves testing random VPU patches for Tinkerboards.

[Myy]

VPU patches re-ordered, remade and tested against 5.1 release kernel, with the usual patches combined.

 

I retested the H264 driver with a slight fix and it's better, but there's still a few blocks not placed correctly here and there.

This might be resolved this month, though.

 

Anyway, it's MPEG-2 decoding only for the moment.

 

Note to @TonyMac32, if you want to import these patches, the actual kernel patch list I'm using is there : https://github.com/Miouyouyou/RockMyy/blob/master/GetPatchAndCompileKernel.sh#L57

 

 

Now, what remains is FFmpeg.

I don't remember if Ubuntu is still using libav, or went back to FFmpeg. I remember that the projects forked for some weird reasons and Debian went with libav...

So, basically, do I create a package, or just a tarball with the libraries. With just the tarball, you'll have to extract them somewhere and run LD_LIBRARY_PATH=/path/to/ffmpeg/libs to use them with mpv and such.

With the Debian package, you get the advantages (automatic dependencies handling) along with all the troubles (packages conflicts).

 

I still have to test bootlin libva too, since it seems to go well with VLC.

[JMCC]

2 hours ago, Myy said:

Now, what remains is FFmpeg.

Correct me if I'm wrong, but I understand that the new V4L2 driver only supports HW decoding, not encoding, right?

 

In that case, IMO the usefulness of an standalone FFmpeg is very limited.  What I mean is that you won't get a noticeable benefit in transcoding with FFmpeg if you only accelerate the decoding. Not to mention possible colorspace conversion limitations etc (I think I bumped into some of those with the "old" RKMPP FFmpeg implementation).

 

On the other hand, I see the real usefulness of that when FFmpeg is used as part of media players such as MPV or Kodi. And, in those cases, I always found it easier and more efficient to compile its own version of FFmpeg with the app, than link it against the system libs.

 

So my opinion is that it is better to leave alone the system FFmpeg debs,  and compile the new version on a separate directory; that way it will not break other FFmpeg-dependent apps, such as video editors etc.. And let the media players use their own FFmpeg (after all that is what they normally recommend in their compilation instructions), and apply the patch there when necessary.

[Myy]

Ah, indeed, I forgot about statically linking some media players, like MPV, against patched FFmpeg. I'll use that for MPEG-2 support.

 

And, yeah, Chromium and Firefox bundle their own FFmpeg, so they'll need to be patched independently once H.264, VP8 or MPEG-4 full support hit the kernel.

[Myy]

Trying to compile FFMPEG statically worked...

Trying to compile MPV, though, was a disaster since it wants *every potential library* as a statically linked library when trying to link against FFMPEG... Which makes no sense to me, if you link against a statically linked FFMPEG, why would need a static libdrm and libudev ?

Still, throwing "-Wl,-Bstatic" before "-ludev" result in ldd not finding the right libraries and I don't see how to resolve the problem in a "nice" way (meaning not hacking through the configure script).

 

 

So, anyway, I decided to provide the dynamically linked FFMPEG and mpv instead.

 

You'll need the following libraries to use the provided libraries :

ld-linux-armhf.so.3 (Available on any ARM debian system)
libc.so.6 (GlibC - installed by default)
libdrm.so.2 (libdrm2)
libm.so.6 (GlibC - installed by default)
libpthread.so.0 (GlibC - installed by default)
libudev.so.1 (libudev1)
libva-drm.so.2 (libva-drm2)
libva.so.2 (libva2)
libva-x11.so.2 (libva-x11-2)
libvdpau.so.1 (libvdpau1)
libX11.so.6 (libx11-6)
libxcb-shape.so.0 (libxcb-shape0)
libxcb.so.1 (libxcb1)
libxcb-xfixes.so.0 (libxcb-xfixes0)
libz.so.1 (zlib1g)

 

So, basically :

apt install libdrm2 libudev1 libva-drm2 libva2 libva-x11-2 libvdpau1 libx11-6 libxcb-shape0 libxcb1 libxcb-xfixes0 zlib1g

Or maybe just

apt install libdrm2 libudev1 libva2 libva-x11-2 libvdpau1 libx11-6  libxcb1 zlib1g

 

For mpv, you'll need :

ld-linux-armhf.so.3 (Available on any ARM debian system)
libass.so.9 (libass9)
libc.so.6 (GlibC)
libdl.so.2 (Available on any ARM debian system)
libdrm.so.2 (libdrm2)
libEGL.so.1 (libegl1)
libgbm.so.1 (libgbm1)
libjpeg.so.8 (libjpeg1)
libluajit-5.1.so.2 (libluajit-5.1-2)
libm.so.6 (GlibC)
libpthread.so.0 (GlibC)
librt.so.1 (GlibC)
libz.so.1(zlib1g)

So basically :

apt install libass9 libdrm2 libegl1 libgbm1 libjpeg1 libluajit-5.1-2 zlib1g

 

ffmpeg-mpv-dyn.tar.xz ffmpeg-includes.tar.xz

[Myy]

Alright, I understand that downloading some precompiled binary by who-knows-who, and tinkering here and there to get it working, sounds shady and fucking annoying.

 

So, instead, I made a script that allows you to download a fresh official FFmpeg, apply @Kwiboo and @jernej patches against it to get V4L2 Request API support, and compile the whole thing (binary and dynamic libraries).

Now, this *won't* install the patched FFmpeg, for reasons stated above.

 

https://github.com/Miouyouyou/FFmpeg-V4L2-Request 

 

I didn't apply bbrezillon patches, since I haven't tested his latest modifications and, ATM, there were issues here and there. I'd love some good H264 support, but you'll have to wait a bit.

[Myy]

2 hours ago, Myy said:

I didn't apply bbrezillon patches, since I haven't tested his latest modifications and, ATM, there were issues here and there. I'd love some good H264 support, but you'll have to wait a bit.

Yeah, just retried with his latest patches (both kernel and FFmpeg using some new flag M2M_HOLD_CAPTURE_BUF WIP) and it's better, but blocks are still flickering a little.

However, most of the issues are gone so, I guess that between mid-June~July H264 support will be top notch.

 

VP8 support has been added to the driver, but either his FFmpeg is not able to decode this format, or MPV refuses to use this hardware decoder ("codec vp8 is not on whitelist" error).

[petatester]

On 5/14/2019 at 1:49 PM, Myy said:

... So, just to see what are the latest LKML gossips, I went to LKML.org. Besides the common people throwing a tantrum because they provided their work under a license and want to rescind it, for whatever reasons, I saw this :

Hardware-accelerated video decoders used through a firmware instead of hardware registers

 

I skimmed that thread, by curiosity. Basically, Paul Kocialkowski and Nicolas Dufresne are talking about current developments in the Video decoders, and how to deal with angry closed blobs coming from nowhere.

But then, Nicolas then threw this, out of nowhere :

... !!?

 

Ok, let's search on the web for "Hantro G1". Maybe I'll find the Reference Manual for this chip.

And I stumbled on this :

https://github.com/linux4sam/gst1-hantro-g1 (Gstreamer implementation)

https://github.com/linux4sam/g1_decoder

 

Oookay... Maybe this doesn't work on RK3288 boards. I'll have to check.

 

Still, they *REALLY* look similar :

https://github.com/linux4sam/g1_decoder/blob/master/software/source/common/8170enum.h#L38

https://github.com/Miouyouyou/Mainline-VPU-V4L2-Edition/blob/master/rk3288_vpu_regs.h#L155

 

ROCKCHIP ! What the FUCK !? Why did you ask Google to develop a driver for your VPU, while the company already had all the documentation about your chip !?

 

By the way, the chip in itself, is able to decode a good lot of formats :

http://www.verisilicon.com/IPPortfolio_14_58_2_HantroG1.html

 

 

 

Wow i didn't know that the vpu had an independent manufacturer. Gonna read more info about the hantro

[jernej]

Just now, petatester said:

Wow i didn't know that the vpu had an independent manufacturer.

There is also VP9 decoding IP core , which is present in Allwinner H6, NXP i.MX8 and I think Rockchip has it too. It may be connected to Hantro and Google, but I'm unsure who is the original author.

[Myy]

1 hour ago, jernej said:

There is also VP9 decoding IP core , which is present in Allwinner H6, NXP i.MX8 and I think Rockchip has it too. It may be connected to Hantro and Google, but I'm unsure who is the original author.

Interesting find. They seem to talk about filling up a "form" to get more informations, but I can't find that form.

Do you know if the hardware decoder documentation is still available on this site ?

[jernej]

5 minutes ago, Myy said:

Do you know if the hardware decoder documentation is still available on this site ?

It was never publicly available AFAIK. However, you can download i.MX8 documentation which has small overview of this core and registers description.

[e97]

Is V4L2 for rk3288 encoder mainline or is there a patch set?

[gounthar]

On 5/19/2019 at 12:47 AM, Myy said:

Alright, I understand that downloading some precompiled binary by who-knows-who, and tinkering here and there to get it working, sounds shady and fucking annoying.

 

So, instead, I made a script that allows you to download a fresh official FFmpeg, apply @Kwiboo and @jernej patches against it to get V4L2 Request API support, and compile the whole thing (binary and dynamic libraries).

Now, this *won't* install the patched FFmpeg, for reasons stated above.

 

https://github.com/Miouyouyou/FFmpeg-V4L2-Request 

 

I didn't apply bbrezillon patches, since I haven't tested his latest modifications and, ATM, there were issues here and there. I'd love some good H264 support, but you'll have to wait a bit.

Hi @Myy,

this did not work for me today, so I entered a new issue there. Thanks.