Mainline VPU

[jernej]

4 hours ago, Kwiboo said:

with ffmpeg command the cpu will be busy doing pixel format conversion (sw pixel conversion is also activated with null output).

This can be avoided by using

-hwaccel_output_format drm_prime

argument. Works good with null output.

[xmixahlx]

confirmed x264 10bit hwaccel is working. thanks so much, @Kwiboo

 

using kodi-git, ffmpeg 4.2.3 + new rkvdec branch work, linux 5.7-rc7 + librelec rockchip patches + linuxtv WIP branch diff + drm-next branch diff

 

questions:

 

will the ffmpeg patches be upstreamed? or moved into a separate library? wondering how this will evolve.

 

(and echoing my previous questions wondering about vaapi support for hantro+rkvdec and ffmpeg -decoders list issue above.)

Edited May 26, 2020 by xmixahlx

[jernej]

2 hours ago, xmixahlx said:

will the ffmpeg patches be upstreamed? or moved into a separate library? wondering how this will evolve.

That is the plan. However that can't happen until codecs interfaces in kernel are promoted to stable. That will probably take at least a few more kernel cycles. Nobody wants to screw it up user facing API so everyone is cautious. Actually, not all features are yet supported in kernel, that's also the reason you need kernel patches with experimental improvements.

 

2 hours ago, xmixahlx said:

wondering about vaapi support

Work on initial vaapi library stalled, but others might take over... Nothing useful at the moment unless you understand both APIs well and you're able to fix it.

[xmixahlx]

thanks @jernej ! really appreciate all of your work with @Kwiboo to provide these capabilities. awesome.

 

unfortunately, i'm not a kernel developer, but would love to help out. testing? documentation? LMK.

 

vaapi support for pinebookpro to allow hwaccel on firefox/wayland would be epic. in addition to the obvious youtube/video functions it seems like that would make jitsi meet usable, too.

[jernej]

FYI, VAAPI lib is just userspace translation layer between two apis. No kernel hacking needed, apart from applying LE patches.

[xmixahlx]

thanks. i have collected the vaapi hantro bits on the pine64 forum thread, which basically refers to ph5's work on libva, libva-v4l2-request, and gstreamer-vaapi to add support for hantro, but it seems broken.

 

vaapi for mpeg2 on hantro does work -- tested on vlc.

[jernej]

2 hours ago, xmixahlx said:

but it seems broken.

You have to make sure that libva-v4l2-request uses  same *-ctrl.h files that are in kernel sources under include/media/. For starters, just copy them over from kernel to lib source folder. Then you might need to adjust lib sources to new field names. Unless these files are completely the same, you won't have any success.

[xmixahlx]

thanks for the guidance, @jernej!

 

with both hantro and rkvdec supporting h264, and rkvdec having expanded capabilities, does it make sense to rely/prioritize rkvdec when detected? how does ffmpeg/v4l2-request decide which one to use?

[xmixahlx]

as i posted in the pine64 hwaccel thread:

the kernel i am testing now, working well is: (getting ready for 5.7!)

next20200529
pinebookpro patch: https://pastebin.com/YK4LXeyv
Kwiboo's LibreElec 1000 patch
Kwiboo's v4l2 patch: https://pastebin.com/gaQSZQ6B
Kwiboo's drm patch: https://pastebin.com/b1BWZ8Cg

responsiveness is actually significantly better than 5.7-rc7 (!?), and my nvme drive is also much better handled. interesting.

also, i would recommend compiling hantro and rkvdec into the kernel, not modules, so that they are detected in the same order consistently.

[xmixahlx]

also:

 

ffmpeg 4.2.3 with v4l2-request-hwaccel-4.2.2-rkvdec patch: https://pastebin.com/uF5kpABD
kodi-git
libva-v4l2-request_master + pulls 29+30
libva_master + pulls 332+340
(for libva 332 delete the 5th chunk, version bump, or apply against libva 2.6.0)
 

[xmixahlx]

@Kwiboo fyi i did encounter graphical artifacts similar to what rubenvb has noted with 10bit h264 and h265 using ffmpeg_master, ffmpeg_v4l2-request-hwaccel-master, and cherry picking last few rkvdec commits from v4l2-request-hwaccel-4.2.2-rkvdec.

 

however, the system stack i mentioned above earlier today works perfectly.

 

 

[Kwiboo]

2 hours ago, xmixahlx said:

fyi i did encounter graphical artifacts similar to what rubenvb has noted with 10bit h264 and h265 using ffmpeg_master, ffmpeg_v4l2-request-hwaccel-master, and cherry picking last few rkvdec commits from v4l2-request-hwaccel-4.2.2-rkvdec.

 

Thanks for testing, normally I try to keep ffmpeg_v4l2-request-hwaccel-master and ffmpeg_v4l2-request-hwaccel-4.2.2 in sync but I have not yet updated or tested the -master branch, I will try to update it in next few days.

 

 

Background on my different ffmpeg branches:

v4l2-request-hwaccel-x.y.z: Main branch that most of the time matches the patches used in LibreELEC master, x.y.z (4.2.2) matches the ffmpeg version used in Kodi master. This branch is usually tested on Rockchip/hantro/rkvdec by me and on Allwinner/cedrus by @jernej, fixup commits usually gets squashed right before we pick out a new patchset for use in LibreELEC.

v4l2-request-hwaccel-master: This branch should contain the patches in v4l2-request-hwaccel-x.y.z on top of ffmpeg master, may see some delays at times.

v4l2-request-hwaccel-4.2.2-rkvdec: work-in-progress branch for parts that currently only is expected to work on rockchip/rkvdec, e.g. 10-bit hevc decoding on cedrus stops working in this branch due to patches to handle rkvdec 10-bit decoding, patches should eventually flow to -x.y.z/-master branches once ready.

 

[xmixahlx]

understood. tyvm for context.

 

5.7 is released! thanks again for your work and others getting patches upstreamed.

 

i noticed vc1 and mpeg4 hwaccel work in @jernej
repos. is that coming to rk3399?

 

also bootlin has been active with v4l2-request-test with new hantro and other branches. and the bext branch was deleted in libva-v4l2-request. hopeful that these will be updated for v4l2 in 5.7!

 

with ffmpeg not displaying v4l2request hwaccels any more, it seems that mpv hwdec doesn't work. should i be doing something different?

[jernej]

24 minutes ago, xmixahlx said:

i noticed vc1 and mpeg4 hwaccel work in @jernej
repos.

That is very much work in progress, so it doesn't work anywhere yet, except basic MPEG4 on Cedrus. I'm not sure if I'll ever finish MPEG4 because it has a ton of variants and even Cortex A7 is capable decoding it. On the other hand, VC-1 has only 3 profiles and it's much more demanding for CPU decoding.

[xmixahlx]

understood. it does seem mpeg4 is a mess,.

 

hwaccel is a much bigger topic on the pinebookpro due to battery, although i'm sure power usage is still a concern with the usual sbc format.

 

obviously with libreelec and kodi you are focused on hwaccel decoding as many formats as possible. (thank you!) do you have any insight into hantro encoding status? i see hantro-vpu-enc detected in the kernel with jpeg support and just not aware of what is happening on that front.

[jernej]

25 minutes ago, xmixahlx said:

do you have any insight into hantro encoding status?

I don't. As you said, I'm solely focused on decoding. Maybe @Kwiboo knows?

[xmixahlx]

@Kwiboo did you have any idea about hantro encoding? ^

 

also, ffmpeg 4.3 and linux 5.8-rc1 are released and can't wait to update!

 

fyi, i have a new project, pbp-tools on git that is collecting these hwaccel things for pinebook pro:

https://github.com/xmixahlx/pbp-tools

https://forum.pine64.org/showthread.php?tid=10190

 

any idea on the ffmpeg -decoders bug not listing v4lrequest? that seems to break things like mpv that use detection. ffmpeg and kodi work OK.

 

Edited June 16, 2020 by xmixahlx

[xmixahlx]

@Kwiboo ffmpeg 4.3-rkvdec working well with most supported codecs. i do have some new pixellations with vp9. just checked, they were there in the 4.2.2-rkvdec, too.

 

in kodi, vp9 10/12bit and x265 12bit formats that are not hwaccel supported just freeze or crash kodi and are not being decoded via SW.

 

[Kwiboo]

23 hours ago, xmixahlx said:

ffmpeg 4.3-rkvdec working well with most supported codecs

 

Thanks for testing!, kodi has just moved to ffmpeg 4.3 so LibreELEC will follow as soon as all ffmpeg patches has been rebased.

 

23 hours ago, xmixahlx said:

i do have some new pixellations with vp9

 

Do you have a sample to share? Also note that only Profile 0 is supported and videos using frame resize is expected to have issues.

 

23 hours ago, xmixahlx said:

in kodi, vp9 10/12bit and x265 12bit formats that are not hwaccel supported just freeze or crash kodi and are not being decoded via SW.

 

I do not expect these formats to work, the kodi drmprime video codec and renderer sw decoding parts have mainly been tested with 8-bit video.

The crash should probably be fixed, unfortunately 10/12-bit sw decoding is not high on priority list, in future when ffmpeg filters are supported in kodi drmprime video codec converting to a 8-bit pixel format may be possible.

[xmixahlx]

@Kwiboo i've also tested your latest linux-rockchip WIP branch work patched to linux 5.8-rc2 with ffmpeg 4.3-rkvdec. working great on pinebookpro, minus the vp9 issue.

 

for the vp9 issue, this was either broken after rebasing 4.2.2, or with a newer kernel. i do recall it working fine earlier.

i've uploaded my clips here: (they are all the same short sample re-encoded with ffmpeg from big buck bunny with various codecs and settings for testing. )

https://github.com/xmixahlx/media-samples/tree/master/videos

 

 

[gounthar]

I'm not sure I understood everything in this thread (in fact, I'm sure I got less than 10%).
Anyway... My goal is to have hardware encoding and decoding for the Orange Pi 4B and RK3399 running Armbian.
I'm still running 5.4.49-rockchip64 kernel.
Is there an article somewhere for newbies in order to get a working hardware-enhanced version of ffmpeg for these boards?

Thanks.

[catalinii]

Hi @Kwiboo,

 

I did some testing using your version 2 of the patch:

1) Few seconds after starting the video, the screen gets blank for few seconds.

2) If the decoder gets stuck is hard to make it work again. I tried the soft reset patch does not help much.

A sample video is here (has few missing packets): https://minisatip.org/tmp/tmp2.ts

 

Thanks

[P.P.A.]

@xmixahlx How much of what your pbp-tools scripts are doing is specifically tailored to the Pinebook Pro (and would cause incompatibilities elsewhere)? Can most of them be run on any RK3399 platform?

[xmixahlx]

most scripts are not pbp specific, but that has been the focus so i am not sure for other devices. i would say test and give feedback after reading the scripts. i will be making these more generic as i expect to get a few HardRock64 devices when they are available.

 

the most specific is the linux build script because of the pbp patch and kernel config, but you can override this with KERNELCONFIG and use your own config to build for almost any rockchip device. I will make the pbp patch overrideable.

 

back on topic:

 

kwiboo's linux-rockchip WIP branch is working great - and we have a LOT of activity for v4l2 and rkvdec on linux-media. excited for 5.9.

 

kwiboo's ffmpeg 4.3 branch patch applies to 4.3.1 and master as well as release/4.3 -- all working well!

 

i shared the vp9 file that gives me problems above. the same file decodes fine with intel vaapi.

 

the ffmpeg frontend still does not share v4l2request support, so software that uses libavcodec like mpv are still broken due to lack of decoder detection and selection capability.

 

wildly thankful for the work of kwiboo and jernej and others. thanks!

 

 

 

 

[rubenvb]

Current Kodi master (currently hash 9f6d978f646) with the v4l2-request-hwaccel-4.3-rkvdec ffmpeg branch leads to audio only when starting video playback. The log produces this:

2020-07-23 20:32:06.003 T:57711    INFO <general>: Creating InputStream
2020-07-23 20:32:06.015 T:57711    INFO <general>: Creating Demuxer
2020-07-23 20:32:06.166 T:57711    INFO <general>: Opening stream: 0 source: 256
2020-07-23 20:32:06.166 T:57711    INFO <general>: Creating video codec with codec id: 27
2020-07-23 20:32:06.166 T:57711    INFO <general>: CDVDVideoCodecDRMPRIME::Open - using decoder H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
2020-07-23 20:32:06.167 T:57711    INFO <general>: Creating video thread
2020-07-23 20:32:06.168 T:57718    INFO <general>: running thread: video_thread
2020-07-23 20:32:06.168 T:57711    INFO <general>: Opening stream: 1 source: 256
2020-07-23 20:32:06.168 T:57711    INFO <general>: Finding audio codec for: 86020
2020-07-23 20:32:06.169 T:57711    INFO <general>: CDVDAudioCodecFFmpeg::Open() Successful opened audio decoder dca
2020-07-23 20:32:06.169 T:57711    INFO <general>: Creating audio thread
2020-07-23 20:32:06.169 T:57719    INFO <general>: running thread: CVideoPlayerAudio::Process()
2020-07-23 20:32:06.169 T:57711    INFO <general>: Opening stream: 2 source: 256
2020-07-23 20:32:06.172 T:57718    INFO <general>: CVideoPlayerVideo - Stillframe left, switching to normal playback
2020-07-23 20:32:06.180 T:57719    INFO <general>: Creating audio stream (codec id: 86020, channels: 8, sample rate: 48000, no pass-through)
2020-07-23 20:32:06.184 T:57718   ERROR <general>: CDVDVideoCodecDRMPRIME::AddData - send packet failed: Invalid data found when processing input (-1094995529)
2020-07-23 20:32:06.184 T:57718   ERROR <general>: CDVDVideoCodecDRMPRIME::AddData - send packet failed: Invalid data found when processing input (-1094995529)
2020-07-23 20:32:06.185 T:57718   ERROR <general>: CDVDVideoCodecDRMPRIME::AddData - send packet failed: Invalid data found when processing input (-1094995529)
...
2020-07-23 20:32:06.589 T:57718   ERROR <general>: CDVDVideoCodecDRMPRIME::AddData - send packet failed: Invalid data found when processing input (-1094995529)
2020-07-23 20:32:07.006 T:57718   ERROR <general>: CDVDVideoCodecDRMPRIME::Drain - send packet failed: Invalid data found when processing input (-1094995529)
2020-07-23 20:32:07.006 T:57718   ERROR <general>: CDVDVideoCodecDRMPRIME::GetPicture - receive frame failed: Invalid data found when processing input (-1094995529)
2020-07-23 20:32:07.006 T:57718    INFO <general>: CVideoPlayerVideo - Stillframe detected, switching to forced 23.976024 fps
2020-07-23 20:32:07.047 T:57718   ERROR <general>: CDVDVideoCodecDRMPRIME::GetPicture - receive frame failed: Invalid data found when processing input (-1094995529)
2020-07-23 20:32:07.064 T:57718    INFO <general>: CVideoPlayerVideo - Stillframe left, switching to normal playback
2020-07-23 20:32:07.064 T:57718   ERROR <general>: CDVDVideoCodecDRMPRIME::AddData - send packet failed: End of file (-541478725)
2020-07-23 20:32:07.064 T:57718   ERROR <general>: CDVDVideoCodecDRMPRIME::GetPicture - receive frame failed: Invalid data found when processing input (-1094995529)
2020-07-23 20:32:07.065 T:57718   ERROR <general>: CDVDVideoCodecDRMPRIME::AddData - send packet failed: End of file (-541478725)
2020-07-23 20:32:07.065 T:57718   ERROR <general>: CDVDVideoCodecDRMPRIME::GetPicture - receive frame failed: Invalid data found when processing input (-1094995529)
2020-07-23 20:32:07.148 T:57718   ERROR <general>: CDVDVideoCodecDRMPRIME::AddData - send packet failed: End of file (-541478725)
2020-07-23 20:32:07.231 T:57718   ERROR <general>: CDVDVideoCodecDRMPRIME::AddData - send packet failed: End of file (-541478725)

To be honest, this has been like this for a while, and my old build (kodi hash 9f6d978f646 with the 4.2.2 ffmpeg branch) works fine.

If I translate the hashes into "revisions", I get r55120 (working) vs r55380 (hash e64b3034b56) broken, which means roughly 260 commits that might have broken it for me.

I fear the move to ffmpeg 4.3 is at least partially to blame.

I will try to bisect this better, but the old revision won't build due to some networking related function pointer not being convertible without -fpermissive.

I'll need to find some time to bisect, so I don't know when I'll get to it.

 

Any thoughts or anything I can try to provide more info or at best fix this issue without bisecting kodi?

 

Thanks for everything so far, you guys are great!

[xmixahlx]

i have been updating the kernel, ffmpeg, and kodi in sequence when major new patches from @Kwiboo are updated with consistent success, minus the known issues.

Edited July 25, 2020 by xmixahlx

[P.P.A.]

After my last post here, I successfully applied @xmixahlx's linux-5.8-rc4-hwaccel_20200709.diff to balbes150's Armbian Build fork (as a userpatch). For various reasons I was unable to test it properly though.

 

5.8 having been officially released, I would like to start over and re-trace your steps and patch the kernel the same way you did with 5.7 and 5.8-rc, mainly in order to enable 10-bit HEVC decoding. I read the entirety of this thread and the two at the Pine64; please tell me if I got all of this right:

The first patches you applied were the ones from LibreELEC's kernel at https://github.com/LibreELEC/LibreELEC.tv/tree/master/projects/Rockchip/patches/linux/default – currently these would be:

Spoiler

linux-0001-rockchip-from-5.9.patch
linux-0002-rockchip-from-next.patch
linux-0011-v4l2-from-5.9.patch
linux-0021-drm-from-5.9.patch
linux-0022-drm-from-next.patch
linux-1000-export-mm_trace_rss_stat.patch

Next, you created further patches by diffing vanilla kernel(+patches to get it up to 5.8-rc4)+libreelec patches against @Kwiboo's WIP linux-rockchip fork at https://github.com/Kwiboo/linux-rockchip in order to get his latest improvements that haven't made it into LibreElec yet (like H.265 decoding). The linuxtv-rkvdec-work-in-progress master is already on 5.9-rc1 though – would this create conflicts (or force me to patch the kernel to be diffed against up to 5.9-rc1 as well, which I would rather avoid), and should I use these individual 5.8 branches instead?

Spoiler

https://github.com/Kwiboo/linux-rockchip/commits/rockchip-5.8-from-5.10-v5.8
https://github.com/Kwiboo/linux-rockchip/commits/drm-5.8-from-5.10-v5.8
https://github.com/Kwiboo/linux-rockchip/commits/v4l2-5.8-from-5.9-v5.8

 

What should be done about the kernel config? Should I just look at your pbp patch's config and enable the relevant bits (like CONFIG_VIDEO_ROCKCHIP_VDEC) accordingly in my own? (If it isn't obvious, I'm very new to all of this.)

If the kernel compiles with the LibreELEC patches and the Kwiboo diffs, the next step is compiling FFmpeg 4.3 from https://github.com/Kwiboo/FFmpeg/tree/v4l2-request-hwaccel-4.3.1-rkvdec and finally mpv with the appropriate patch, or Kodi if that doesn't work.

(Hopefully I'm able to do all this within armbian-build using userpatches, seeing how rockchip64-dev seems to be on 5.8.y now.)

[Marco Diego Aurélio Mesqui]

Does https://www.spinics.net/lists/linux-media/msg175721.html and https://gstreamer.freedesktop.org/releases/1.18/ means that gstreamer based apps will be able to use hardware (rkvdec and hantro) to decode h264 streams with linux 5.10 and gstreamer 1.18?

Edited September 8, 2020 by Marco Diego Aurélio Mesqui
typo

[xmixahlx]

@P.P.A. follow the patches from libreelec or pbp-tools. i rebuild with kernel > ffmpeg > kodi. you can see what else i use in pbp-tools.

 

@Marco Diego Aurélio Mesqui h264 and vp8 have been supported for quite some time in development. you can see the v4l2codecs info in the release. easiest way to do this is just use kwiboo's WIP patches now.

[Marco Diego Aurélio Mesqui]

  

Quote

@Marco Diego Aurélio Mesqui h264 and vp8 have been supported for quite some time in development. you can see the v4l2codecs info in the release. easiest way to do this is just use kwiboo's WIP patches now.

 

I mean, with linux 5.10 and gstreamer 2.18 no patches will be needed, right?