Hardware Graphic/Video Acceleration in H3 Mainline

[ATK]

Last night I downloaded "https://dl.armbian.com/orangepiplus2e/Debian_buster_next.7z" (OPi+2E) ... burned it to SD card and fired up the board ..later installed desktop and other software ...BUT when I was installing MPV I observed that few files with "Vulcan" in their names were also installed...so does that mean that mainline next 4.19.xx images for H3  are now fully suitable for multimedia desktops with Hardware graphics/Video acceleration fully supported ?

[ning]

see this: http://linux-sunxi.org/Sunxi-Cedrus

[Igor]

1 hour ago, ATK said:

are now fully suitable for multimedia desktops with Hardware graphics/Video acceleration fully supported ?

Not yet. Perhaps with 5.3.y

[drice]

Mainline kernel up to 5.2 only supports MPEG2 decode in the sunxi-cedrus driver at this time so even if it was working that would be a major limitation. It looks like H264 support has been added in 5.3. The bigger challenge is that the libva-v4l2-request userspace component seems to depend on some features in a custom bootlin kernel (linux-cedrus) that haven't been upstreamed into mainline yet. I'm trying to get a working proof of concept by merging the 5.3.y driver and some of the updated header files from linux-cedrus into the Armbian sunxi-dev kernel, but I don't know if it's going to work. I'm pretty new at kernel stuff.

[FGuerzoni]

I tried to do same thing some months ago without success working with nanopi neo.

I had problems applying the patches (manually derived from https://github.com/bootlin/linux-cedrus  [release-2019.03] ) to armbian dev because I'm quite new to armbian and kernel stuff.

But recently I noticed that both mainline next and 5.3-rc3 added h.264 decoding (baseline profile):

Quote

 

media: cedrus: Add H264 decoding support

Introduce some basic H264 decoding support in cedrus. So far, only the baseline profile videos have been tested, and some more advanced features used in higher profiles are not even implemented.

 

I really cannot manage to point armbian build system to either 5.3-rc3 or next-20190808.

I think also that should be important to check (from https://linux-sunxi.org/Sunxi-cedrus):

Quote

 

The following kernel configuration options must be selected:

CONFIG_MEDIA_SUPPORT

CONFIG_MEDIA_CONTROLLER

CONFIG_V4L_MEM2MEM_DRIVERS

CONFIG_VIDEO_SUNXI_CEDRUS

 

So if anyone could make armbian build system points to either 5.3-rc3 or next-20190808, I think it could work.

Any help would be appreciated.

BR

F

 

[martinayotte]

15 hours ago, FGuerzoni said:

o if anyone could make armbian build system points to either 5.3-rc3

You can try to change Armbian config/sources/sunxi_common.inc file to point to Megous branch 5.3, but be aware that maybe several current patches won't be apply properly ...

[FGuerzoni]

Thank you very much for the tip.

I built the image without any particular issue and did the boot:

 

Welcome to Ubuntu Bionic with Armbian Linux 5.3.0-rc3-sunxi
System load:   0.14 0.24 0.11      Up time:       3 min        
Memory usage:  14 % of 491MB      IP:            192.168.1.187
CPU temp:      42°C               
Usage of /:    6% of 15G       

 

I can start to check if h.264 baseline decoder is working with the current image

Regards

F

 

[guidol]

11 hours ago, FGuerzoni said:

Welcome to Ubuntu Bionic with Armbian Linux 5.3.0-rc3-sunxi
System load:   0.14 0.24 0.11      Up time:       3 min        
Memory usage:  14 % of 491MB      IP:            192.168.1.187
CPU temp:      42°C               
Usage of /:    6% of 15G       

 

 

also a small compile-test for kernel 5.3 did work on the Orange Pi One

 

Welcome to Debian Buster with Armbian Linux 5.3.0-rc3-sunxi
package bsp-kernel[5.93] u-boot[5.93] dtb[5.93] firmware[5.93] config[5.93]

System load:   0.81 0.24 0.08   Up time:       0 min
Memory usage:  14 % of 491MB    IP:            192.168.6.114
CPU temp:      45°C
Usage of /:    7% of 15G

Last login: Sat Aug 10 20:37:36 2019 from 192.168.6.17

root@opi-one(192.168.6.114):~#

 

[drice]

After doing a lot of reading (documentation and code) regarding the Cedrus driver (sunxi-cedrus), LibVA V4L2 Request API driver (libva-v4l2-request), VA-API, and interfacing with video players, I think most of the work will involve VA-API and interfacing with the video players. Or more precisely, updating the video players to interface with the LibVA driver in a way that works with the V4L2 Request API. Based on my testing, the Cedrus driver in sunxi-dev (5.2.6 - MPEG2 only) is working with the native V4L2 Request API as shown by testing with v4l2-request-test and the preset slice data included with that tool. When I run the test, I get hardware accelerated full screen output via DRM.

 

But so far I haven't been able to get any video playback to work with VA-API in mpv or vlc. mpv fails with "device or resource busy" when calling v4l2_set_format() in X11 and says that the hardware decoder isn't compatible with DRM. vlc fails after rendering the first frame with "buffer deadlock prevented". I still have more reading to do, and I have a few things to try that would be as simple as compiling the players with a few additional options set. But if that doesn't work this could end up being a complicated problem to solve.

[FGuerzoni]

@drice

 

how did you manage to get /dev/video0 working as required by v4l2-request-test?

Without it's impossible to have the buffers source.

regards

F

[drice]

/dev/video0 is handled by the Cedrus driver.  This needs to be compiled using the Armbian build system in expert mode. Build a kernel package with the dev kernel and build the Allwinner Cedrus VPU driver as a module (CONFIG_VIDEO_SUNXI_CEDRUS=m). If you don't see the option in the configuration, you need to enable the experimental media controller request API first (CONFIG_MEDIA_CONTROLLER_REQUEST_API=y). Don't get confused by Allwinner sunXi family Video Devices (CONFIG_VIDEO_SUNXI). That's not enough to build the driver. If you enable that and nothing is underneath, you haven't met the prerequisites for the Cedrus driver.

 

I know I had to manually edit some header files to compile libva-v4l2-request but unfortunately I didn't take good notes on that. If you run into trouble compiling v4l2-request-test (undefined identifiers relating to video formats) post back here or search the bootlin linux-cedrus repository for definitions.

[FGuerzoni]

Thank you very much. I missed CONFIG_VIDEO_SUNXI_CEDRUS.

 

After doing the image rebuild I tried the v4l2-request-test.

And both MPEG2 and H.264 baseline decoding work fine.

Just for reference, with nanopi neo (H3) and 864*480 slices, H.264 I frame is decoded in 6ms and B frames are decoded in less than 2ms.

 

The only issue I found is about video_buffers data arrays utilization.

With 16 frames (the default) it seems that only the first frame of 16 is correctly stored in arrays. It seems that arrays with index>0 contain either dirty or no data at all.  Need further investigations.

But setting frames count to 1 in the preset file, which is currently not a limit because operations are serialized, output data are always stored in buffer [0] and I then can save them to disk file. And the Bunny is displayed in the decoded frames. :-)

 

To make the test the quickest as possible I've sent to file only the Y channel. UV channels are available but I've not not currently checked them.

 

 

 

 

[drice]

This is turning out to be much more complicated than I expected. My goal is to be able to play an MPEG2 video with hardware decoding in a window using only mainline kernel drivers and open source usermode drivers. Using MPEG2 because that's what's supported in kernel 5.2 and the MPEG2 support seems to be more mature than H264. I think the problem is that the default Ubuntu packages for vlc, mpv, and ffmpeg aren't compiled with the necessary options to support V4L2 Request API, stateless codec, etc. I'm learning how to rebuild these applications with options that might be compatible with the acceleration components. Another potential issue is that I had to do a workaround to get libva-v4l2-request to compile: the autoconf.sh script for this library checks the package version of libva instead of the library version which causes it to think the installed version (libva2 2.x) is too old when it's actually much newer than the minimum requirement of 1.4.x, so I manually bypassed that check. I don't think it's the cause but if nothing else works I'll compile an older version of libva and see if it works any better.

 

One problem with researching this is that most documentation doesn't explain which kernel version they're using, what usermode driver they're using, etc. If I get this working I'll write up detailed documentation that includes all this information. But I have to get it working first!

[jernej]

15 minutes ago, drice said:

I think the problem is that the default Ubuntu packages for vlc, mpv, and ffmpeg aren't compiled with the necessary options to support V4L2 Request API, stateless codec, etc.

V4L2 request API or more precisely, codec interface is not stable yet, so no program officially supports V4L2 request API yet. Currently easiest way is to use libva-v4l2-request library (if you manage to build it).

[drice]

9 minutes ago, jernej said:

V4L2 request API or more precisely, codec interface is not stable yet, so no program officially supports V4L2 request API yet. Currently easiest way is to use libva-v4l2-request library (if you manage to build it).

 

That's what I meant to write. I've built libva-v4l2-request but it seems like there is an option missing in those packages that doesn't allow it to work properly with the players.

[FGuerzoni]

I really have poor experience with libva-v4l2-request library.

Maybe you could find useful that Sunxi-Cedrus linux release-2019.03 is a patched version of 5.0.7 kernel release

 

About libva-v4l2-request library, I tried to compile it. And yes I too had to disable version check.

And I found that the version from github master was easier to compile (less issues) that the one suggested by the sunxi wiki, which was the release-2019.03

 

[drice]

I compiled libva-v4l2-request release-2019.03 on H3 with kernel 5.3.0-rc3 (sunxi-cedrus module enabled and loaded) and also added some debug logging to parts of libva-v4l2-request to see where it's failing. Here is what I found:

 

1. H264 capability is being reported from vainfo on kernel 5.3:

 

Spoiler

drice@orangepilite:~$ vainfo
libva info: VA-API version 1.1.0
libva info: va_getDriverName() returns -1
libva info: User requested driver 'v4l2_request'
libva info: Trying to open /usr/lib/arm-linux-gnueabihf/dri/v4l2_request_drv_video.so
libva info: Found init function __vaDriverInit_1_1
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.1 (libva 2.1.0)
vainfo: Driver version: v4l2-request
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            :    VAEntrypointVLD
      VAProfileMPEG2Main              :    VAEntrypointVLD
      VAProfileH264Main               :    VAEntrypointVLD
      VAProfileH264High               :    VAEntrypointVLD
      VAProfileH264ConstrainedBaseline:    VAEntrypointVLD
      VAProfileH264MultiviewHigh      :    VAEntrypointVLD
      VAProfileH264StereoHigh         :    VAEntrypointVLD
 

 

2. Using mpv, it seems like it might be trying to open the same device twice or calling some ioctl that it's not supposed to call twice. Notice how the first ioctl returns 0 but then it looks like it makes the same call again and returns -1:

 

Spoiler

drice@orangepilite:~$ mpv --vo=vaapi --hwdec=vaapi Videos/big_buck_bunny_720p_MPEG2_MP2_25fps_3600K.mpeg 
Playing: Videos/big_buck_bunny_720p_MPEG2_MP2_25fps_3600K.mpeg
 (+) Video --vid=1 (mpeg2video 1280x720 25.000fps)
 (+) Audio --aid=1 (mp2 2ch 44100Hz)
[vo/vaapi] OSD format not supported. Disabling OSD.
[vo/vaapi] vaQueryDisplayAttributes(): the requested function is not implemented
v4l2-request: in v4l2_set_format
v4l2-request: type=1, width=1280, height=720, pixelformat=842094158
v4l2-request: in v4l2_setup_format
v4l2-request: type is 1
v4l2-request: is_output is false
v4l2-request: is_mplane is false
v4l2-request: returning from v4l2_setup_format
v4l2-request: calling ioctl VIDIOC_S_FMT on fd 13
v4l2-request: ioctl returned 0
v4l2-request: returning from v4l2_set_format
v4l2-request: in v4l2_set_format
v4l2-request: type=1, width=1280, height=720, pixelformat=842094158
v4l2-request: in v4l2_setup_format
v4l2-request: type is 1
v4l2-request: is_output is false
v4l2-request: is_mplane is false
v4l2-request: returning from v4l2_setup_format
v4l2-request: calling ioctl VIDIOC_S_FMT on fd 13
v4l2-request: ioctl returned -1
v4l2-request: Unable to set format for type 1: Device or resource busy
v4l2-request: returning from v4l2_set_format
[ffmpeg] AVHWFramesContext: Failed to create surface: 1 (operation failed).
Failed to allocate hw frames.
AO: [pulse] 44100Hz stereo 2ch s16
Using conversion filter.
VO: [vaapi] 1280x720 nv12
v4l2-request: in v4l2_set_format
v4l2-request: type=1, width=1280, height=720, pixelformat=842094158
v4l2-request: in v4l2_setup_format
v4l2-request: type is 1
v4l2-request: is_output is false
v4l2-request: is_mplane is false
v4l2-request: returning from v4l2_setup_format
v4l2-request: calling ioctl VIDIOC_S_FMT on fd 13
v4l2-request: ioctl returned -1
v4l2-request: Unable to set format for type 1: Device or resource busy
v4l2-request: returning from v4l2_set_format
[vaapi] vaCreateSurfaces(): operation failed
Could not initialize video chain.
Video: no video
A: 00:00:01 / 00:01:14 (2%)

Exiting... (Quit)
 

 

3. Using vlc, it calls the ioctl twice, but once for an input and once for an output (see the is_output variable, which I added for logging purposes). It seems like it makes it further in the process, but then the errors seen at the end appear continuously until I stop the video (this is the same MPEG2 video as with mpv):

 

Spoiler

drice@orangepilite:~$ vlc
VLC media player 3.0.7.1 Vetinari (revision 3.0.7.1-0-gf3940db4af)
[018e1258] main libvlc: Running vlc with the default interface. Use 'cvlc' to use vlc without interface.
libEGL warning: DRI2: failed to authenticate
[01937b20] main playlist: playlist is empty
Gtk-Message: 23:18:37.545: GtkDialog mapped without a transient parent. This is discouraged.
[0197d720] main filter error: Failed to create video converter
libva info: VA-API version 1.1.0
libva info: va_getDriverName() returns -1
libva info: User requested driver 'v4l2_request'
libva info: Trying to open /usr/lib/arm-linux-gnueabihf/dri/v4l2_request_drv_video.so
libva info: Found init function __vaDriverInit_1_1
libva info: va_openDriver() returns 0
v4l2-request: in v4l2_set_format
v4l2-request: type=1, width=1280, height=720, pixelformat=842094158
v4l2-request: in v4l2_setup_format
v4l2-request: type is 1
v4l2-request: is_output is false
v4l2-request: is_mplane is false
v4l2-request: returning from v4l2_setup_format
v4l2-request: calling ioctl VIDIOC_S_FMT on fd 42
v4l2-request: ioctl returned 0
v4l2-request: returning from v4l2_set_format
v4l2-request: calling v4l2_set_format() from RequestCreateContext
v4l2-request: in v4l2_set_format
v4l2-request: type=2, width=1280, height=720, pixelformat=1395803981
v4l2-request: in v4l2_setup_format
v4l2-request: type is 2
v4l2-request: is_output is true
v4l2-request: is_mplane is false
v4l2-request: returning from v4l2_setup_format
v4l2-request: calling ioctl VIDIOC_S_FMT on fd 42
v4l2-request: ioctl returned 0
v4l2-request: returning from v4l2_set_format
[9c508240] avcodec decoder: Using v4l2-request for hardware decoding
[9c508240] main decoder error: buffer deadlock prevented
[0198f630] blend blend error: no matching alpha blending routine (chroma: YUVA -> VAOP)
[0198f630] main blend error: blending YUVA to VAOP failed
[0198f630] blend blend error: no matching alpha blending routine (chroma: YUVA -> VAOP)
 

 

The same general pattern occurs when playing H264 with VLC, but the repeating errors are different.

 

I'll add more logging when/if I can, but in the meantime I'm posting this info to see if anyone understands it better than I do.

[ning]

it looks like userspace application support is not ready.

[drice]

I'm starting to think that the "Device or resource busy" error might be due to the fact that Xorg is using fbdev instead of the lima DRM driver. I'm going to try to get the lima DRM driver compiled but that seems like it will be a project in itself. My first attempt failed due to an internal compiler error, but I think that was caused by the CPU overheating.

[drice]

I finally managed to compile the Lima driver on the Orange Pi Lite by shutting down X and setting Ninja to use only 2 jobs at a time. Earlier problems were caused by out-of-memory conditions while compiling certain C++ files in Mesa. Unfortunately, all it seems to have done is broken the software OpenGL rendering that used to work:

drice@orangepilite:~$ glxinfo
name of display: :0.0
libGL error: unable to load driver: swrast_dri.so
libGL error: failed to load driver: swrast
X Error of failed request:  GLXBadContext
  Major opcode of failed request:  154 (GLX)
  Minor opcode of failed request:  6 (X_GLXIsDirect)
  Serial number of failed request:  50
  Current serial number in output stream:  49

More research is obviously required.

[drice]

The problem in the previous post (unable to load driver: swrast_dri.so) was because compiling and installing the lima userspace driver somehow changed the DRI library search path from /usr/lib to /usr/local/lib. But after resolving that, I still was unable to figure out how to make Xorg use lima_dri.so instead of swrast_dri.so. I'm setting this project aside until I find new information.

[jernej]

2 hours ago, drice said:

But after resolving that, I still was unable to figure out how to make Xorg use lima_dri.so instead of swrast_dri.so. I'm setting this project aside until I find new information.

Unless you are using this PR, glamor on X11 will not work with lima. Even with this PR, according to one dev on #lima IRC, there are still troubles with slowness. In short, lima is not good enough yet for running X11.

[Dreyk]

Hello all. I spent several days trying to launch Cedrus on Armbian with the Mainline kernel. But i did not achieve any success: vainfo shows that "everything is fine," but the VLC and others are throwing errors. I tried with armbian-dev kernel 5.3.0 (recompiled with Cedar enabled) and with BOOTLIN kernel 5.0 - the result is the same.

 

Why is the Cedar and v4l2_request so easy to use in the LibreELEC project with Mainline kernel ( https://forum.libreelec.tv/thread/17565-nightly-images-for-a64-h3-and-h6-boards/ )? How is it possible to port? Or is it all because of Kodi?

 

If manage to get hardware acceleration of video and 3D to work in Armbian on H3, then some Orange Pi boards will become an excellent mini-desktops. But for some reason no one is doing this, although all the components would seem to be ready... I wish I had enough experience and knowledge to build it all together.

[jernej]

57 minutes ago, Dreyk said:

Why is the Cedar and v4l2_request so easy to use in the LibreELEC project with Mainline kernel ( https://forum.libreelec.tv/thread/17565-nightly-images-for-a64-h3-and-h6-boards/ )? How is it possible to port? Or is it all because of Kodi?

It was not easy at all to come to this point. A lot of new code was written for ffmpeg, cedrus and Kodi to make it work. But since LibreELEC is closed ecosystem, we can afford to make some hacks which would otherwise cause issues with other programs.

 

Anyway, I plan to rework some Cedrus patches these days to remove at least one hack and after that I can try to help you with using that special version of ffmpeg. Reportedly it works fine with unmodified mpv but I didn't test that yet. However, be prepared to pick a lot of patches from latest linux git master. Unfortunately, this is moving target and modified ffmpeg will work with only specific version of Cedrus driver.

 

Once you are able to match ffmpeg and Cedrus driver it's best not to change anything until you have good reason to do so, like extending driver with new features.

[drice]

1 hour ago, Dreyk said:

But for some reason no one is doing this, although all the components would seem to be ready...

 

I thought the same thing until I actually tried to do it. It's much more complicated than I thought, but at least I learned a lot by reading kernel code and documentation so it wasn't a total loss. Hopefully this feature can be introduced eventually as it will be very nice when it works.

[ning]

graphics acceleration i different to video acceleration, opensource grapthics driver is lima, while upstream video driver is cedrus/libva-request.

lima kernel driver is ready and merged to 5.2, and lima userspace driver (mesa) is not ready.

 

what's the status of mesa support? a KEY patch for lima desktop support is submitted for review,, if this patch gets merged, we can rebuild have a try.

 

or you can keep tracking my binary release for mesa: https://github.com/zhangn1985/lima_gpu_drv

[drice]

I think many people don't know the difference between video acceleration and graphics acceleration until they start researching this topic. When I read questions about any type of acceleration for single-board computers I always assume the question is about hardware video decoding unless it specifically mentions 3D or gaming.

[Dreyk]

21 hours ago, jernej said:

Anyway, I plan to rework some Cedrus patches these days to remove at least one hack and after that I can try to help you with using that special version of ffmpeg. Reportedly it works fine with unmodified mpv but I didn't test that yet. However, be prepared to pick a lot of patches from latest linux git master. Unfortunately, this is moving target and modified ffmpeg will work with only specific version of Cedrus driver.

I am convinced that so many people will be very grateful to you if you make a successful guide on how to run Cedrus + ffmpeg + MPV + Kodi on an ordinary distribution with the Mainline kernel like Armbian, even if it requires some hacks.

[Alerino Reis]

Hi there, RetrOrangePi developer here. You will definitely get the perfect solution from Jernej, but this is just to keep your hopes high, achieved by experimenting with several sources so i will need some time to get documentation and tutorials straight

Spoiler

Spoiler

 

 

Edited August 22, 2019 by Alerino Reis
hiding images for clean thread view

[Dreyk]

48 minutes ago, Alerino Reis said:

 You will definitely get the perfect solution from Jernej, but this is just to keep your hopes high, achieved by experimenting with several sources so i will need some time to get documentation and tutorials straight

Good news! Did you manage to do this only for Kodi or also for MPV? In any case, I will be glad to your tutorial.