Jump to content
  • 0

The VPU driver


Myy
 Share

Question

Alright, the biggest issue with the ASUS Tinkerboard being "resolved", I'm starting to take care of the VPU driver, in order to use it with mainline kernels.

 

The hardware video encoding/decoding facilities are the real meat of the RK3288, and recent Rockchip boards.

The VPU driver aims to use these facilities in order to provide the smooth video playback experience that every Netflix/Hulu addict expects when buying such boards.

 

Which mean, once this part dealt with, you might be able to enjoy your Multimedia oriented distributions/softwares on your Rockchip boards (MiQi, Tinkerboard and maybe Pine64 too !)

 

So the driver is already available in Rockchip 4.4 kernels, and started to be ported in an Out-Of-Tree fashion by phhusson on Github, and then got the attention of wzyy2, one of the Rockchip guy, which updated it and took care of multiple bugs that were present in this version.

 

I'm now reassembling the code, include files and all in my rockchip-vcodec repository, patching the code to use it with the latest 4.13 kernels.

 

Now, the problems are : I don't get the big picture from the user side. So it's kinda hard to test this quickly.

 

Rockchip seems to rely on their library, MPP, and patched gstreamer packages if I understand correctly. Last time I tried the "test-suite" of the MPP library, things were "clunky". Since it's mostly maintained by one person, ayaka, I'm pretty sure that it's the kind of test-suite that only the owner clearly understands. I know that problem, as I did the same thing with some of my libraries.

 

 

So, basically, the main goals are :

  • ! Compile the VPU driver without issues. I took care of that for now. The driver compiles, loads AND unloads correctly, without issues.
  • ? Understand clearly what packages, patches and software are needed to use the VPU services provided by the VPU driver.
  • ? Enjoy 1080p movies without a hitch on RK3288 boards (and others Rockchip boards if possible) !

 

I'll add these goals on the Github pages so that everyone gets how things are going, from the GH page.

 

If you have any issue with the driver and/or it's use, file it on my repository and I'll see what I can do.

Link to comment
Share on other sites

Recommended Posts

When discussing a problem make sure to provide full logs!

  • 0

OK, got it compiled into 4.13.3:  http://sprunge.us/aBaA

 

Both HEVC-service and VPU-service available in /dev/

 

[Edit]  I forgot they were using Stretch, tried to install the Rockchip Mali driver and nuked my xserver.  Tried to install their xserver but Xenial does not support a couple libraries they now need.  Building a Stretch image to give it a try.  I'll have to try this on 4.14 next as I'd rather not stuff (this level of) experimental code into Next.

Link to comment
Share on other sites

  • 0

I've ported the new code provided by Ayaka but I still need to test it. Since I'm using a different MiQi, I have to recompile mpp (done) and mpv.

 

The last one is posing several issues since it wants a pretty recent version of FFMPEG. I'll see whether I compile a new local version of ffmpeg of if I upgrade from Ubuntu Xenial to Ubuntu Artful to resolve this issue.

Link to comment
Share on other sites

  • 0

Hi,

 

Please can you tell me the  current state of this work?

 

I wish to us a Tinker board, for Shinobi DVR  (cctv), and Rstp stream cameras.  The H264 streams are handled by ffmeg in the Shinobi  application.

 

many thanks

 

regards

 

Patrick

Link to comment
Share on other sites

  • 0

So, I'm still trying to get this "thing" working on mainline kernels and my last issue is that the setup is done, I launch a decode pass, the IRQ handler is triggered but the output buffer remains untouched.

However, the thing I don't have any error messages. I see that some VPU registers changed after the (failed) decode pass, but most of the changed registers are completely undocumented.

 

And by "undocumented", I mean, even RKMPP and the Chromium V4L2 do not document these registers.

 

Meanwhile, I saw that the old drivers use the IOMMU DMA API, so I tried to enable the IOMMU_DMA API on Rockchip systems but that led to the Video Output MMU breaking, along with the DRM drivers firing BUG_ON after BUG_ON.

I was still able to boot the system by unplugging the screen, but the VPU driver issue remained identical.

Loads, runs, trigger the IRQ, no output.

 

I've even setup an IOMMU Fault handler, just in case, but it's never called. I'm currently trying to look at how to setup a DMA fault handler, maybe that's the issue here.

Meanwhile, I tried to disable an old patch I kept around in my kernels and... that led to the VPU MMU clocks not starting anymore... Yay...

 

As a last resort, I'm asking for various hints on the Rockchip Linux and IOMMU Linux Mailing lists. Still... I don't expect much.

 

So I'm close but still not there.

Maybe it's a simple bug in the code that I'm not seeing.

Link to comment
Share on other sites

  • 0
On 8/18/2018 at 9:12 AM, Myy said:

So, I'm still trying to get this "thing" working on mainline kernels and my last issue is that the setup is done, I launch a decode pass, the IRQ handler is triggered but the output buffer remains untouched.

However, the thing I don't have any error messages. I see that some VPU registers changed after the (failed) decode pass, but most of the changed registers are completely undocumented.

 

 

 

Did they send you any hint?

Link to comment
Share on other sites

  • 0

No. However, they sent some patches to the LKML with an implementation that seems to work with mainline kernels.
So, I'm currently trying to get a kernel 5.0 working in order to test the VPU on it, with these drivers, and see how it goes.

Link to comment
Share on other sites

  • 0
15 hours ago, Myy said:

No. However, they sent some patches to the LKML with an implementation that seems to work with mainline kernels.
So, I'm currently trying to get a kernel 5.0 working in order to test the VPU on it, with these drivers, and see how it goes.

Sounds good!

Does it mean that there is more people pushing for VPU support on the tinkerboard? Or they made a generic vpu implementation?
Don't forget to post any progress :pp

Link to comment
Share on other sites

  • 0

@Myy,

Any chance you will be working on RK3399 5.0.rcX ?  (GPU / VPU). I have been looking at your work for NanoPi-T4 but the label ** NOT TESTED ** was a bit discouraging.

I can try to help testing or to provide some feedback.

Link to comment
Share on other sites

  • 0
8 hours ago, Myy said:

Well, I've been able to get a 5.0-rc2 working now. So I'll try the VPU patches this week and see if I can get decode anything with them.

 

Other people seems to be eager too, it seems.

Wow! i hate not being able to understand anything of those patches, looks like tons of work.

I bet you rockchip will copy your changes and publish as theirs some time.

 

So someone also tried porting google's vpu work on rk3288? is that compatible with debian?

 

Link to comment
Share on other sites

  • 0

I will give the RK3399 a try. Note that ayaka patches on the mailing list provided DTS bindings for RK3399, so those might be what you're looking for.

https://lkml.org/lkml/2019/1/5/124

 

About Chromebook specific kernels, you might be able to adapt the patches to mainline kernels and get them working. But the patched kernels are generally frozen way beyond mainline, so it might be better to just use Rockchip 4.4 kernel at that point.

 

Now, the patch doesn't provide DTS bindings for RK3288 so I'll have to work on this, this week.

Link to comment
Share on other sites

  • 0

I tried to integrate the driver and I was able to compile it, after a few modifications, however while I can load it manually, it seems to fail (silently) to initialize the hardware, so I guess I need to review the DTS nodes (and the code).

 

However, the driver still need modifications.

1. #include <soc/rockchip/pm_domains.h> does not exist in mainline kernels.

2. rockchip_pmu_idle_request does not exist in mainline kernels. However, I already have an old patch that adds it. That said, I'm not sure the added code is valid.

Using my patch, you need to add #include <linux/rockchip_pmu.h> to each mpp_dev_ file... I guess I'll rewrite my patch to generate <soc/rockchip/pm_domains.h> instead and avoid additional modifications.

 

So, yeah, it compiles, it loads, it does nothing. I'll have a look at the last part.

Link to comment
Share on other sites

  • 0

Okay, the driver is loading now... So now, I have to pull out a setup that can use "mpp service"... @JMCC Is your setup useable with MPP service ?

 

I'll try to rewrite the patches and upload them, along with a complete build, in order to let people play with it, see if they can get something out of it.

 

There's still a *BIG* fat warning [    6.565010] Failed to attached device ff9a0000.video-codec to IOMMU_mapping

Now... The nodes have been split in two and ff9a0000.video-codec is the Video Encoder. The Video decoder is at ff9a0400.video-codec so... maybe it's working fine ?

It's still highly probable that it will just crash in a very bad way.

 

Anyway, I'll need some testers that are used to the whole MPP stuff.

Link to comment
Share on other sites

  • 0
  • 0

And here's a build : https://miouyouyou.fr/RockMyy/builds/RockMyy-5.0.0-rc2-HighFive-VPU-Beta.tar.xz

 

Note that, with these builds, local SSH drop continuously so... if you don't have a keyboard/mouse plugged to the Tinkerboard, you might want to pass on it.

Also, you have to understand how to install an unofficial kernel.

Just wait if you don't.

Link to comment
Share on other sites

  • 0
11 hours ago, Myy said:

And here's a build : https://miouyouyou.fr/RockMyy/builds/RockMyy-5.0.0-rc2-HighFive-VPU-Beta.tar.xz

 

Note that, with these builds, local SSH drop continuously so... if you don't have a keyboard/mouse plugged to the Tinkerboard, you might want to pass on it.

Also, you have to understand how to install an unofficial kernel.

Just wait if you don't.

So much progress in a single day!

 

I'm downloading the build and i will try to use it, got mouse+keyboard so gonna be fine i hope :)

 

Im not sure about how to install an unofficial kernel but i will try by unzipping the build on root a let it replace everything it asks.

Edit: i tried it and i guess failed as it does not longer boot.

I replaced /boot/, /lib/ & /usr/ folder from one partition and i also applied the same files from /boot/ folder to the boot partition.

When i give it power, does nothing, black screen.

 

I noticed /boot/ folder replaces all files except "rk3288-miniarm.dtb" which is the one set in extlinux.conf. Switching from that file to rk3288-tinker.dtb throws a Kernel Panic.

 

Maybe i'm not using the right base image (TinkerOS 2.0.8) i will try over an armbian one.

Edit2: Now i tried over Armbian 5.70 - Ubuntu bionic 4.4.166 and i get the black screen too.

Edit3: Now i tried over Armbian 5.70 - Ubuntu bionic 4.19 and.... IT BOOTS!

 

The problem wih 4.19 is that Media Script only supports 4.4 so i can't install it :(

Link to comment
Share on other sites

  • 0

If you get a kernel panic, just try to get a photo or a retranscription of what's happening.

 

I'd highly recommend you to get an USB<->RS232 adapter that you can plug on your computer and the Tinkerboard, in order to get a serial output when kernel crashes. That way, you can launch "picocom" on your computer and do something like :

picocom -b 115200 /dev/ttyUSB0

 

And then copy-paste the big error message that appear on the serial console.

 

To quit picocom, you'll need to do CTRL+A then CTRL+Q

Link to comment
Share on other sites

  • 0
15 hours ago, petatester said:

Now i tried over Armbian 5.70 - Ubuntu bionic 4.4.166 and i get the black screen too.

Edit3: Now i tried over Armbian 5.70 - Ubuntu bionic 4.19 and.... IT BOOTS!

 

The 4.4-4.18 images are looking for "miniarm" because we had working images before the mainline device tree was useful.  4.19+ inages the u-boot is looking for tinker dtb, but also has a miniarm DTB to maintain compatibility for the upgrade path.  TL/DR: So if you had no miniarm DTB the boot would fail.  

 

Ah, also, the old boot scripts (pre 4.19) hate the tinker dtb because of overlays.  I turned a bunch of gpio things off so the logic was sane for the overlays.

Link to comment
Share on other sites

  • 0

OK, I have the driver compiled into the dev kernel, enabled, loads on boot:

 

root@tinkerboard:~# lsmod
Module                  Size  Used by
overlay                90112  1
fuse                   98304  3
zstd                   16384  4
gpio_keys              20480  0
zram                   24576  5
rockchip_vpu           32768  0
rockchip_rga           20480  0
rk_crypto              24576  0
videobuf2_dma_sg       20480  1 rockchip_rga
videobuf2_dma_contig    20480  1 rockchip_vpu
videobuf2_vmalloc      16384  1 rockchip_vpu
v4l2_mem2mem           20480  2 rockchip_vpu,rockchip_rga
videobuf2_memops       16384  3 videobuf2_dma_sg,videobuf2_dma_contig,videobuf2_vmalloc
snd_soc_hdmi_codec     16384  1
videobuf2_v4l2         24576  3 rockchip_vpu,v4l2_mem2mem,rockchip_rga
videobuf2_common       40960  4 rockchip_vpu,v4l2_mem2mem,videobuf2_v4l2,rockchip_rga
syscon_reboot_mode     16384  0
dw_wdt                 16384  0
reboot_mode            16384  1 syscon_reboot_mode
r8723bs               561152  0
rockchip_thermal       24576  0
cpufreq_dt             16384  0
thermal_sys            57344  2 rockchip_thermal,cpufreq_dt
sch_fq_codel           20480  6
ip_tables              24576  0
mali_kbase            335872  2
dw_hdmi_i2s_audio      16384  0
realtek                16384  1

@JMCC if I'm not mistaken this isn't going to work with the rockchip software, as it's looking for the vpu_service and hevc_service, correct?  being a V4L2 setup probably changes the dynamic a bit...

Link to comment
Share on other sites

  • 0
19 minutes ago, TonyMac32 said:

being a V4L2 setup probably changes the dynamic a bit...

Yes, it certainly does. I don't know if the media script will work with this or not, depends whether the defaults of the programs I compiled were to enable or disable v4l2-m2m (I think at least ffmpeg in mpv has it enabled). I won't have access to my board in some days, so I cannot test now.

Link to comment
Share on other sites

  • 0

Ah, the rockchip_vpu driver is the driver written by Ezequiel Garcia, which only supports JPEG encoding at the moment.

 

The driver is named rockchip_mpp. You might want to disable the Rockchip VPU in the staging drivers, and enable Rockchip MPP which is the option at the bottom of the staging section.

If you can't see it, maybe the integration within the Kconfig or Makefile isn't done correctly (´・ω・` ).

Link to comment
Share on other sites

  • 0

@Myy I missed your comment on the initial commit, I need to quite waiting until midnight to do the more interesting stuff.  :unsure:

 

I'm being presented with 5 options under the rockchip_mpp kconfig:  I've got MPP_VDEC_DEVICE, vpu decovder V1 and V2, vpu encoder V1 and V2. Are they all needed?

Link to comment
Share on other sites

  • 0

Done, and loaded.  Now to see what the fallout is.  ;-)

 

tony@tinkerboard:~/Desktop$ mpv --hwdec=auto *.mp4
Playing: Avengers_ Infinity War.mp4
[ffmpeg/demuxer] mov,mp4,m4a,3gp,3g2,mj2: stream 0, timescale not set
 (+) Video --vid=1 (*) (h264 1280x720 29.970fps)
     Video --vid=2 [P] (png)
 (+) Audio --aid=1 (*) (aac 2ch 44100Hz)
File tags:
 Comment: Invader. Annihilator. So-called savior. As Thanos moves ever closer to omnipotence, the fate of the universe rests with the Avengers.
 Date: 2018
 Title: Avengers: Infinity War
libGL error: unable to load driver: rockchip_dri.so
libGL error: driver pointer missing
libGL error: failed to load driver: rockchip
libGL error: unable to load driver: rockchip_dri.so
libGL error: driver pointer missing
libGL error: failed to load driver: rockchip
[vo/gpu/opengl] Suspected software renderer or indirect context.
Failed to open VDPAU backend libvdpau_rockchip.so: cannot open shared object file: No such file or directory
mpi: mpp version: 598cae3 author: Jacob Chen DEBIAN: update rules for release_20171218-2
hal_h264d_api: hal_h264d_init mpp_buffer_group_get_internal used ion In
mpp_rt: NOT found ion allocator
mpp_rt: found drm allocator
mpp_drm: os_allocator_drm_alloc handle_to_fd failed ret -1
mpp_buffer: mpp_buffer_create failed to create buffer with size 4040
mpp_hal: mpp_hal_init hal h264d_rkdec init failed ret -1
mpp_hal: mpp_hal_init could not found coding type 7
mpp_dec: mpp_dec_init could not init hal
mpp: error found on mpp initialization
mpp: WARNING: setup buffer group before decoder init
mpp: command 310002 param 0x2b89ea0 ret -1
[ffmpeg/video] h264_rkmpp: Failed to assign buffer group (code = -1)
[ffmpeg/video] h264_rkmpp: Failed to initialize RKMPP decoder.
Could not open codec.
VO: [gpu] 1280x720 yuv420p
AO: [pulse] 44100Hz stereo 2ch float
AV: 00:00:56 / 02:29:49 (0%) A-V:  0.000 Dropped: 13

Audio/Video desynchronisation detected! Possible reasons include too slow
hardware, temporary CPU spikes, broken drivers, and broken files. Audio
position will not match to the video (see A-V status field).

@JMCC  I'll stop clogging your media scrip thread with development noise, we can put it all here.  This is after doing your recommended HWdecode.  It isn't v4l2, it is rkmpp, I had some wires crossed.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Answer this question...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
 Share

×
×
  • Create New...