[Mainline VPU]

Been a few months since I've worked on trying to get VPU support on mainline but I was able to get pretty close working with xmixahlx on the Pine64 IRC. He made a forum post specifying all the resources we used: https://forum.pine64.org/showthread.php?tid=9171. I'll give some insight into my process:

I started with the Manjaro ARM kernel (5.5 which was very close to vanilla mainline) and applied patches created by diffing it against specific parts of
- https://github.com/bbrezillon/linux/tree/rk/vdec-h264-v9%2Bhantro-h264
- https://github.com/Kwiboo/linux-rockchip/tree/v4l2-5.5-from-list-v5.5-rkvdec
- https://github.com/LibreELEC/LibreELEC.tv/tree/master/projects/Rockchip/patches/linux/default

such that I would be able to compile
- https://github.com/pH5/libva/tree/hantro-h264
- https://github.com/pH5/libva-v4l2-request
- https://github.com/Kwiboo/FFmpeg/tree/v4l2-request-hwaccel-4.2.2

without any errors.

With this done, I began testing. MPEG2 was working with Hantro however H264 didn't have a codec for profile 100 (High) which is what most content is encoded in.

mpv -hwdec bbb_sunflower_native_60fps_normal.mp4
>>> [ffmpeg/video] h264: No support for codec h264 profile 100.

xmixahlx Has kept the thread I mentioned up to date on any new progress he finds so I suggest checking it out.

[[RK3399] Which VPU driver is used?]

I'm trying to get VPU acceleration on Manjaro ARM and I'm having trouble figuring out how Armbian is able to leverage the VPU with Hantro still incomplete. I know it uses RKMPP but I don't know which driver or patches I need to enable it in the Kernel and DTS. Any help is appreciated, thanks in advance.

[hal_h264d_api: Assertion failed at hal_h264d_init ... hard mode error]

Found the an MPP github issue pertaining to my exact problem. https://github.com/rockchip-linux/mpp/issues/52

[Does anyone have VPU working with RK3399 ?]

On 11/27/2018 at 12:13 PM, jerryn said:

The ioctl segfault issue was the arhitecture flags for the compile we're set to unknown. I set it manually to armv7

Did you set it to armv7 in CMakeLists? I tried doing the same (since the rk3399 reports as aarch64 which doesn't match any of the compare strings) and it wasn't able to compile.

[hal_h264d_api: Assertion failed at hal_h264d_init ... hard mode error]

The assertion error happens on line 154 in this file:

https://github.com/rockchip-linux/mpp/blob/67e80ebfe46558c2eb50218e57b50d51d6df2be2/mpp/hal/rkdec/h264d/hal_h264d_api.c

 

MPP_RET hal_h264d_init(void *hal, MppHalCfg *cfg)
{
    MppHalApi *p_api = NULL;
    MPP_RET ret = MPP_ERR_UNKNOW;
    H264dHalCtx_t *p_hal = (H264dHalCtx_t *)hal;
    VpuHardMode hard_mode = MODE_NULL;
    RK_U32 hard_platform = 0;

    INP_CHECK(ret, NULL == p_hal);
    memset(p_hal, 0, sizeof(H264dHalCtx_t));

    p_api = &p_hal->hal_api;

    p_hal->frame_slots  = cfg->frame_slots;
    p_hal->packet_slots = cfg->packet_slots;
    p_hal->fast_mode = cfg->fast_mode;
    //!< choose hard mode
    {
        RK_U32 mode = 0;
        RK_U32 vcodec_type = 0;
        mpp_env_get_u32("use_mpp_mode", &mode, 0);
        vcodec_type = mpp_get_vcodec_type();
        mpp_assert(vcodec_type & (HAVE_RKVDEC | HAVE_VPU1 | HAVE_VPU2));
        if ((mode <= RKVDEC_MODE) && (vcodec_type & HAVE_RKVDEC)) {
            hard_mode = RKVDEC_MODE;
            hard_platform = HAVE_RKVDEC;
        } else if (vcodec_type & HAVE_VPU1) {
            hard_mode = VDPU1_MODE;
            hard_platform = HAVE_VPU1;
        } else if (vcodec_type & HAVE_VPU2) {
            hard_mode = VDPU2_MODE;
            hard_platform = HAVE_VPU2;
        }
        H264D_DBG(H264D_DBG_HARD_MODE, "set_mode=%d, hw_spt=%08x, use_mode=%d\n",
                  mode, vcodec_type, hard_mode);
    }
    switch (hard_mode) {
    case RKVDEC_MODE:
        p_api->init    = rkv_h264d_init;
        p_api->deinit  = rkv_h264d_deinit;
        p_api->reg_gen = rkv_h264d_gen_regs;
        p_api->start   = rkv_h264d_start;
        p_api->wait    = rkv_h264d_wait;
        p_api->reset   = rkv_h264d_reset;
        p_api->flush   = rkv_h264d_flush;
        p_api->control = rkv_h264d_control;
        cfg->device_id = HAL_RKVDEC;
        break;
    case VDPU1_MODE:
        p_api->init    = vdpu1_h264d_init;
        p_api->deinit  = vdpu1_h264d_deinit;
        p_api->reg_gen = vdpu1_h264d_gen_regs;
        p_api->start   = vdpu1_h264d_start;
        p_api->wait    = vdpu1_h264d_wait;
        p_api->reset   = vdpu1_h264d_reset;
        p_api->flush   = vdpu1_h264d_flush;
        p_api->control = vdpu1_h264d_control;
        cfg->device_id = HAL_VDPU;
        break;
    case VDPU2_MODE:
        p_api->init    = vdpu2_h264d_init;
        p_api->deinit  = vdpu2_h264d_deinit;
        p_api->reg_gen = vdpu2_h264d_gen_regs;
        p_api->start   = vdpu2_h264d_start;
        p_api->wait    = vdpu2_h264d_wait;
        p_api->reset   = vdpu2_h264d_reset;
        p_api->flush   = vdpu2_h264d_flush;
        p_api->control = vdpu2_h264d_control;
        cfg->device_id = HAL_VDPU;
        break;
    default:
        mpp_err_f("hard mode error, value=%d\n", hard_mode);
        mpp_assert(0);
        break;
    }

The value of  `hard_mode` is 0 when it fails. The error goes on to say "failed to find device for coding 7 type 0".

[hal_h264d_api: Assertion failed at hal_h264d_init ... hard mode error]

I'm trying to make a script to run on the Pinebook Pro and I'm having mpp errors while trying to leverage it in ffmpeg. I'm not sure what might be causing the error but according to THIS THREAD , compiling with `-DHAVE_DRM='ON` should resolve it (which it doesn't in my case). I'm using the 20171218 release of rkmpp and ffmpeg 4.2. Any help is appreciated.

mmp build:

cmake -DHAVE_DRM='ON' -DCMAKE_BUILD_TYPE:STRING='Release' -DCMAKE_INSTALL_PREFIX:PATH='/usr' \ -DRKPLATFORM:BOOL='ON' -Wno-dev \ -DAVSD_TEST:BOOL='OFF' -DH264D_TEST:BOOL='OFF' -DH265D_TEST:BOOL='OFF' -DJPEGD_TEST:BOOL='OFF' \ -DMPI_DEC_TEST:BOOL='OFF' -DMPI_ENC_TEST:BOOL='OFF' -DMPI_RC2_TEST:BOOL='OFF' \ -DMPI_RC_TEST:BOOL='OFF' -DMPI_TEST:BOOL='OFF' -DMPP_BUFFER_TEST:BOOL='OFF' \ -DMPP_ENV_TEST:BOOL='OFF' -DMPP_INFO_TEST:BOOL='OFF' -DMPP_LOG_TEST:BOOL='OFF' \ -DMPP_MEM_TEST:BOOL='OFF' -DMPP_PACKET_TEST:BOOL='OFF' -DMPP_PLATFORM_TEST:BOOL='OFF' \ -DMPP_TASK_TEST:BOOL='OFF' -DMPP_THREAD_TEST:BOOL='OFF' -DVP9D_TEST:BOOL='OFF' \ -DVPU_API_TEST:BOOL='OFF' .. 
make -j$THREADS
sudo make install

 

ffmpeg build:
 

./configure --prefix=/usr --enable-gpl --enable-version3 --enable-nonfree --enable-static --enable-gmp \
        --enable-gnutls --enable-libass --enable-libbluray --enable-libcdio --enable-libfdk-aac \
        --enable-libfreetype --enable-libmp3lame --enable-libtheora --enable-libvorbis --enable-libx264 \
        --enable-libxcb --enable-opencl --enable-libdrm --arch='armv8-a+crc+crypto' \
        --cpu='cortex-a72.cortex-a53' --enable-rkmpp --enable-lto --enable-hardcoded-tables \
        --disable-debug
make -j$THREADS
sudo make install
sudo ldconfig

 

error:

sudo ffmpeg -benchmark -vcodec h264_rkmpp -i demo.mkv -f null -
ffmpeg version 4.2 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 9.1.0 (GCC)
  configuration: --prefix=/usr --enable-gpl --enable-version3 --enable-nonfree --enable-static --enable-gmp --enable-gnutls --enable-libass --enable-libbluray --enable-libcdio --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libtheora --enable-libvorbis --enable-libx264 --enable-libxcb --enable-opencl --enable-libdrm --arch=armv8-a+crc+crypto --cpu=cortex-a72.cortex-a53 --enable-rkmpp --enable-lto --enable-hardcoded-tables --disable-debug 
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
Input #0, matroska,webm, from 'demo.mkv':
  Metadata:
    COMPATIBLE_BRANDS: iso6avc1mp41
    MAJOR_BRAND     : dash
    MINOR_VERSION   : 0
    ENCODER         : Lavf58.29.100
  Duration: 00:05:13.80, start: -0.007000, bitrate: 2887 kb/s
    Stream #0:0: Video: h264 (High), yuv420p(tv, bt709, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 29.97 fps, 29.97 tbr, 1k tbn, 59.94 tbc (default)
    Metadata:
      HANDLER_NAME    : ISO Media file produced by Google Inc.
      DURATION        : 00:05:13.780000000
    Stream #0:1(eng): Audio: opus, 48000 Hz, stereo, fltp (default)
    Metadata:
      DURATION        : 00:05:13.801000000
mpi: mpp version: 
hal_h264d_api: Assertion vcodec_type & ((0x00000200) | (0x00000001) | (0x00000002)) failed at hal_h264d_init:119
hal_h264d_api: hal_h264d_init hard mode error, value=0
hal_h264d_api: Assertion 0 failed at hal_h264d_init:169
mpp_device: mpp_device_init failed to find device for coding 0 type 7
hal_h264d_api: p_hal->vpu_socket <= 0
mpp_hal: mpp_hal_init hal h264d_rkdec init failed ret -2
mpp_hal: mpp_hal_init could not found coding type 7
mpp_dec: mpp_dec_init could not init hal
mpp_rt: NOT found ion allocator
mpp_rt: found drm allocator
mpp: error found on mpp initialization
mpp: WARNING: setup buffer group before decoder init
mpp: command 310002 param 0xaaab0831a360 ret -1
[h264_rkmpp @ 0xaaab0833b720] Failed to assign buffer group (code = -1)
[h264_rkmpp @ 0xaaab0833b720] Failed to initialize RKMPP decoder.
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (h264_rkmpp) -> wrapped_avframe (native))
  Stream #0:1 -> #0:1 (opus (native) -> pcm_s16le (native))
Error while opening decoder for input stream #0:0 : Unknown error occurred
bench: maxrss=23880kB

 

[Compil ffmpeg for RK3328 (v88 mini 3)]

On 6/24/2019 at 12:55 PM, martos said:

Ah !

The rockchip mpp ( https://github.com/rockchip-linux/mpp )

and ffmpeg ( hardware data path for libswscale )

have make so change to encode in hardware , so i need to make some new test  ...

Come back wit result soon ...

Did you manage to figure out how to properly utilize hardware acceleration using ffmpeg + RockChip MPP? I'm working on a similar project that would benefit greatly from this and an update on your work would be very appreciated.