Nanopi neo air ffmpeg with Cedrus H264

[a.rodionov]

Armbianmonitor:

https://drive.google.com/file/d/1ik7BWZdcjR7WsVOnyZB7M9mV0Qu0JQtY/view?usp=sharing

Hi all. 

I have a board nanopi neo air with armbian linux on it.   We with my colleague try to setting up ffmpeg with cedrus264 for hardware accelerated  capturing video from ov5640-like camera device. 

There is my  armbianmonitor.  

There are my v4l2-ctl -d /dev/videox -D out:

 

Spoiler

Driver Info:
    Driver name      : cedrus
    Card type        : cedrus
    Bus info         : platform:cedrus
    Driver version   : 5.4.51
    Capabilities     : 0x84208000
        Video Memory-to-Memory
        Streaming
        Extended Pix Format
        Device Capabilities
    Device Caps      : 0x04208000
        Video Memory-to-Memory
        Streaming
        Extended Pix Format
Media Driver Info:
    Driver name      : cedrus
    Model            : cedrus
    Serial           : 
    Bus info         : platform:cedrus
    Media version    : 5.4.51
    Hardware revision: 0x00000000 (0)
    Driver version   : 5.4.51
Interface Info:
    ID               : 0x0300000c
    Type             : V4L Video
Entity Info:
    ID               : 0x00000001 (1)
    Name             : cedrus-source
    Function         : V4L2 I/O
    Pad 0x01000002   : 0: Source
      Link 0x02000008: to remote pad 0x1000004 of entity 'cedrus-proc': Data, Enabled, Immutable
 

Spoiler

Driver Info:
    Driver name      : sun6i-video
    Card type        : sun6i-csi
    Bus info         : platform:camera
    Driver version   : 5.4.51
    Capabilities     : 0x84200001
        Video Capture
        Streaming
        Extended Pix Format
        Device Capabilities
    Device Caps      : 0x04200001
        Video Capture
        Streaming
        Extended Pix Format
Media Driver Info:
    Driver name      : sun6i-csi
    Model            : Allwinner Video Capture Device
    Serial           : 
    Bus info         : 
    Media version    : 5.4.51
    Hardware revision: 0x00000000 (0)
    Driver version   : 5.4.51
Interface Info:
    ID               : 0x03000003
    Type             : V4L Video
Entity Info:
    ID               : 0x00000001 (1)
    Name             : sun6i-csi
    Function         : V4L2 I/O
    Pad 0x01000002   : 0: Sink, Must Connect
      Link 0x02000007: from remote pad 0x1000006 of entity 'Borei_VideoDriver 0-003c': Data, Enabled, Immutable
 

After some reconfiguration of dts, we can successfully capture video in low fps with 

ffmpeg -f v4l2 -framerate 25 -video_size 960x720 -i /dev/video1 output.mkv

but with 

ffmpeg -f v4l2 -channel 0 -video_size 960x720 -i /dev/video1 -pix_fmt nv12 -r 30 -b:v 64k -c:v cedrus264 test.mp4

ffmpeg failed: 

Spoiler

root@nanopiair:/home/pi# ffmpeg -f v4l2 -channel 0 -video_size 960x720 -i /dev/video1 -pix_fmt nv12 -r 30 -b:v 64k -c:v cedrus264 test.mp4 > ffmpeg_error.txt
ffmpeg version git-2015-01-22-f86a076 Copyright (c) 2000-2014 the FFmpeg developers
  built on Jul 14 2020 09:02:41 with gcc 8 (Debian 8.3.0-6)
  configuration: --prefix=/usr --enable-nonfree --enable-gpl --enable-vdpau --enable-libx264 --enable-libv4l2
  libavutil      54.  6.100 / 54.  6.100
  libavcodec     56.  0.101 / 56.  0.101
  libavformat    56.  2.100 / 56.  2.100
  libavdevice    56.  0.100 / 56.  0.100
  libavfilter     5.  0.102 /  5.  0.102
  libswscale      3.  0.100 /  3.  0.100
  libswresample   1.  1.100 /  1.  1.100
  libpostproc    53.  0.100 / 53.  0.100
Input #0, video4linux2,v4l2, from '/dev/video1':
  Duration: N/A, start: 1581.872201, bitrate: 248832 kb/s
    Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 960x720, 248832 kb/s, 30 fps, 30 tbr, 1000k tbn, 1000k tbc
File 'test.mp4' already exists. Overwrite ? [y/N] y
[cedrus264 @ 0x241ff20] VE Open error.
Output #0, mp4, to 'test.mp4':
    Stream #0:0: Video: h264, q=2-31, 128 kb/s, 30 fps
    Metadata:
      encoder         : Lavc56.0.101 cedrus264
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (cedrus264))
Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
 

 

Ffmpeg was buided from FFmpeg-Cedrus, also we  try follow this and use this binaries - result haven't changed. 

 

After some inspecting sources of ffmpeg-cedrus we found that ffmpeg try to open /dev/cedar_dev device, which don't exists in our system.  Ffmpeg fails on opening device. If we replace /dev/cedar_dev by /dev/video0, where is our cedrus driver, ffmpeg fails on ioctl IOCTL_GET_ENV_INFO.  In source, it is 99 or 101 line.

We check kernel config for configuration, provided by Sunxi-Cedrus and determine that cedrus was configured right. 

 

Also we  have tried all described actions on official armbian buster image for nanopi neo air and we have got the same result. 

What we does wrong? 

 

[a.rodionov]

Full .dts, which is works for us, now is  placed in uboborov repository.

Edited July 24, 2020 by a.rodionov
i did accidentally pasted past answer

[@lex]

Encoding using cedrus264 was for legacy kernel.

I think Uboborov has done some work with mainline kernel for encoding with cedrus. Search the forum for "encoding" or uboborov

[a.rodionov]

18 hours ago, @lex said:

Encoding using cedrus264 was for legacy kernel.

I think Uboborov has done some work with mainline kernel for encoding with cedrus. Search the forum for "encoding" or uboborov

Thank you for you answer. What kernel version do you mean as legacy kernel? In armbian build tool legacy kernel is 4.19 kernel and it has no cedrus at all, including /dev/cedar_dev. We found topic from Uboborov(this), and his driver  (this) with hardware h264 encoder (this). We build fresh 4.19 kernel, Uboborov's driver and install it to our system. Driver has loaded successfully (determine by lsmod). We change dts, as it changed in his github, but still there isn't /dev/cedar_dev. Also, as i already said in my first post, we also try to use Uboborov's ffmpeg (this), including precompiled binaries (found there).  

[@lex]

Legacy = Kernel 3.x

Mainline (?) >= 4.x

 

I have not tested the @ubobrov solution, i think you should go with this one: https://github.com/uboborov/sunxi-cedar-mainline

 

[Werner]

1 hour ago, @lex said:

Legacy = Kernel 3.x

Mainline (?) >= 4.x

 

I have not tested the @ubobrov solution, i think you should go with this one: https://github.com/uboborov/sunxi-cedar-mainline

 

About a year ago this was correct. All 3.x kernels have been dropped due to lack of upstream support.

 

[a.rodionov]

Thanks to all for help.  We have setuped uboborov sunxi-cedar-mainline in 5.7 armbian kernel.  I think, early we did some errors in dts.  Our current dts we published in https://github.com/Rexarrior/sunxi-cedar-mainline.  Big thanks to uboborov! 

[gounthar]

So does the hardware acceleration work for you @a.rodionov ?

[a.rodionov]

18 hours ago, gounthar said:

So does the hardware acceleration work for you @a.rodionov ?

Yes, it is.

On nanopi neo air with armbian 20.08.0 (5.7.8) it works.

 

FFmpeg with 960x720 input from our camera* encode mp4 by h264_omx with 15-20 fps.  If i use cedrus264 as codec, ffmpeg can easy provide 30 fps. With default cam500b  i get 60 fps 640x480 (that is maximum of the camera).  Fps numbers i get from ffmpeg log and i think  this results is accurate enough. 

* our camera  isn't ov5640 default camera and 30 fps likely it's maximum fps. We haven't camera's datasheet. 

[matteog]

@a.rodionov we are in need of enabling h264 encoding on a Neo Air platform, can you please explain what you did in order to make it work?
Custom ffmpeg, kernel rebuild, or both of them?
You will be very helpful!!
Thanks

[a.rodionov]

02.05.2021 в 18:38, matteog сказал:

@a.rodionov we are in need of enabling h264 encoding on a Neo Air platform, can you please explain what you did in order to make it work?
Custom ffmpeg, kernel rebuild, or both of them?
You will be very helpful!!
Thanks

Hello. I m sorry for long answer. 

 

So, I use rebuilded ffmpeg with cedrus, cedrus driver and modified dts from @ubobrov.  Dts and driver is here.  You can try to follow this

 

I m hope I could help you. 

 

[matteog]

32 minutes ago, a.rodionov said:

I use rebuilded ffmpeg with cedrus, cedrus driver and modified dts from @ubobrov.

 

Thank you for the reply.
So no kernel build necessary? Just the .ko module?
On which kernel did you do this?
Thanks!

[a.rodionov]

3 часа назад, matteog сказал:

Thank you for the reply.
So no kernel build necessary? Just the .ko module?
On which kernel did you do this?
Thanks!

You are welcome. Yes, you need just .ko module.  I  used 5.7 kernel, but everything that i did works fine wih 5.4+ kernels. If i will speak more accuracy, i should to say that i have builded custom kernel in this project, but i did it for another changes, not for h264 encoding. 

[schunckt]

Hi!

 

That sounds interesting to me as well. I'll give it a try on a   NanoPi Duo2     5.10.34-sunxi

Does that solution use the vaapi layer?

 

Forgot to mention - possibly this is no longer necessary at all since the cedrus driver already appears as /dev/video0

but im still unsure if and how this is usable ...

 

  T.

Edited May 26, 2021 by schunckt
comment added

[lam96pt]

On 7/14/2020 at 11:31 PM, @lex said:

Encoding using cedrus264 was for legacy kernel.

I think Uboborov has done some work with mainline kernel for encoding with cedrus. Search the forum for "encoding" or uboborov

 

On 7/14/2020 at 4:00 PM, a.rodionov said:

Armbianmonitor:

https://drive.google.com/file/d/1ik7BWZdcjR7WsVOnyZB7M9mV0Qu0JQtY/view?usp=sharing

Hi all. 

I have a board nanopi neo air with armbian linux on it.   We with my colleague try to setting up ffmpeg with cedrus264 for hardware accelerated  capturing video from ov5640-like camera device. 

There is my  armbianmonitor.  

There are my v4l2-ctl -d /dev/videox -D out:

 

  Reveal hidden contents

Driver Info:
    Driver name      : cedrus
    Card type        : cedrus
    Bus info         : platform:cedrus
    Driver version   : 5.4.51
    Capabilities     : 0x84208000
        Video Memory-to-Memory
        Streaming
        Extended Pix Format
        Device Capabilities
    Device Caps      : 0x04208000
        Video Memory-to-Memory
        Streaming
        Extended Pix Format
Media Driver Info:
    Driver name      : cedrus
    Model            : cedrus
    Serial           : 
    Bus info         : platform:cedrus
    Media version    : 5.4.51
    Hardware revision: 0x00000000 (0)
    Driver version   : 5.4.51
Interface Info:
    ID               : 0x0300000c
    Type             : V4L Video
Entity Info:
    ID               : 0x00000001 (1)
    Name             : cedrus-source
    Function         : V4L2 I/O
    Pad 0x01000002   : 0: Source
      Link 0x02000008: to remote pad 0x1000004 of entity 'cedrus-proc': Data, Enabled, Immutable
 

  Reveal hidden contents

Driver Info:
    Driver name      : sun6i-video
    Card type        : sun6i-csi
    Bus info         : platform:camera
    Driver version   : 5.4.51
    Capabilities     : 0x84200001
        Video Capture
        Streaming
        Extended Pix Format
        Device Capabilities
    Device Caps      : 0x04200001
        Video Capture
        Streaming
        Extended Pix Format
Media Driver Info:
    Driver name      : sun6i-csi
    Model            : Allwinner Video Capture Device
    Serial           : 
    Bus info         : 
    Media version    : 5.4.51
    Hardware revision: 0x00000000 (0)
    Driver version   : 5.4.51
Interface Info:
    ID               : 0x03000003
    Type             : V4L Video
Entity Info:
    ID               : 0x00000001 (1)
    Name             : sun6i-csi
    Function         : V4L2 I/O
    Pad 0x01000002   : 0: Sink, Must Connect
      Link 0x02000007: from remote pad 0x1000006 of entity 'Borei_VideoDriver 0-003c': Data, Enabled, Immutable
 

After some reconfiguration of dts, we can successfully capture video in low fps with 

ffmpeg -f v4l2 -framerate 25 -video_size 960x720 -i /dev/video1 output.mkv

but with 

ffmpeg -f v4l2 -channel 0 -video_size 960x720 -i /dev/video1 -pix_fmt nv12 -r 30 -b:v 64k -c:v cedrus264 test.mp4

ffmpeg failed: 

  Reveal hidden contents

root@nanopiair:/home/pi# ffmpeg -f v4l2 -channel 0 -video_size 960x720 -i /dev/video1 -pix_fmt nv12 -r 30 -b:v 64k -c:v cedrus264 test.mp4 > ffmpeg_error.txt
ffmpeg version git-2015-01-22-f86a076 Copyright (c) 2000-2014 the FFmpeg developers
  built on Jul 14 2020 09:02:41 with gcc 8 (Debian 8.3.0-6)
  configuration: --prefix=/usr --enable-nonfree --enable-gpl --enable-vdpau --enable-libx264 --enable-libv4l2
  libavutil      54.  6.100 / 54.  6.100
  libavcodec     56.  0.101 / 56.  0.101
  libavformat    56.  2.100 / 56.  2.100
  libavdevice    56.  0.100 / 56.  0.100
  libavfilter     5.  0.102 /  5.  0.102
  libswscale      3.  0.100 /  3.  0.100
  libswresample   1.  1.100 /  1.  1.100
  libpostproc    53.  0.100 / 53.  0.100
Input #0, video4linux2,v4l2, from '/dev/video1':
  Duration: N/A, start: 1581.872201, bitrate: 248832 kb/s
    Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 960x720, 248832 kb/s, 30 fps, 30 tbr, 1000k tbn, 1000k tbc
File 'test.mp4' already exists. Overwrite ? [y/N] y
[cedrus264 @ 0x241ff20] VE Open error.
Output #0, mp4, to 'test.mp4':
    Stream #0:0: Video: h264, q=2-31, 128 kb/s, 30 fps
    Metadata:
      encoder         : Lavc56.0.101 cedrus264
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (cedrus264))
Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
 

 

Ffmpeg was buided from FFmpeg-Cedrus, also we  try follow this and use this binaries - result haven't changed. 

 

After some inspecting sources of ffmpeg-cedrus we found that ffmpeg try to open /dev/cedar_dev device, which don't exists in our system.  Ffmpeg fails on opening device. If we replace /dev/cedar_dev by /dev/video0, where is our cedrus driver, ffmpeg fails on ioctl IOCTL_GET_ENV_INFO.  In source, it is 99 or 101 line.

We check kernel config for configuration, provided by Sunxi-Cedrus and determine that cedrus was configured right. 

 

Also we  have tried all described actions on official armbian buster image for nanopi neo air and we have got the same result. 

What we does wrong? 

 

I got the same error on orange pi pc and newest armbian (mainline based kernel 5.10.y) on it. Please show me how did you fix it step by step. I spent 3 days but not work

 

this is my command

Quote

sudo ffmpeg-3.1.4 -f v4l2 -channel 0 -video_size 1280x720 -i /dev/video0 -pix_fmt nv12 -r 22 -qp 30 -c:v cedrus264 -c:a None night_video_test4_1920x1080.mp4

ffmpeg failed:

Quote

sudo ffmpeg-3.1.4 -f v4l2 -channel 0 -video_size 1280x720 -i /dev/video0 -pix_fmt nv12 -r 22 -qp 30 -c:v cedrus264 -c:a None night_video_test4_1920x1080.mp4
[sudo] password for lam:
ffmpeg version 3.1.4 Copyright (c) 2000-2016 the FFmpeg developers
  built with gcc 4.8 (Ubuntu/Linaro 4.8.4-2ubuntu1~14.04.3)
  configuration: --prefix=/usr --enable-nonfree --enable-gpl --enable-version3 --enable-vdpau --enable-libx264 --enable-libmp3lame --enable-libpulse --enable-libv4l2 --enable-libcedrus264
  libavutil      55. 28.100 / 55. 28.100
  libavcodec     57. 48.101 / 57. 48.101
  libavformat    57. 41.100 / 57. 41.100
  libavdevice    57.  0.101 / 57.  0.101
  libavfilter     6. 47.100 /  6. 47.100
  libswscale      4.  1.100 /  4.  1.100
  libswresample   2.  1.100 /  2.  1.100
  libpostproc    54.  0.100 / 54.  0.100
Input #0, video4linux2,v4l2, from '/dev/video0':
  Duration: N/A, start: 109.484156, bitrate: 147456 kb/s
    Stream #0:0: Video: rawvideo (YUY2 / 0x32595559), yuyv422, 1280x720, 147456 kb/s, 10 fps, 10 tbr, 1000k tbn, 1000k tbc
File 'night_video_test4_1920x1080.mp4' already exists. Overwrite ? [y/N] y
[cedrus264 @ 0x16e0060] VE Open error.
Output #0, mp4, to 'night_video_test4_1920x1080.mp4':
    Stream #0:0: Unknown: none
    Metadata:
      encoder         : Lavc57.48.101 cedrus264
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (cedrus264))
Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height

 

[schunckt]

Regarding the fixed dts, can one provide the original dts? I'd like to have some base to diff because i want to figure out what has been changed.

 

I'll try to port these to a Nannopi Duo2  6.1.30

 

 

  T.

[vk vk]

Hi all. for kernels 5.1x and 6.1.xx - is it any working hardware encoding solutions for H3 soc? for Nano Pi neo/neo2 ?