XU4 h264_v4l2m2m ffmpeg encoding

sgjava · December 13, 2019

Armbianmonitor:

My end game is to use hardware h264 encoding for my security camera streams (using OpenCV, etc) . The first step is testing ffmpeg. It appears that all the necessary pieces are already present in the release I'm using. If I use:

ffmpeg -i centaur_2.mpg -acodec aac -vcodec h264 -b:v 2M -pix_fmt nv21 test.mp4

CPU stays below 12% and I get 44 FPS. Seems like hardware encoding is working here.

If I use:

ffmpeg -i centaur_2.mpg -acodec aac -vcodec h264_v4l2m2m -b:v 2M -pix_fmt nv21 test.mp4

About the same performance, but the file is 10x larger!

If I use:

ffmpeg -i centaur_2.mpg -acodec aac -vcodec libx264 -b:v 2M -pix_fmt nv21 test.mp4

About the same as h264. So it looks to me like h264_v4l2m2m is used by h264 and libx264 codecs?

Linux 4.14.150-odroidxu4 #1 SMP PREEMPT Mon Oct 28 07:56:57 CET 2019 armv7l armv7l armv7l GNU/Linux

BOARD=odroidxu4
BOARD_NAME="Odroid XU4"
BOARDFAMILY=odroidxu4
BUILD_REPOSITORY_URL=https://github.com/armbian/build
BUILD_REPOSITORY_COMMIT=1221d592
VERSION=5.95
LINUXFAMILY=odroidxu4
BRANCH=default
ARCH=arm
IMAGE_TYPE=stable
BOARD_TYPE=conf
INITRD_ARCH=arm
KERNEL_IMAGE_TYPE=zImage

ffmpeg -encoders | grep 264

 V..... libx264              libx264 H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 (codec h264)
 V..... libx264rgb           libx264 H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 RGB (codec h264)
 V..... h264_omx             OpenMAX IL H.264 video encoder (codec h264)
 V..... h264_v4l2m2m         V4L2 mem2mem H.264 encoder wrapper (codec h264)
 V..... h264_vaapi           H.264/AVC (VAAPI) (codec h264)

sgjava · December 14, 2019

Just a follow up while trying to run 3 cameras on the UX4 like I was doing with HK's Ubuntu release. Armbian always is showing a large amount of nice time. I'm not sure if this is the hardware encoder, but I'm not seeing this with HK's release. I'm also having issues where the XU4 just dies and shuts down. I might dig around in the logs, but there's definitely something up. I'll switch back to HK release and try the same configuration and see what happens.

Armbian release:

HK release:

gounthar · December 14, 2019

Interesting, to say the least.

Igor · December 14, 2019

Can you provide a kernel config diff too see if there is something odd?

sgjava · December 14, 2019

Trying HK's image now with same configuration. Using Linux video 4.14.157-171 #1 SMP PREEMPT Wed Dec 4 08:21:54 -03 2019 armv7l armv7l armv7l GNU/Linux. I'll circle back once I test this out to get past my current need. Looks like hardware encoding is embedded already as well.

sgjava · December 15, 2019

OK, so I'm seeing the same behavior with HK's image. I have sample code in my OpenCV install script. https://github.com/sgjava/install-opencv/blob/master/opencv-python/codeferm/writer.py just takes a source video and encodes it to whatever Four CC is set. If I use X264 or H264 I get:

%Cpu(s):  9.9 us,  1.1 sy, 24.6 ni, 63.2 id,  0.0 wa,  1.0 hi,  0.2 si,  0.0 st

2019-12-15 12:47:40,293 INFO     writer OpenCV 4.2.0-pre
2019-12-15 12:47:40,294 INFO     writer Input file: ../../resources/traffic.mp4
2019-12-15 12:47:40,294 INFO     writer Output file: ../../output/writer-python.avi
2019-12-15 12:47:40,354 INFO     writer Resolution: 480x360
2019-12-15 12:47:52,655 INFO     writer 925 frames
2019-12-15 12:47:52,656 INFO     writer 75.3 FPS, elapsed time: 12.29 seconds

859844 Dec 15 12:53 writer-python.avi

With XVID (MP4 simple profile) I get:

%Cpu(s):  9.9 us,  0.4 sy,  0.0 ni, 89.1 id,  0.0 wa,  0.5 hi,  0.0 si,  0.0 st

2019-12-15 12:50:05,547 INFO     writer OpenCV 4.2.0-pre
2019-12-15 12:50:05,547 INFO     writer Input file: ../../resources/traffic.mp4
2019-12-15 12:50:05,547 INFO     writer Output file: ../../output/writer-python.avi
2019-12-15 12:50:05,608 INFO     writer Resolution: 480x360
2019-12-15 12:50:21,139 INFO     writer 925 frames
2019-12-15 12:50:21,139 INFO     writer 59.6 FPS, elapsed time: 15.52 seconds

3025586 Dec 15 12:50 writer-python.avi

I love the smaller file I get with H264, but the CPU overhead is much larger than I though it would be using hardware encoding. I'm going to play around with ffmpeg directly and see if I'm seeing the same issues.

sgjava · December 17, 2019

Using ffmpeg -i motion-19-05-10.avi -an -vcodec h264_v4l2m2m -b:v 2M -pix_fmt nv21 test.mp4 I get the results I expected. Hardware encoding works fine. The key is to look for Stream #0:0 -> #0:0 (mpeg4 (mpeg4_v4l2m2m) -> h264 (h264_v4l2m2m)). The issue with the CPU nice time was caused by OpevCV using the software H264 encorder/decoder. I need to look for a way for OpenCV to use the hardware encoder for video encoding.

Basically h264_v4l2m2m is working as expected in HK's image, so it should work the same way in Armbian.

sgjava · December 18, 2019

I ended up using ffmpeg-python since I know ffmpeg hardware encoding/decoding work. Unfortunately OpenCV doesn't allow a lot of flexibility on the video back end (VideoWriter). I'll end up using a ffmpeg adaptor for my security system since ffmpeg-python allows me to set vcodec and convert numpy arrays. To hardware encode h264 use:

#!/usr/bin/env python
import ffmpeg

(
    ffmpeg
    .input('test.avi')
    .output('out.mp4', vcodec='h264_v4l2m2m', pix_fmt='nv21', **{'b:v': 2000000})
    .run()
)

Output #0, mp4, to 'out.mp4':
  Metadata:
    encoder         : Lavf57.83.100
    Stream #0:0: Video: h264 (h264_v4l2m2m) (avc1 / 0x31637661), nv21, 1280x720, q=2-31, 2000 kb/s, 7 fps, 14336 tbn, 7 tbc
    Metadata:
      encoder         : Lavc57.107.100 h264_v4l2m2m