QtWebEngine/VideoAcceleration

Qt WebEngine does not currently support hardware accelerated decoding of videos.

Introduction

One of the most complex operations performed in the internet browsers is the video decoding and encoding. These compute operations can be executed as part of the software implementation which runs on the general purpose CPU, or they can be hardware accelerated and executed by the dedicated hardware. In case of regular PCs, the video hardware is integrated into a GPU (manufactured by Intel, AMD or Nvidia) and for the embedded platforms it is in most cases a part of independent IP core included in SoC.

The use of dedicated video hardware offloads general propose CPU from intensive decodec/codec algorithms. This results in a better interaction performance and it is crucial for embedded devices, where the CPU processing power is limited. Moreover, using an optimized dedicated hardware usually results in the better video performance, quality and the lower power consumption compared than software based decoding. However, hardware decoders rarely support all possible video formats, video profiles and codes' algorithms. Therefore, the video acceleration support is limited in that perspective.

Video playback in Chromium

• https://www.chromium.org/audio-video
• https://www.chromium.org/developers/design-documents/video
• http://www.chromium.org/developers/design-documents/video-playback-and-compositor
• https://www.chromium.org/developers/design-documents/multi-process-architecture

In general the video can be played in Chromium using:

• Pepper Plugins APIs, PNaCl (see pp::VideoDecoder), pepper flash plugin, WideVine plugin
• WebRTC video encoders and decoders
• HTML <video> tag (by using <video> pipeline)

To keep things stable, secure and running with good performance chromium browser has a multi-process-architecture. The video playback is also affected by this design and it is split between separate processes. In most cases, the code that performs encoding or decoding of video executes within the renderer process. Since this process is sandboxed, the video hardware is not accessible from it. IPC is used for communication between the renderer and the GPU process that has access to the hardware decoders.

In case of an HTML <video> tag, blink::WebMediaPlayer interface is used to create the video decoding pipeline in the render process. The pipeline consists of various elements like: data source, demuxer, audio decoding, video decoding, audio rendering, and video rendering.

The basic video decode class hierarchy looks like:

Two classes are currently used for software based video playback, VpxVideoDecoder and FFmpegVideoDecoder. They wrap two corresponding libraries:

• libvpx to play video formats like vp8, vp9
• ffmpeg to play other video formats, for example: ogg, mp4(H.264)

Two classes, media::VideoDecodeAccelerator (VDA) and media::VideoEncodeAccelerator (VEA), are the main interfaces for of all video hardware acceleration in Chromium.

Chromium video acceleration platform support

Official Chromium enables video acceleration for Windows, macOS, and ChromeOS builds.

• Mac using Video toolbox (class VTVideoDecodeAccelerator)
• ChromeOS on x86_64 using VA-API (class VaapiVideoDecodeAccelerator)
• ChromeOS on arm/tegra using V4l2 (class V4L2VideoDecodeAccelerator)
• Android using platform MediaCodec
• Windows using DirectX Video Acceleration (class DXVAVideoDecodeAccelerator)

On Linux, Chromium Project is mostly focused on 'desktop builds' which have enough processing power for video encoding and decoding using software based multithreaded ffmpeg. Therefore, to avoid dealing with a possible large number of hardware related bugs, acceleration support on chrome is by default disabled.

• https://bugs.chromium.org/p/chromium/issues/detail?id=463440
• https://bugs.chromium.org/p/chromium/issues/detail?id=137247

However, there is an ongoing community effort to enable video acceleration using VA-API and some Linux distros like 'Arch' patch Chromium already downstream:

• https://codereview.chromium.org/1743543004
• https://codereview.chromium.org/1872393002
• https://chromium-review.googlesource.com/c/chromium/src/+/532294

Qt WebEngine

Current status of video decode acceleration for Qt WebEngine 5.9 and 5.10 is as follows:

• Windows video acceleration does not work, see reasoning in https://codereview.qt-project.org/#/c/202314/ . Bottom line, this has not worked in any Qt WebEngine release due to ANGLE not working with multiple threads. Note this shold fixed after 5.13, tested with 5.15.1 and works.

• macOS video acceleration does not work, see reasoning in https://bugreports.qt.io/browse/QTBUG-60002 and attached patch. This used to work in versions <= Qt 5.8 (needs confirmation). This should be fixed with 5.15.1 however requires opengl core profile.

  QSurfaceFormat surfaceFormat = QSurfaceFormat::defaultFormat();
  surfaceFormat.setRenderableType(QSurfaceFormat::OpenGL);
  surfaceFormat.setVersion(4, 1);
  surfaceFormat.setProfile(QSurfaceFormat::CoreProfile);
  QSurfaceFormat::setDefaultFormat(surfaceFormat);

• Linux (including embedded) video acceleration does not work. Work-in-progress VA-API support patch can be found at https://codereview.qt-project.org/#/c/209204/

Some considerations for future

• Enabling VA API in Qt 5.11
• Implementing a video playback using punch hole for basic use cases
• Supporting GStreamer [1]
• Supporting i.MX VPU hardware video decoding [2]
• Supporting RaspberryPi hardware decoder [3]
• Supporting Zero-Copy for ffmpeg, vpx [4]