GStreamer

GStreamer is a library for constructing graphs of media-handling components. The applications it supports range from simple Ogg/Vorbis playback, audio/video streaming to complex audio (mixing) and video (non-linear editing) processing.
— gstreamer.freedesktop.org

Overview

We implemented two custom plugins (one for video and one for audio) in order to turn a stream of bytes into a stream of valid Moonlight RTP packets.
This allows users to be able to compose Gstreamer pipelines in any form or shape and, near the end, plug our plugin so that it can be streamed to Moonlight.

The plugin

Our plugin sits between the encoded bitestream and a downstream UDP sink that will send the RTP packets to Moonlight
Figure 1. Our plugin sits between the encoded bitestream and a downstream UDP sink that will send the RTP packets to Moonlight

The basic flow is the following:

  • The plugin receives a GstBuffer of binary data: the encoded video or audio stream into a format that can be decoded back by the Moonlight client.

  • We’ll split this into a GstBufferList of valid RTP packets following the Moonlight specifications; this includes:

    • split the original buffer into a set of correctly sized chunks

    • Encrypt the chunks when enabled

    • add RTP headers to each chunk

    • create additional RTP packets with the FEC information

  • Push the resulting list downstream

An example pipeline for *video* encoding and streaming
Figure 2. An example pipeline for video encoding and streaming
An example pipeline for *audio* encoding and streaming
Figure 3. An example pipeline for audio encoding and streaming

This flow is general enough to enable us to use the same plugin for H.264, HEVC and AV1 (and any other encoder that will be supported in future). Downstream, it’s also trivial to switch between UDP/TCP or even a complete different delivery method if needed.

We decided to split between audio and video because they have different RTP packet structure, non overlapping properties, different FEC encoding and different encryption requirements but the basic flow is the same for both plugins.

Given that this is a direct transformation from one input buffer to another output buffer we decided to use GstBaseTransform as the base class. This will implement some boilerplate code for us so that we can focus on the main _transform virtual method implementation.

Following the nomenclature used in the official GStreamer RTP plugins we called them rtpmoonlightpay_video and rtpmoonlightpay_audio as in RTP Moonlight Payloader video/audio.

Code

GStreamer uses GObjects: a mechanism to retrofit Objects into C. I suggest you to get a bit familiar with it before diving into the code; here’s a great and simple tutorial.

Following the official docs at Plugin Write’s Guide we started off with the official bash script that creates the boilerplate code from the selected base class; namely: