The cast_streaming
component provides a wrapper around all Cast mirroring and
Cast remoting receiver functionality in a platform-agnostic way. The Cast
Streaming transport/protocol is
implemented
by the
Openscreen library,
and media rendering is handled by the Chromium media stack, so this component
acts as an intermediary between the two. It is currently used both as part of
Fuchsia WebEngine and the //components/cast_receiver
Chromecast
implementation, which can be used to run a Cast Streaming receiver on Linux.
This component does NOT provide any Cast sender support, although some code in
//media/cast/openscreen
is shared with the sender.
enable_cast_receiver = true
cast_streaming_enable_remoting = true
Starting the cast_streaming
component in the browser process has two parts:
- Calling
SetNetworkContextGetter()
to set the network context to use for the streaming session. - Creating
a new
ReceiverSession
object and callingReceiverSession::StartStreamingAsync()
on it.
In the ContentRendererClient
for the service, a
new instance
of cast_streaming::ResourceProvider
must be
returned
by CreateCastStreamingResourceProvider()
function
overload.
The remainder of integration is already taken care of in the //content
layer.
The code for this component can roughly be broken down into a few parts, as reflected by the component’s directory structure:
This section of code is responsible for sending frame data from Openscreen to
the media pipeline. This is located in /browser/frame
, /renderer/frame
, and
/common/frame
.
This section of code is responsible for sending media::Renderer
commands to
the embedder-specific Renderer on top of which cast_streaming
is running. It
is located at /browser/control
, /renderer/control
, and /common/control
.
Selection of this Renderer
is as with vanilla Chromium - through the
MediaFactory
.
No cast_streaming
-specific changes are required.
A subset of Control which is responsible for translating control commands to and
from the cast_streaming
media remoting protocol, a proto-based communication.
Code is located within the /control
directory, e.g
/browser/control/remoting
.
An alternative implementation of the Renderer-process side of the frame implementation. This section is minimal, as its implementation has largely been integrated into the standard Frame flow to avoid code duplication.
In the diagrams below, note the following:
- Blue boxes represent external code (e.g. Openscreen, Embedder-controlled infra)
- Yellow boxes are used to represent the Renderer-process classes
- White boxes represent the Browser-process classes
- Gray boxes represent remoting-only classes
- Purple lines represent mojo calls
The startup process for the cast_streaming
component can most easily be
visualized by splitting up the Browser and Renderer processes. Very few
communications are made between the two, so they can largely be viewed
independently.
On the browser side
ReceiverSessionImpl
creates and starts aCastStreamingSession
, which in turn creates aPlaybackCommandDispatcher
and starts anopenscreen::cast::ReceiverSession
.- The
openscreen::ReceiverSession
negotiates a session and passes it toCastStreamingSession
, If it’s a remoting session, wait for the “real” configs to be sent from the sender side. - In either case,
CastStreamingSession::StartStreamingSession()
then creates the remainder of the requisite objects. - Then, the first frame is “preloaded”. The first frame of data is
read
and its timestamp is used to call
StartPlayingFrom()
on the associatedmedia::Renderer
instance in the renderer process. In the case of remoting, more frames must also be requested from the remote renderer running on the streaming sender.
The files involved in this section are located mainly at the top-level of each
directory (i.e. //components/cast_streaming/browser
or
//components/cast_streaming/renderer
), which make calls into frame and control
as necessary.
Starting of the Control and Frame pathways in the Renderer process is entirely separate, so the two can be examined separately:
For the former of these, the setup is triggered by the MediaFactory
which owns
the
root-level singleton
objects responsible for enabling the flow. In order to create a
media::Renderer
mojo connection between the browser and renderer processes,
ResourceProviderImpl
is
used as an intermediary
to avoid timing issues. The receiver side of this pipe is
passed
to the PlaybackCommandForwardingRenderer
during its creation, which will
act in response to
these commands by forwarding them to the underlying embedder-specific
media::Renderer
instance, as well as
forwarding back
any RendererClient
events.
Creation of the Frame channel is slightly more complex. First, an override from
the embedder-specific ContentBrowserClient
triggers usage of the
FrameInjectingDemuxer
. The renderer-process being “ready” is signified by both
of:
- Receipt of a call
by the
DemuxerConnector
- Initialization
of the
FrameInjectingDemuxer
At that point, the browser process will
send
the OnStreamsInitialized()
call which
provides the connection information
to create all remaining objects and begin pulling frames from the browser
process.
At a high level, sending frame data to the media pipeline in the renderer process works as follows:
- Following a
DemuxerStream::Read()
call, theFrameInjectingDemuxerStream
triggers aGetBuffer()
mojo call. - The
DemuxerStreamDataProvider
receives this call, and makes a request to theStreamConsumer
to get a frame. - If none exist,
wait for them.
In the remoting case, also
send a notification
to the remoting sender to have more frames sent
through
the
RpcDemuxerStreamHandler
. - Once frames are received, parse the
DecoderBuffer
, write thedata()
field to a pipe and return the remainder to theDemuxerStreamDataProvider
. - The
DemuxerStreamDataProvider
receives and then sends this frame data to the Renderer, where it gets combined back with its data and provided to theFrameInjectingDemuxerStream
.
A config change can be triggered in two ways:
- Changing of the config during an ongoing Remoting session.
- Re-configuration of the streaming session.
Changing the config during an ongoing remoting session occurs in a number of steps:
- Steps 1-3 above send a request to the remoting sender to send more data.
- Instead of (or in addition to) frames, a new
AudioDecoderConfig
/VideoDecoderConfig
are sent. - The receiver will then “reset its state” by:
- Stopping all flowing data (so nothing incorrect gets displayed).
- Flushing any data still in the media pipeline.
- Clearing any remoting state.
- Re-initializing
the streaming session with
StartStreamingSession()
.
- The pre-load flow will
again be used
to start playback of the streaming session. In an ideal world, it would be safe
to wait for a
StartPlayingFrom()
call from the remote sender, but in practice that will only be sent intermittently and cannot be relied on. - The
DemuxerStreamDataProvider
will send the new config to the Renderer process. - The
FrameInjectingDemuxerStream
will receive a newStreamConsumer
associated with this new stream, and may then will return the config as part of the next (or ongoing)Read()
call.
This situation occurs either when an ongoing mirroring session re-configures itself (e.g. as the result of Chrome changing the quality of an ongoing session) or when the user changes between mirroring and remoting.
As with previous sections, this scenario is largely the same as the remoting
section, except that instead of a new config being received through the
ReadUntil()
call, the OnNegotiated()
function will
immediately provide the
new config and “reset the
state” of the pipeline.
TODO(crbug.com/1208196): Add these details
In the Renderer process, the same media::DemuxerStream
and media::Renderer
instances are used, even when the stream is re-initialized. Rather than
re-creating the entire pipeline, Flush()
and StartPlayingFrom()
commands are
sent and the same instances are used.
Preloading was a concept added relatively late into the development of the
cast_streaming
component to solve a number of edge cases:
- When mirroring, the sender does not always start from
pts = 0 ms
. In such cases, naively callingStartPlayingFrom(0 ms)
as has historically been relied upon in WebEngine does not work. - When remoting, a
StartPlayingFrom()
command is not always sent.
In order to account for such cases, a StartPlayingFrom()
must be “injected
in”, for which the timestamp of the first frame is required. It is also true
that this approach decreases the playback delay between the sender and receiver,
but that is more of a happy coincidence than the original goal of this workflow.