diff --git a/text/0055-webrtc-simulcast.md b/text/0055-webrtc-simulcast.md new file mode 100644 index 0000000..7bb130f --- /dev/null +++ b/text/0055-webrtc-simulcast.md @@ -0,0 +1,136 @@ +# Summary + +Add Simulcast support to WebRTC output. + +Simulcast is a WebRTC protocol feature that allows a uploader to send multiple encodes of one track. This is built into the protocol so every WebRTC +ingest already understands this. These layers can be different resolutions, bitrates and even codecs. + +# Motivation + +The sucess/demand for this feature has already been demonstrated with [Twitch's Multiple Encodes](https://help.twitch.tv/s/article/multiple-encodes). +These are my personal reasons I am working on this. + +* **Low Latency** - Transcodes add latency. WebRTC is chosen by OBS users because they want an interactive streams, transcodes break that. + +* **Cheaper** - Running a streaming service is expensive. This would allow more streaming services to exist and for more self hosting. + +* **Prevent Tampering/Ad Insertion** - WebRTC provides APIs for E2E Encryption. This means a user can upload Simulcast and share the keys viewers. Services can't modify the media anymore. + +* **Better Quality** - 'Free Transcodes' is a race to the bottom. Streaming sites do the cheapest/lowest quality possible (to save money). Simulcast gives control to the user. + +* **P2P Streaming** - Simulcast + E2E would enable P2P. You can have a mesh of viewers that distribute media to each other. Since things are E2E Encrypted you can detect tampering. + +# Design + +## Provide a `QSpinBox` with description to configure Simulcast Layers + +When users select `WHIP` for their service a description will appear below `Ignore streaming service setting recommendiations`. + +This QSBinBox will say `Simulcast Layers`. This value will be 1 by default. This value can be 1 to 4. +The Height/Width/Bitrate of each layer is scaled from the users global choice. So depending on the users choice. + +* 1 Layer + * 100% Height/Width/Bitrate + +* 2 Layers + * 100% Height/Width/Bitrate + * 50% Height/Width/Bitrate + +* 3 Layers + * 100% Height/Width/Bitrate + * 66% Height/Width/Bitrate + * 33% Height/Width/Bitrate + +* 4 Layers + * 100% Height/Width/Bitrate + * 75% Height/Width/Bitrate + * 50% Height/Width/Bitrate + * 25% Height/Width/Bitrate + +## Create multiple encoders with the requested settings. + +A new member will be added to the `BasicOutputHandler` class. This new member `std::vector simulcastEncoders` will contain +the encoders. + +`BasicOutputHandler` will query `SimulcastLayers`. If greater then 0 it will loop and create the encoders. + +## Configure PeerConnection in `obs_output_info.start` + +`WHIPOutput::Start` will use `obs_output_get_video_encoder2` to query how many encoders exist. It will create a WebRTC offer with the requested details. + +This offer contains the number of layers and the bitrate/resolution of them. The server can process this and decide to accept it or reject. + +The server may respond with an error that 'Simulcast Configuration was not accepted'. The user can then modify +how many layers they send and try again. Services can configure this ahead of time in OBS. + +## Server can reject layers + +A server may wish to only accept a subset of the layers offered. Today if the user rejects any layers we throw an error and say how many +layers were accepted. The user can lower the amount of layers they are sending and try again. Services in their documentation will be +expected to tell users how they want video encoded and how many layers they are willing to accept. + +# Proposed UX + +## Stream Page + +![Service Example](./0055-webrtc-simulcast/service-page.png) + +## Simple Output Mode + +No changes will be made. + +## Advanced Output Mode + +No changes will be made. + +# Alternatives + +## Allow users to configure Bitrate/Resolution manually + +This would require a large amount of UI work. The consensus was this would be frustrating for users. Nothing stops this from being done in the future though. + +## Extend WHIP Protocol to enable 'Server Driven Configuration' + +WHIP is an existing protocol with many users of Simulcast already. Something that does 'Server Driven Configuration' could be added to the protocol, but +it would have to be done in a way that doesn't break existing clients. The value/advantage of WHIP is that it maintains compatability with 13 years of WebRTC servers and clients. + +This is something we could take to the IETF. I don't anticipate much support since clients already have implemented 'Configuration driven by enviroment' (See point below) + +One possibility is `GetClientConfiguration` is opt-in and driven via `services.json`. + +## Configuration driven by enviroment + +WebRTC clients today configure the amount of layers by observing the enviroment. They measure CPU Load/Network usage and enable layers dynamically. This +is really powerful since the use cases for OBS are so diverse. If someone is streaming a demanding game they will want less layers. If you are just streaming +your desktop you can accomodate more. + +This could be possible in the future, but requires a lot more work. + +# Drawbacks + +## Additional Complexity + +Simulcast will add additional code to the OBS code base. If this feature is unused it would be extra burden on the project for no value. + +## More Compute/Network Usage + +Simulcast will require extra compute and network. If users encounter hardware/network overusage they will need to reduce bitrate/resolution. + +In the future I would like to write a dynamic simulcast feature. The WebRTC protocol allows layers to be disabled/enabled at anytime. We could detect network +overusage and disable layers to ensure the streamer has the best experience possible. We would then re-enable them when the network and/or hardware recovers. + +I develop and use OBS on a T420 (a laptop from 2011). When streaming using WebRTC and using x264 I see ~6% CPU usage in OBS. With Simulcast enabled I see ~8%. I can gather +more fine performance information if that helps. + +# Additional Information + +How these layers are played is entirely controlled by the server operator. + +With Broadcast Box I provide a drop down so users can explicitly choose the layer they want to watch. + +![Broadcast Box](./0055-webrtc-simulcast/broadcast-box.png) + +A better implementation would be to dynamically measure the users internet and deliver them the best rendition their internet can support. + +Implementation: +* [PR](https://github.com/obsproject/obs-studio/pull/9165) diff --git a/text/0055-webrtc-simulcast/broadcast-box.png b/text/0055-webrtc-simulcast/broadcast-box.png new file mode 100644 index 0000000..6769e10 Binary files /dev/null and b/text/0055-webrtc-simulcast/broadcast-box.png differ diff --git a/text/0055-webrtc-simulcast/service-page.png b/text/0055-webrtc-simulcast/service-page.png new file mode 100644 index 0000000..e36e775 Binary files /dev/null and b/text/0055-webrtc-simulcast/service-page.png differ