Making a Theater in VRChat or: How The Vantage Works
Published 4/20/18 - Last Updated 5/22/20
The Vantage is my private theater inside of VRChat. I've been working on it for quite a while now, slowly updating and improving it. The Vantage currently supports 3D video and 5.1 surround sound.
What is outlined below is not an exhaustive in-depth step-by-step guide. It is a description of how it all works. Knowledge of Unity, VRC Triggers, and VRChat world-building is assumed. Start with the Developer menu on the VRChat website if you are new to building VRChat worlds. And be sure to check out the example scenes in the VRC SDK. Also, know that theaters and playing back video in VRChat will become easier over time. Be sure to send VRChat feedback on this topic.
The Player
I use VRC_SyncVideoPlayer.
The Screen
The screen currently supports standard video, Side-by-side (SBS) and Over-Under (OU) 3D formats along with real-time global illumination to cause the environment to glow in response to the video. This is all done with a set of 4 different screens that are enabled and disabled as needed.
Each of these screens has a unique material applied. They are fed the video with a single render texture from your Video Player component. My theater makes use of a set of shaders made by TCL987 for the 3D screens. They can be found on his GitHub repository.
- Standard
- A standard material with the Video Player render texture in the Emissive slot, Albedo set to black and the Emission strength at 1.
- Side-by-side 3D
- A material that uses TCL's 3D shader configured for SBS with the Video Player Render Texture in the texture slot.
- Over-under 3D
- A material that uses TCL's 3D shader configured for OU with the Video Player Render Texture in the texture slot.
- Glowing
A material configured just like the material for the Standard screen, but the emission strength is increased to increase the glow that the world will receive. Additionally, add the VRC_Custom Renderer Behavior component to this screen.
The rest of your world must also be properly set up with baked lighting and Light Probes.
The Video
VRC_SyncVideoPlayer supports video files that Unity natively supports. I prefer to use video files that have an MP4 container containing an H.264 video track and AAC audio track(s). I use Handbrake to convert my video files.
As far as Bitrate and other quality settings go, those all depend on the content of videos you are encoding, the server you are hosting the videos from, how many people are watching, and your taste in video quality. I usually shoot for an average bitrate of ~4.0 Mbit/s for the final file with my configuration.
For a video to work well in VRChat with VRC_SyncVideoPlayer for a wide range of users it should have these properties:
- MP4 Container
- H.264 video
- 48khz AAC Audio
- Constant framerate (do not use variable framerate)
- Framerate of 30fps or less
- Resolution of 1920x1088 or lower
Additionally, the following things should be true about the video file:
- B-frames should not be turned off. For example, don't encode with with the `Zero Latency` tuning preset in Handbrake. This is new in VRChat 2020/Unity 2018.
- The video should be encoded with the 'fast decode' option turned on in Handbrake to turn off advanced compression techniques.
- The video should be 'Web Optimized' (Sometimes called 'Fast Start') to move the moov atom of the MP4 container to the start of the file. - This will allow video playback to start faster than it would otherwise.
The Audio
The Vantage currently supports 5.1 surround sound. This is achieved through converting the audio to be six discrete AAC audio tracks. It is then played in the theater through six Audio Source components positioned where 5.1 speakers go. This same process and speaker setup allows for videos with stereo audio to be played in stereo too!
My encoding process:- Encode the video in Handbrake with the settings I want for the video track, while setting the audio codec to passthru to carry the source audio directly into the output file that will be used in the next step.
- I then take this resulting video file and run it through FFMPEG with the following command(s) to convert the source surround sound/stereo audio track into six discrete audio tracks, one for each channel. (Note that this command also forces 48khz audio and enables the 'Fast Start' option. Both mentioned above.)
FFMPEG 3.4.x:
ffmpeg -i INPUT.MKV -vcodec copy -filter_complex channelsplit=channel_layout=5.1 -acodec aac -ar 48000 -movflags faststart OUTPUT.MP4
FFMPEG 4.x and newer:
ffmpeg -i INPUT.MKV -filter_complex "channelsplit=channel_layout=5.1[FL][FR][FC][LFE][BL][BR]; [FL]aformat=channel_layouts=mono[FL]; [FR]aformat=channel_layouts=mono[FR]; [FC]aformat=channel_layouts=mono[FC]; [LFE]aformat=channel_layouts=mono[LFE]; [BL]aformat=channel_layouts=mono[BL]; [BR]aformat=channel_layouts=mono[BR]" -map 0:v -map '[FL]' -map '[FR]' -map '[FC]' -map '[LFE]' -map '[BL]' -map '[BR]' -vcodec copy -acodec aac -ar 48000 -movflags faststart OUTPUT.MP4
To get the Video Player to output to the right speakers, create six separate Audio Source GameObjects and place them in your theater and configure your Video Player component with them in this order (This changed in Unity 2018):
Note: Unprocessed videos will play all audio through the first audio source (Front Left) in this speaker configuration. If Front left is not your preferred location, you can move it as needed. I use an animation controller and triggers to move the position of the speaker from Front Left to Center. You could also create two theaters with different speaker setups.
Video Hosting
Videos can, of course, be played back from YouTube, Vimeo and similar hosts that are supported by VRC. But to host your files, such as ones with the Surround Sound in them, you will need to use a web host. The key is to have it be fast, have good routing to everyone in the theater, and having it cost something you can afford.
Changing the Video
Currently, the video that plays when you press the play button can be dynamically changed in The Vantage. This is done by a server-side script. I use my own script that I wrote and host for my own use. Previously I was using one created by the VRC community member Euan. Without Euan's script and his help, The Vantage would not be what it is today. Unfortunately, I can not give you access to either script.
Thank You
Thank you to TCL, Hackspanner, Euan, Afromana, Morgus, WACOMalt, Madman and many others for all their help testing The Vantage over the years.