Making a Theater in VRChat or: How The Vantage Works

Published 4/20/18 - Last Updated 2/21/19

The Vantage is my private theater inside of VRChat. I've been working on it for quite awhile now, slowly updating and improving it. The Vantage currently supports 3D video and 5.1 surround sound.

What is outlined below is not an exhaustive in-depth step-by-step guide. It is a description of how it all works. Knowledge of Unity, VRC Triggers, and VRChat world building is assumed. Start with the Developer menu on the VRChat website if you are new to building VRChat worlds. And be sure to check out the example scenes in the VRCSDK. Also, know that theaters and playing back video in VRChat will become easier over time. Be sure to send VRChat feedback on this topic.

The Player

I use VRC_SyncVideoPlayer.

The Screen

The screen currently supports standard video, Side-by-side (SBS) and Over-Under (OU) 3D formats along with real-time global illumination to cause the environment to glow in response to the video. This is all done with a set of 4 different screens that are enabled and disabled as needed.

Each of these screens has a unique material applied. They are fed the video with a single render texture from your Video Player component. My theater makes use of a set of shaders made by TCL987 for the 3D screens. They can be found on his GitHub repository.

Standard
A standard material with the Video Player render texture in the Emissive slot, Albedo set to black and the Emission strength at 1.
Side-by-side 3D
A material that uses TCL's 3D shader configured for SBS with the Video Player Render Texture in the texture slot.
Over-under 3D
A material that uses TCL's 3D shader configured for OU with the Video Player Render Texture in the texture slot.
Glowing
A material configured just like the material for the Standard screen, but the emission strength is increased to increase the glow that the world will receive. Additionally, add the 'VRC_Custom Render Behavior' VRCSDK script to this object. The rest of your world must also be properly set up with baked lighting and Light Probes.

The Video

VRC_SyncVideoPlayer supports video files that Unity natively supports. I prefer to use video files that have an MP4 container containing an H.264 video track and AAC audio track(s). I use Handbrake to convert my video files.

As far as Bitrate and other quality settings go, those all depend on the content of videos you are encoding, the server you are hosting the videos from, how many people are watching, and your taste in video quality.

For a video to work well in VRChat with VRC_SyncVideoPlayer it should:
  • Use an MP4 Container
  • Use H.264 video
  • Use AAC Audio
  • Use a constant framerate
  • Be encoded with advanced H.264 compression techniques turned off (Use 'fast decode', don't use variable framerates, etc.)
  • Be 'Web Optimized'/'Fast Start' to move the moov atom of the MP4 container to the start of the file
  • Be encoded with bitrates that work with the server you are using and the number of people watching

The Audio

The Vantage currently supports 5.1 surround sound. This is achieved through converting the audio to be six discrete AAC audio tracks. It is then played in the theater through six Audio Source components positioned where 5.1 speakers go.

My encoding process:
  • Encode the video in Handbrake with the settings I want for the video track, while setting the surround sound audio track to passthru.
  • I then take this resulting video file and run it through FFMPEG with the following command(s) to convert the surround sound audio track into six discrete audio tracks, one for each channel. (Note that this command also enables the 'Fast Start' option mentioned above).
    FFMPEG 3.4.x:
    ffmpeg -i INPUT.MKV -vcodec copy -filter_complex channelsplit=channel_layout=5.1 -acodec aac -movflags faststart OUTPUT.MP4
    FFMPEG 4.x and above:
    ffmpeg -i INPUT.MKV
           -filter_complex "channelsplit=channel_layout=5.1[FL][FR][FC][LFE][BL][BR];
                            [FL]aformat=channel_layouts=mono[FL];
                            [FR]aformat=channel_layouts=mono[FR];
                            [FC]aformat=channel_layouts=mono[FC];
                            [LFE]aformat=channel_layouts=mono[LFE];
                            [BL]aformat=channel_layouts=mono[BL];
                            [BR]aformat=channel_layouts=mono[BR]"
           -map 0:v -map '[FL]' -map '[FR]' -map '[FC]' -map '[LFE]' -map '[BL]' -map '[BR]'
           -vcodec copy -acodec aac -movflags faststart OUTPUT.MP4
           
Speaker Config

To get the Video Player to output to the right speakers, create six separate Audio Source GameObjects and place them in your theater and configure your Video Player component with them in this order:

Note: You will notice that if you play non-5.1 surround sound videos in this speaker configuration that they will play all audio through the first audio source (Right Rear). To fix this you will need to move the speaker between the front of the theater and its position in the back depending on the content you are playing. I achieve this with an animation controller and triggers that activate the animation. You could also just create two theaters with different speaker setups.

Video Hosting

Videos can, of course, be played back from YouTube and Vimeo and similar sources that are supported by VRC. But to host your own files, such as ones with the Surround Sound in them, you will need to use a web host. The key is to have it be fast, have good routing to everyone in the theater, and having it cost something you can afford.

Changing the Video

Currently, the video that plays when you press the play button can be dynamically changed in The Vantage. This is done by a server-side script. I use my own script that I wrote and host for my own use, but up until recently, I was using one created by the VRC community member Euan. Without Euan's script and his help, The Vantage would not be what it is today. Unfortunately, I can not give you access to either script.