Driving a Real Browser Inside a VRChat World (VRCBrowserBridgeWorld)

VRCBrowserBridgeWorld is an experiment in driving a real browser from inside a VRChat world. You stand in the world, click and drag at a screen, hit some keys, and on the other end a real browser follows along; the picture gets pushed back into the world. Effectively it turns VRChat into the input side of a remote desktop.

This repo is only the world end, the UdonSharp half. The external gateway that actually drives the browser and pushes the RTSP stream back lives elsewhere and has to run separately. The boundary was drawn clean from the start: the world only turns player actions into events and sends them out, and all the dirty work is thrown outside.

The core is a custom protocol. Every interaction is serialized into one line of JSON, tagged with a VRCBRIDGE prefix, and written to the log. The external gateway tails the log plus grabs the Udon sync variables, and that’s how it fishes the events out. An event looks like this:

{"v":1,"sessionId":"demo","seq":1,"playerId":"1","playerName":"ExampleUser","source":"gateway","type":"pointer_down","x":0.5,"y":0.5,"button":"left"}

The coordinates x/y are normalized UV, 0 to 1, so the outside doesn’t care how big the screen is in the world or which way it faces; it just maps them proportionally onto the real window. Event types cover pretty much everything: pointer move/down/up, scroll wheel, text, URL, key press, viewport, stream config, plus back/forward/refresh. Sending events all goes through BridgeRouter, which builds the JSON, pushes it via Udon manual sync variables, and writes the log along the way.

There’s no mouse in VRChat, so clicking the screen is done with a raycast. BridgeInputSurface casts a ray from the player’s head at that quad, turns the hit point into UV, and sends a pointer or scroll event by coordinate — wherever your head points, that’s the cursor. Pointer move is throttled, otherwise one turn of your head floods the log with a stream of move events. On top of that, BridgeCommandButton each maps to one action, and BridgeControlPanel wraps another layer to make things one-click. Video comes back over AVPro RTSP, pointing at rtsp://127.0.0.1:8554/browser by default.

Hand-building Udon scenes is too painful, so there’s a set of editor tools hanging under the menu VRC Browser Bridge/. One click on Build Bridge Console generates the materials, the console screen, and the command panel all at once. The UdonSharp program asset keeps breaking, so a Repair one-click fix was added. There are also two test items that let you verify the protocol is correct without entering the world. The project is still very WIP, based on Unity 2022.3.22f1 and UdonSharp, and no license is included in the repo.