Building a Remote Playwright Debugging Environment with Docker + VNC

Background: Why we needed custom Playwright code in the platform

Manet is an AI digital worker platform built at ByteDance. Its core capability is enabling AI agents to autonomously execute browser tasks around the clock. Initially, agent browser control relied entirely on LLM + atomic tools (click, type, navigate), which worked well for simple pages.

As more business systems came onboard, two problems emerged:

• Accuracy drops on complex pages: Multi-layered iframes and dynamically rendered React/Vue components caused element-location accuracy to fall sharply.
• High cost for fixed workflows: For fully scripted processes, routing every action through LLM inference was burning tokens continuously.

The solution was clear: introduce custom Playwright code execution, letting teams write deterministic scripts for complex or repetitive flows. But that raised a harder question — how do users debug Playwright scripts inside the platform?

Understanding Playwright's execution model

CDP: the foundation

Playwright controls browsers via CDP (Chrome DevTools Protocol) — the same WebSocket protocol underlying DevTools, Playwright, and Puppeteer. The browser launches with --remote-debugging-port exposed, and clients send JSON commands over WebSocket.

Understanding this matters because the three approaches we evaluated differ fundamentally in who holds the CDP connection and where Playwright code runs.

Why Playwright debugging is hard

Debugging Playwright is difficult due to layered complexity: AJAX-driven pages that change state unpredictably, fragile selectors broken by any DOM update, subtle timing issues around debounce and animations, and silent failures where a form appears to succeed but data never saves.

All of this means: Playwright debugging must be visual, stoppable, and show live browser state.

Three approaches we evaluated

Option 1: Local SDK

Ship an npm scaffold package. Engineers install it locally, authenticate to the platform, open Playwright UI mode, debug, then publish the script. We actually built this (pw-sdk), with generate, login, and publish commands.

Pros: Zero network latency, full local control, integrates with VS Code.
Cons: Requires Node.js setup — a hard barrier for non-engineers. Also introduces the "works on my machine" problem.

Option 2: VNC-based remote environment

Spin up a Docker container server-side. Inside: a full browser + VNC server + Playwright. Users access a split-pane page — code editor on the left, live VNC view on the right. Code changes sync to the container in real time.

Pros: Zero setup for users, consistent environment, transparent updates.
Cons: VNC latency, container lifecycle complexity, ~20 person-days to build.

Option 3: Browser extension

A Chrome extension providing an in-browser code editor connected via a local proxy. Faster to build (~12 person-days) but limited API permissions cap its capabilities.

Decision

Dimension	Local SDK	VNC Remote	Browser Extension
User barrier	High (needs env setup)	Low (browser only)	Medium (install plugin)
Debug experience	Best	Medium (some latency)	Medium
Env consistency	Poor	Best	Medium
Build cost	Medium	High (~20 days)	Medium (~12 days)

We chose Option 2 (VNC remote environment) and kept Option 1 as an advanced tool for engineers. The platform's primary users are non-engineers — zero configuration was a hard requirement.

Building the VNC debugging environment

Architecture

User browser
  ├── Code editor (Monaco)
  └── VNC view (noVNC WebSocket)
        ↕ Nginx reverse proxy
Container
  ├── VNC service (screen streaming)
  ├── Base service (Node.js)
  │     ├── /playwright/sync-code
  │     ├── /playwright/sync-url
  │     ├── /playwright/sync-cookies
  │     └── /playwright/restart
  └── Playwright controller

Real-time code sync

When a user edits a line, the change needs to reach the container immediately. The frontend watches for code changes with 500ms debounce, calls /playwright/sync-code to write the file, then the base service restarts Playwright to reload. Cookies are injected server-side via the BFF layer — never touching the frontend.

Container lifecycle

Each debug session gets its own container. The frontend sends a heartbeat every 5 minutes; the server destroys the container after 15 minutes of silence. Re-entering the debug page spins up a fresh container.

Code sandbox security

User-submitted scripts run in a server-side vm2 sandbox: restricted module access (only SDK-provided tools), execution timeouts against infinite loops, memory and CPU limits, and container-level process isolation as a second defence line.

Gotchas we hit

Code sync timing: Users would click "run" before the file was fully written, executing stale code. Fix: make sync-code synchronous — don't respond until the write is confirmed, and block the run button until the response arrives.

Nginx path collisions: Multiple containers share one Nginx instance. We assign each container a unique ID and route via path prefix: /debug/{containerId}/vnc and /debug/{containerId}/api.

Playwright UI mode in headless Docker: Playwright's --ui mode has rendering issues under Docker + Xvfb. We switched to Playwright's HTML reporter + a custom execution interface, showing raw Chromium in the VNC view.

Cookie inject timing: The cookie sync request can arrive before the container's Node.js service is ready. Fix: poll the container health endpoint; only sync cookies after a 200 OK.

Outcomes

• Non-engineers can write and debug Playwright scripts entirely self-serve.
• Fixed-flow tasks migrated to deterministic scripts — LLM token spend dropped significantly.
• Complex-page success rates (multi-iframe, dynamic rendering) went from unreliable to near-100%.

Key takeaways

1. User barrier beats technical elegance. The local SDK had better DX — but it was useless for non-engineers. Zero configuration was the real requirement.
2. Synchronous write, then run. Any "write then execute" flow must confirm the write before executing.
3. Heartbeat over timeout. Heartbeats give the server a clear liveness signal; timeouts lie.
4. Auth injection belongs server-side. Never pass credentials through the frontend, even in dev tools.
5. The two approaches coexist. Local SDK for engineers, remote VNC for everyone else.