openscreen/docs/testing/windows-native-cursor.md

Windows native cursor test pipeline

This branch includes two Windows-focused diagnostics for fast iteration on native cursor capture and rendering. They are intentionally local developer tools: they create short videos and JSON reports so cursor changes can be inspected without doing a full manual record/edit/export cycle.

Native sampler diagnostic

npm run test:cursor-native:win

This script does not launch OpenScreen. It:

• starts a Windows GetCursorInfo sampler
• moves the real OS pointer with SetCursorPos
• captures native cursor handles, hotspots, assets, and standard IDC_* cursor types
• writes normalized CursorRecordingData
• generates an abstract preview video
• generates a real-screen preview video using screenshots of the current desktop

The output directory is printed in the command result, for example:

C:\Users\<user>\AppData\Local\Temp\openscreen-cursor-native-...

Useful files:

• report.json: sample counts, asset counts, cursor handles, and generated artifact paths
• cursor-recording-data.json: sidecar-compatible cursor data
• preview.webm: abstract path/asset/hotspot preview
• real-capture-preview.webm: real desktop screenshot background with reconstructed cursor overlay
• assets/*.png: raw cursor bitmaps captured from Windows

Environment overrides:

$env:CURSOR_TEST_DURATION_MS = "3000"
$env:CURSOR_TEST_SAMPLE_INTERVAL_MS = "16"
$env:CURSOR_TEST_SCREEN_FRAME_INTERVAL_MS = "80"
$env:CURSOR_TEST_OUTPUT_DIR = "C:\temp\openscreen-cursor-test"
npm run test:cursor-native:win

OpenScreen preview capture

npm run capture:openscreen-preview

This script launches the real Electron app, injects a fixture video plus cursor sidecar data, opens the editor, captures frames from the actual OpenScreen preview UI, and encodes them into a WebM.

By default it uses the latest cursor-recording-data.json generated by npm run test:cursor-native:win. To force a specific sidecar:

$env:CURSOR_RECORDING_DATA_PATH = "C:\path\to\cursor-recording-data.json"
npm run capture:openscreen-preview

Useful environment overrides:

$env:OPENSCREEN_PREVIEW_SKIP_BUILD = "true"
$env:OPENSCREEN_PREVIEW_FRAME_COUNT = "120"
$env:OPENSCREEN_PREVIEW_FPS = "30"
$env:OPENSCREEN_PREVIEW_OUTPUT_DIR = "C:\temp\openscreen-preview"
npm run capture:openscreen-preview

Useful files:

• openscreen-preview.webm: video of the real OpenScreen editor preview
• frames/*.png: captured preview frames
• report.json: fixture paths, source sidecar, frame count, and output path

What these tests validate

Together, the scripts make it quick to inspect:

• whether Windows cursor samples are visible and continuous
• whether native hotspots stay anchored when scaling to 3x
• whether standard Windows cursors are recognized via IDC_*
• whether high-quality SVG cursor replacements follow the native hotspot
• whether the real OpenScreen preview renders the same cursor behavior as the diagnostic pipeline

They are not a full substitute for an end-to-end manual recording pass. Before shipping cursor changes, also test a real capture session and export from the packaged app.

Known Gap

Windows native cursor Click Bounce is currently backlogged. Size, Smoothing, and Motion Blur can be validated through preview/export, but Click Bounce has not shown a visible effect in packaged-app manual testing. The current diagnostic can observe synthetic click metadata, but that is not enough to validate the real OpenScreen record -> preview -> export path.

Track the open item in docs/engineering/windows-native-recorder-roadmap.md under Native Cursor Click Bounce Is Not Visibly Applied.

Native Windows capture backend

The app now routes Windows recordings through an external WGC helper instead of Electron getDisplayMedia. This is meant to remove the coordinate and clock split that made the reconstructed cursor drift in the preview/export path.

Current native availability rules:

• Windows 10 build 19041 or newer
• a helper executable is available

The helper currently implements display/window video capture, system audio loopback, default microphone capture, Media Foundation webcam capture, and DirectShow fallback for selected virtual cameras such as NVIDIA Broadcast. Webcam frames are composed into the primary MP4 as a bottom-right picture-in-picture overlay, and black webcam warmup frames are ignored until the first visible frame is available.

Build OpenScreen's helper locally:

npm run build:native:win

Smoke-test the helper directly:

npm run test:wgc-helper:win
npm run test:wgc-helper:win -- --capture-cursor
npm run test:wgc-window:win
npm run test:wgc-audio:win
npm run test:wgc-mic:win
npm run test:wgc-mixed-audio:win
npm run test:wgc-webcam:win

For local diagnostics with another compatible helper, point OpenScreen at that executable:

$env:OPENSCREEN_WGC_CAPTURE_EXE = "C:\path\to\wgc-capture.exe"
npm run build-vite
npm run dev

The helper receives one JSON config argument, emits JSON lifecycle events, prints the legacy Recording started marker, accepts stop on stdin, and prints Recording stopped. Output path: <path>. See electron/native/README.md for the exact contract and build output paths.