feat: add native Windows recorder helper

This commit is contained in:
EtienneLescot
2026-05-05 16:07:07 +02:00
parent d21e5eb34c
commit 062cf2a87c
27 changed files with 2873 additions and 139 deletions
+51 -47
View File
@@ -1,48 +1,52 @@
# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
pnpm-debug.log*
lerna-debug.log*
node_modules
dist
dist-electron
dist-ssr
*.local
.env
# Editor directories and files
.vscode/*
.zed/
!.vscode/extensions.json
.idea
.DS_Store
*.suo
*.ntvs*
*.njsproj
*.sln
*.sw?
release/**
*.kiro/
.claude/
# npx electron-builder --mac --win
# Playwright
test-results
playwright-report/
# Vitest browser mode screenshots
__screenshots__/
# shell files
/shell.sh
# Nix
result
result-*
.direnv/
#kilocode
# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
pnpm-debug.log*
lerna-debug.log*
node_modules
dist
dist-electron
dist-ssr
*.local
.env
# Native helper build outputs
/electron/native/wgc-capture/build/
/electron/native/bin/
# Editor directories and files
.vscode/*
.zed/
!.vscode/extensions.json
.idea
.DS_Store
*.suo
*.ntvs*
*.njsproj
*.sln
*.sw?
release/**
*.kiro/
.claude/
# npx electron-builder --mac --win
# Playwright
test-results
playwright-report/
# Vitest browser mode screenshots
__screenshots__/
# shell files
/shell.sh
# Nix
result
result-*
.direnv/
#kilocode
.kilo/
@@ -0,0 +1,202 @@
# Windows Native Recorder Roadmap
OpenScreen's Windows recorder should be owned by one native backend. Electron capture can remain available for non-Windows platforms and temporary developer diagnostics, but Windows production recording should not silently fall back to `getDisplayMedia` / `MediaRecorder`.
## Goals
- Capture displays and windows through Windows Graphics Capture (WGC).
- Render the native Windows cursor as OpenScreen's high-quality scalable cursor overlay.
- Capture system audio through WASAPI loopback.
- Capture microphone audio through WASAPI.
- Mix system audio and microphone audio into the primary screen recording.
- Capture webcam video natively and keep it as a separate editable OpenScreen media stream.
- Keep preview/export aligned because screen video, audio, webcam, and cursor share one native timing origin.
- Keep exported MP4s Windows-friendly: H.264 video plus AAC audio. Opus-in-MP4 is not an acceptable Windows export target.
- Package the native helper with the Windows app.
## Non-Goals
- Replacing the editor/export pipeline.
- Flattening webcam into the screen recording. The editor currently treats webcam as editable picture-in-picture media, so the native recorder should preserve a separate `webcamVideoPath`.
- Adding a native fallback for macOS or Linux in this branch.
## Target Architecture
The renderer keeps the existing recording controls. On Windows, `useScreenRecorder` sends a complete recording request to Electron and does not assemble Windows `MediaStream` tracks with `MediaRecorder`.
Electron owns the native recording session:
- resolves the selected source;
- resolves output paths;
- starts cursor sampling;
- starts the helper process;
- sends pause/resume/stop/cancel commands;
- writes `RecordingSession` manifests;
- reports explicit errors when a Windows-native capability is unavailable.
The helper owns Windows media capture:
- WGC screen/window frames;
- WASAPI system loopback;
- WASAPI microphone input;
- Media Foundation webcam capture;
- Media Foundation encoding/muxing;
- stream timestamp normalization.
## Helper Contract V2
The helper receives a single JSON argument:
```json
{
"schemaVersion": 2,
"recordingId": 1234567890,
"source": {
"type": "display",
"sourceId": "screen:0:0",
"displayId": 123,
"windowHandle": null,
"bounds": { "x": 0, "y": 0, "width": 1920, "height": 1080 }
},
"video": {
"fps": 60,
"width": 1920,
"height": 1080,
"bitrate": 18000000
},
"audio": {
"system": { "enabled": true },
"microphone": { "enabled": true, "deviceId": "default", "gain": 1.4 }
},
"webcam": {
"enabled": true,
"deviceId": "default",
"width": 1280,
"height": 720,
"fps": 30,
"bitrate": 18000000
},
"outputs": {
"screenPath": "C:\\Users\\me\\recording-123.mp4",
"webcamPath": "C:\\Users\\me\\recording-123-webcam.mp4",
"manifestPath": "C:\\Users\\me\\recording-123.session.json"
}
}
```
The helper emits newline-delimited JSON events to stdout:
```json
{ "event": "ready", "schemaVersion": 2 }
{ "event": "recording-started", "timestampMs": 1234567890 }
{ "event": "warning", "code": "audio-device-unavailable", "message": "..." }
{ "event": "recording-stopped", "screenPath": "...", "webcamPath": "..." }
{ "event": "error", "code": "unsupported-window-source", "message": "..." }
```
During migration, Electron also accepts the current textual helper messages so existing display-only smoke tests keep working.
## Implementation Phases
### 1. Native Session Boundary
- Add a structured Windows native recording request type.
- Pass source kind, audio flags, microphone device, webcam flags, and output paths into the helper.
- On Windows, do not silently fall back to Electron capture. If the helper is unavailable or a native feature is missing, show a clear error.
- Keep Electron fallback only for non-Windows and optional developer diagnostics.
Acceptance:
- Display-only recording still works.
- Enabling an unsupported native feature returns an explicit native error instead of recording through Electron.
### 2. WASAPI System Audio
Status: initial implementation landed. The helper captures the default render endpoint with WASAPI loopback, passes the runtime mix format into `MFEncoder`, and muxes AAC audio into the primary MP4. Long-run drift correction and explicit silence insertion remain follow-up hardening work.
- Add `WasapiLoopbackCapture`.
- Capture the default render endpoint in shared loopback mode.
- Keep `WasapiLoopbackCapture` responsible only for device activation, packet capture, and packet timestamps.
- Keep `MFEncoder` responsible for all Media Foundation stream definitions and muxing.
- Feed the endpoint mix format into `MFEncoder` as the single source of truth for audio stream shape: sample rate, channel count, bits per sample, block alignment, average bytes/sec, and subtype (`PCM` or `Float`).
- Encode the primary screen MP4 with H.264 video and AAC audio through one `IMFSinkWriter`.
- Timestamp audio from the captured frame count in 100ns units. The first implementation uses the WASAPI packet timeline; later drift correction will add explicit silence or resampling if long recordings show measurable clock skew.
- Treat microphone mixing as a later phase. System loopback must land first without introducing renderer-side audio code.
Acceptance:
- Screen MP4 has an AAC audio track when system audio is enabled.
- A 5-minute recording has audio/video duration drift below one frame.
SSOT rules for this phase:
- `src/lib/nativeWindowsRecording.ts` is the renderer/main TypeScript request contract.
- `docs/engineering/windows-native-recorder-roadmap.md` is the feature-level contract and phase checklist.
- `WgcSession::captureWidth()/captureHeight()` is the encoded screen frame size until a dedicated native scaling stage exists.
- `WasapiLoopbackCapture::inputFormat()` is the runtime audio format source used by `MFEncoder`.
- No duplicated hard-coded audio format assumptions in `main.cpp`.
### 3. WASAPI Microphone
- Add microphone device enumeration and stable device-id mapping.
- Capture selected/default microphone through WASAPI.
- Apply OpenScreen's current mic gain policy.
- Mix microphone and system audio before AAC encoding.
Acceptance:
- Mic-only, system-only, and mixed audio recordings produce a valid AAC track.
- Device unplug/permission failure produces an explicit error or warning.
### 4. Webcam Capture
- Add Media Foundation webcam source reader.
- Select 1280x720/30fps or nearest supported format.
- Encode webcam to `recording-<id>-webcam.mp4`.
- Synchronize webcam timestamps to the native session clock.
- Store `webcamVideoPath` in the OpenScreen session manifest.
Acceptance:
- Editor loads the native screen recording and the native webcam recording.
- Webcam layout controls behave the same as today.
### 5. Native Window Capture
- Resolve Electron `window:*` selections to an `HWND`.
- Use WGC `CreateForWindow(HWND)`.
- Handle window close, minimize, resize, DPI scaling, and monitor moves.
- Return clear errors for unsupported protected windows.
Acceptance:
- Capturing a normal app window works with cursor/audio/mic/webcam.
- Window resize and movement do not corrupt the recording.
### 6. Runtime Controls
- Add pause/resume commands to the helper.
- Add cancel command that removes partial screen/webcam outputs.
- Keep restart as stop-discard-start from Electron until the helper supports a native restart event.
Acceptance:
- Pause/resume keeps preview duration coherent.
- Cancel leaves no stale media/session/cursor files.
### 7. Test Pipeline
- `npm run test:wgc-helper:win`: display-only helper smoke test.
- `npm run test:wgc-audio:win`: validates AAC track presence and duration.
- `npm run test:wgc-window:win`: captures a fixture window by HWND.
- `npm run test:wgc-webcam:win`: validates webcam output when a webcam is available, otherwise skips explicitly.
- Packaging check: confirms the helper is in `app.asar.unpacked`.
- Export check: exported MP4s generated from native recordings keep an AAC audio track when the source has audio.
## Ship Criteria
- Windows display capture works with cursor, system audio, microphone, and webcam.
- Windows window capture works with cursor, system audio, microphone, and webcam.
- Preview and export show no cursor position drift.
- Preview and export show no measurable audio/video/webcam drift.
- Windows production builds do not depend on Electron capture fallback.
+34
View File
@@ -83,3 +83,37 @@ Together, the scripts make it quick to inspect:
- whether the real OpenScreen preview renders the same cursor behavior as the diagnostic pipeline
They are not a full substitute for an end-to-end manual recording pass. Before shipping cursor changes, also test a real capture session and export from the packaged app.
## Native Windows capture backend
The app now routes Windows recordings through an external WGC helper instead of Electron `getDisplayMedia`. This is meant to remove the coordinate and clock split that made the reconstructed cursor drift in the preview/export path.
Current native availability rules:
- Windows 10 build 19041 or newer
- a helper executable is available
The helper currently implements display video capture and system audio loopback. Window capture, microphone audio, and webcam capture are part of the native recorder roadmap and fail explicitly instead of silently falling back to Electron capture on Windows.
Build OpenScreen's helper locally:
```powershell
npm run build:native:win
```
Smoke-test the helper directly:
```powershell
npm run test:wgc-helper:win
npm run test:wgc-audio:win
```
For local diagnostics with another compatible helper, point OpenScreen at that executable:
```powershell
$env:OPENSCREEN_WGC_CAPTURE_EXE = "C:\path\to\wgc-capture.exe"
npm run build-vite
npm run dev
```
The helper receives one JSON config argument, emits JSON lifecycle events, prints the legacy `Recording started` marker, accepts `stop` on stdin, and prints `Recording stopped. Output path: <path>`. See `electron/native/README.md` for the exact contract and build output paths.
+10 -8
View File
@@ -4,10 +4,11 @@
"appId": "com.siddharthvaddem.openscreen",
"asar": true,
// .node binaries can't be dlopen'd from inside an asar — must live unpacked.
"asarUnpack": [
"node_modules/uiohook-napi/**/*",
"**/*.node"
],
"asarUnpack": [
"node_modules/uiohook-napi/**/*",
"**/*.node",
"electron/native/bin/**"
],
"productName": "Openscreen",
"npmRebuild": true,
"buildDependenciesFromSource": true,
@@ -15,10 +16,11 @@
"directories": {
"output": "release/${version}"
},
"files": [
"dist",
"dist-electron",
"!*.png",
"files": [
"dist",
"dist-electron",
"electron/native/bin/**/*",
"!*.png",
"!preview*.png",
"!*.md",
"!README.md",
+18
View File
@@ -72,6 +72,24 @@ interface Window {
error?: string;
}>;
setRecordingState: (recording: boolean, recordingId?: number) => Promise<void>;
isNativeWindowsCaptureAvailable: () => Promise<{
success: boolean;
available: boolean;
helperPath?: string;
reason?: string;
error?: string;
}>;
startNativeWindowsRecording: (
request: import("../src/lib/nativeWindowsRecording").NativeWindowsRecordingRequest,
) => Promise<import("../src/lib/nativeWindowsRecording").NativeWindowsRecordingStartResult>;
stopNativeWindowsRecording: (discard?: boolean) => Promise<{
success: boolean;
path?: string;
session?: import("../src/lib/recordingSession").RecordingSession;
message?: string;
discarded?: boolean;
error?: string;
}>;
discardCursorTelemetry: (recordingId: number) => Promise<void>;
getCursorTelemetry: (videoPath?: string) => Promise<{
success: boolean;
+500 -43
View File
@@ -1,3 +1,5 @@
import { type ChildProcessWithoutNullStreams, spawn } from "node:child_process";
import { constants as fsConstants } from "node:fs";
import fs from "node:fs/promises";
import { createRequire } from "node:module";
import os from "node:os";
@@ -14,6 +16,7 @@ import {
shell,
systemPreferences,
} from "electron";
import type { NativeWindowsRecordingRequest } from "../../src/lib/nativeWindowsRecording";
import {
normalizeProjectMedia,
normalizeRecordingSession,
@@ -36,6 +39,7 @@ import { registerNativeBridgeHandlers } from "./nativeBridge";
const PROJECT_FILE_EXTENSION = "openscreen";
const SHORTCUTS_FILE = path.join(app.getPath("userData"), "shortcuts.json");
const RECORDING_FILE_PREFIX = "recording-";
const RECORDING_SESSION_SUFFIX = ".session.json";
const ALLOWED_IMPORT_VIDEO_EXTENSIONS = new Set([".webm", ".mp4", ".mov", ".avi", ".mkv"]);
@@ -250,6 +254,12 @@ const MAX_CURSOR_SAMPLES = 60 * 60 * 30; // 1 hour @ 30Hz
let cursorRecordingSession: CursorRecordingSession | null = null;
let pendingCursorRecordingData: CursorRecordingData | null = null;
let nativeWindowsCaptureProcess: ChildProcessWithoutNullStreams | null = null;
let nativeWindowsCaptureOutput = "";
let nativeWindowsCaptureTargetPath: string | null = null;
let nativeWindowsCaptureWebcamTargetPath: string | null = null;
let nativeWindowsCaptureRecordingId: number | null = null;
let nativeWindowsCursorOffsetMs = 0;
function normalizeCursorSample(sample: unknown): CursorRecordingSample | null {
if (!sample || typeof sample !== "object") {
@@ -404,6 +414,215 @@ function getSelectedSourceId() {
return typeof selectedSource?.id === "string" ? selectedSource.id : null;
}
function getSelectedDisplay() {
const sourceDisplayId = Number(selectedSource?.display_id);
if (!Number.isFinite(sourceDisplayId)) {
return null;
}
return screen.getAllDisplays().find((display) => display.id === sourceDisplayId) ?? null;
}
function resolveUnpackedAppPath(...segments: string[]) {
const resolved = path.join(app.getAppPath(), ...segments);
if (app.isPackaged) {
return resolved.replace(/\.asar([/\\])/, ".asar.unpacked$1");
}
return resolved;
}
function getNativeWindowsCaptureHelperCandidates() {
const envPath = process.env.OPENSCREEN_WGC_CAPTURE_EXE?.trim();
const archTag = process.arch === "arm64" ? "win32-arm64" : "win32-x64";
return [
envPath,
resolveUnpackedAppPath(
"electron",
"native",
"wgc-capture",
"build",
"Release",
"wgc-capture.exe",
),
resolveUnpackedAppPath("electron", "native", "wgc-capture", "build", "wgc-capture.exe"),
resolveUnpackedAppPath("electron", "native", "bin", archTag, "wgc-capture.exe"),
].filter((candidate): candidate is string => Boolean(candidate));
}
async function findNativeWindowsCaptureHelperPath() {
if (process.platform !== "win32") {
return null;
}
for (const candidate of getNativeWindowsCaptureHelperCandidates()) {
try {
await fs.access(candidate, fsConstants.X_OK);
return candidate;
} catch {
// Try the next configured helper location.
}
}
return null;
}
function isWindowsGraphicsCaptureOsSupported() {
if (process.platform !== "win32") {
return false;
}
const [, , build] = process.getSystemVersion().split(".").map(Number);
return Number.isFinite(build) && build >= 19041;
}
async function startCursorRecording(recordingId?: number) {
if (cursorRecordingSession) {
pendingCursorRecordingData = await cursorRecordingSession.stop();
cursorRecordingSession = null;
}
pendingCursorRecordingData = null;
cursorRecordingSession = createCursorRecordingSession({
getDisplayBounds: getSelectedSourceBounds,
maxSamples: MAX_CURSOR_SAMPLES,
platform: process.platform,
sampleIntervalMs: CURSOR_SAMPLE_INTERVAL_MS,
sourceId: getSelectedSourceId(),
startTimeMs:
typeof recordingId === "number" && Number.isFinite(recordingId) ? recordingId : undefined,
});
try {
await cursorRecordingSession.start();
} catch (error) {
console.error("Failed to start cursor recording session:", error);
cursorRecordingSession = null;
}
}
async function stopCursorRecording() {
if (!cursorRecordingSession) {
return;
}
try {
pendingCursorRecordingData = await cursorRecordingSession.stop();
} catch (error) {
console.error("Failed to stop cursor recording session:", error);
pendingCursorRecordingData = null;
} finally {
cursorRecordingSession = null;
}
}
async function writePendingCursorTelemetry(videoPath: string) {
const telemetryPath = `${videoPath}.cursor.json`;
if (pendingCursorRecordingData && pendingCursorRecordingData.samples.length > 0) {
await fs.writeFile(telemetryPath, JSON.stringify(pendingCursorRecordingData, null, 2), "utf-8");
}
pendingCursorRecordingData = null;
}
function shiftPendingCursorTelemetry(offsetMs: number) {
if (!pendingCursorRecordingData || !Number.isFinite(offsetMs) || offsetMs <= 0) {
return;
}
pendingCursorRecordingData = {
...pendingCursorRecordingData,
samples: pendingCursorRecordingData.samples
.map((sample) => ({
...sample,
timeMs: Math.max(0, sample.timeMs - offsetMs),
}))
.sort((a, b) => a.timeMs - b.timeMs),
};
}
function waitForNativeWindowsCaptureStart(proc: ChildProcessWithoutNullStreams) {
return new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => {
cleanup();
reject(new Error("Timed out waiting for native Windows capture to start"));
}, 12000);
const onOutput = (chunk: Buffer) => {
nativeWindowsCaptureOutput += chunk.toString();
if (nativeWindowsCaptureOutput.includes("Recording started")) {
cleanup();
resolve();
}
};
const onError = (error: Error) => {
cleanup();
reject(error);
};
const onExit = (code: number | null) => {
cleanup();
reject(
new Error(
nativeWindowsCaptureOutput.trim() ||
`Native Windows capture exited before recording started (code=${code ?? "unknown"})`,
),
);
};
const cleanup = () => {
clearTimeout(timer);
proc.stdout.off("data", onOutput);
proc.stderr.off("data", onOutput);
proc.off("error", onError);
proc.off("exit", onExit);
};
proc.stdout.on("data", onOutput);
proc.stderr.on("data", onOutput);
proc.once("error", onError);
proc.once("exit", onExit);
});
}
function waitForNativeWindowsCaptureStop(proc: ChildProcessWithoutNullStreams) {
return new Promise<string>((resolve, reject) => {
const onOutput = (chunk: Buffer) => {
nativeWindowsCaptureOutput += chunk.toString();
};
const onClose = (code: number | null) => {
cleanup();
const match = nativeWindowsCaptureOutput.match(/Recording stopped\. Output path: (.+)/);
if (match?.[1]) {
resolve(match[1].trim());
return;
}
if (code === 0 && nativeWindowsCaptureTargetPath) {
resolve(nativeWindowsCaptureTargetPath);
return;
}
reject(
new Error(
nativeWindowsCaptureOutput.trim() ||
`Native Windows capture exited with code=${code ?? "unknown"}`,
),
);
};
const onError = (error: Error) => {
cleanup();
reject(error);
};
const cleanup = () => {
proc.stdout.off("data", onOutput);
proc.stderr.off("data", onOutput);
proc.off("close", onClose);
proc.off("error", onError);
};
proc.stdout.on("data", onOutput);
proc.stderr.on("data", onOutput);
proc.once("close", onClose);
proc.once("error", onError);
});
}
function setCurrentRecordingSessionState(session: RecordingSession | null) {
currentRecordingSession = session;
currentVideoPath = session?.screenVideoPath ?? null;
@@ -412,10 +631,10 @@ function setCurrentRecordingSessionState(session: RecordingSession | null) {
export function registerIpcHandlers(
createEditorWindow: () => void,
createSourceSelectorWindow: () => BrowserWindow,
_createCountdownOverlayWindow: () => BrowserWindow,
createCountdownOverlayWindow: () => BrowserWindow,
getMainWindow: () => BrowserWindow | null,
getSourceSelectorWindow: () => BrowserWindow | null,
_getCountdownOverlayWindow?: () => BrowserWindow | null,
getCountdownOverlayWindow?: () => BrowserWindow | null,
onRecordingStateChange?: (recording: boolean, sourceName: string) => void,
_switchToHud?: () => void,
) {
@@ -553,6 +772,282 @@ export function registerIpcHandlers(
createEditorWindow();
});
ipcMain.handle("countdown-overlay-show", async (_, value: number, runId: number) => {
const overlayWindow = getCountdownOverlayWindow?.() ?? createCountdownOverlayWindow();
if (overlayWindow.isDestroyed()) {
return;
}
if (!overlayWindow.isVisible()) {
overlayWindow.showInactive();
}
if (overlayWindow.webContents.isLoading()) {
await new Promise<void>((resolve) => {
overlayWindow.webContents.once("did-finish-load", () => resolve());
});
}
overlayWindow.webContents.send("countdown-overlay-value", value, runId);
});
ipcMain.handle("countdown-overlay-set-value", (_, value: number, runId: number) => {
const overlayWindow = getCountdownOverlayWindow?.();
if (!overlayWindow || overlayWindow.isDestroyed()) {
return;
}
overlayWindow.webContents.send("countdown-overlay-value", value, runId);
});
ipcMain.handle("countdown-overlay-hide", (_, runId: number) => {
const overlayWindow = getCountdownOverlayWindow?.();
if (!overlayWindow || overlayWindow.isDestroyed()) {
return;
}
overlayWindow.webContents.send("countdown-overlay-value", null, runId);
overlayWindow.hide();
});
ipcMain.handle("is-native-windows-capture-available", async () => {
if (!isWindowsGraphicsCaptureOsSupported()) {
return { success: true, available: false, reason: "unsupported-os" };
}
const helperPath = await findNativeWindowsCaptureHelperPath();
return helperPath
? { success: true, available: true, helperPath }
: { success: true, available: false, reason: "missing-helper" };
});
ipcMain.handle(
"start-native-windows-recording",
async (_, request: NativeWindowsRecordingRequest) => {
try {
if (!isWindowsGraphicsCaptureOsSupported()) {
return {
success: false,
error: "Windows Graphics Capture requires Windows 10 build 19041 or newer.",
};
}
if (nativeWindowsCaptureProcess) {
return { success: false, error: "Native Windows capture is already running." };
}
const helperPath = await findNativeWindowsCaptureHelperPath();
if (!helperPath) {
return { success: false, error: "Native Windows capture helper is not available." };
}
if (!request?.source?.sourceId) {
return {
success: false,
error: "Native Windows capture request is missing a source.",
};
}
const recordingId =
typeof request.recordingId === "number" && Number.isFinite(request.recordingId)
? request.recordingId
: Date.now();
const outputPath = path.join(RECORDINGS_DIR, `${RECORDING_FILE_PREFIX}${recordingId}.mp4`);
const webcamOutputPath = path.join(
RECORDINGS_DIR,
`${RECORDING_FILE_PREFIX}${recordingId}-webcam.mp4`,
);
const sourceDisplay =
request.source.type === "display" && typeof request.source.displayId === "number"
? (screen.getAllDisplays().find((display) => display.id === request.source.displayId) ??
null)
: getSelectedDisplay();
const bounds = sourceDisplay?.bounds ?? getSelectedSourceBounds();
const displayId =
typeof request.source.displayId === "number" && Number.isFinite(request.source.displayId)
? request.source.displayId
: Number(selectedSource?.display_id);
const config = {
schemaVersion: 2,
recordingId,
outputPath,
sourceType: request.source.type,
sourceId: request.source.sourceId,
displayId: Number.isFinite(displayId) ? displayId : 0,
windowHandle: request.source.windowHandle ?? null,
fps: request.video.fps,
videoWidth: request.video.width,
videoHeight: request.video.height,
displayX: bounds.x,
displayY: bounds.y,
displayW: bounds.width,
displayH: bounds.height,
hasDisplayBounds: true,
captureSystemAudio: request.audio.system.enabled,
captureMic: request.audio.microphone.enabled,
microphoneDeviceId: request.audio.microphone.deviceId ?? null,
microphoneGain: request.audio.microphone.gain,
webcamEnabled: request.webcam.enabled,
webcamDeviceId: request.webcam.deviceId ?? null,
webcamWidth: request.webcam.width,
webcamHeight: request.webcam.height,
webcamFps: request.webcam.fps,
outputs: {
screenPath: outputPath,
webcamPath: webcamOutputPath,
},
source: {
type: request.source.type,
sourceId: request.source.sourceId,
displayId: Number.isFinite(displayId) ? displayId : null,
windowHandle: request.source.windowHandle ?? null,
bounds,
},
video: request.video,
audio: request.audio,
webcam: request.webcam,
};
console.info("[native-wgc] starting Windows capture", {
helperPath,
source: request.source,
audio: request.audio,
webcam: request.webcam,
bounds,
sourceId: selectedSource?.id ?? null,
usedDisplayMatch: Boolean(sourceDisplay),
outputPath,
});
await fs.mkdir(RECORDINGS_DIR, { recursive: true });
nativeWindowsCaptureOutput = "";
nativeWindowsCaptureTargetPath = outputPath;
nativeWindowsCaptureWebcamTargetPath = request.webcam.enabled ? webcamOutputPath : null;
nativeWindowsCaptureRecordingId = recordingId;
nativeWindowsCursorOffsetMs = 0;
const cursorStartTimeMs = Date.now();
await startCursorRecording(cursorStartTimeMs);
console.info("[native-wgc] cursor sampler ready", {
cursorStartTimeMs,
warmupMs: Date.now() - cursorStartTimeMs,
});
const proc = spawn(helperPath, [JSON.stringify(config)], {
cwd: RECORDINGS_DIR,
stdio: ["pipe", "pipe", "pipe"],
windowsHide: true,
});
nativeWindowsCaptureProcess = proc;
await waitForNativeWindowsCaptureStart(proc);
const captureStartedAtMs = Date.now();
nativeWindowsCursorOffsetMs = Math.max(0, captureStartedAtMs - cursorStartTimeMs);
console.info("[native-wgc] capture started", {
captureStartedAtMs,
cursorOffsetMs: nativeWindowsCursorOffsetMs,
});
const source = selectedSource || { name: "Screen" };
if (onRecordingStateChange) {
onRecordingStateChange(true, source.name);
}
return {
success: true,
recordingId,
path: outputPath,
helperPath,
};
} catch (error) {
console.error("Failed to start native Windows recording:", error);
nativeWindowsCaptureProcess?.kill();
nativeWindowsCaptureProcess = null;
nativeWindowsCaptureTargetPath = null;
nativeWindowsCaptureWebcamTargetPath = null;
nativeWindowsCaptureRecordingId = null;
nativeWindowsCursorOffsetMs = 0;
await stopCursorRecording();
return { success: false, error: String(error) };
}
},
);
ipcMain.handle("stop-native-windows-recording", async (_, discard?: boolean) => {
const proc = nativeWindowsCaptureProcess;
const preferredPath = nativeWindowsCaptureTargetPath;
const preferredWebcamPath = nativeWindowsCaptureWebcamTargetPath;
const recordingId = nativeWindowsCaptureRecordingId ?? Date.now();
if (!proc) {
return { success: false, error: "Native Windows capture is not running." };
}
try {
const stoppedPathPromise = waitForNativeWindowsCaptureStop(proc);
proc.stdin.write("stop\n");
const stoppedPath = await stoppedPathPromise;
const screenVideoPath = stoppedPath || preferredPath;
if (!screenVideoPath) {
throw new Error("Native Windows capture did not return an output path.");
}
await stopCursorRecording();
if (discard) {
pendingCursorRecordingData = null;
await Promise.all([
fs.rm(screenVideoPath, { force: true }),
preferredWebcamPath ? fs.rm(preferredWebcamPath, { force: true }) : Promise.resolve(),
fs.rm(`${screenVideoPath}.cursor.json`, { force: true }),
]);
return { success: true, discarded: true };
}
shiftPendingCursorTelemetry(nativeWindowsCursorOffsetMs);
await writePendingCursorTelemetry(screenVideoPath);
let webcamVideoPath: string | undefined;
if (preferredWebcamPath) {
try {
await fs.access(preferredWebcamPath, fsConstants.R_OK);
webcamVideoPath = preferredWebcamPath;
} catch {
webcamVideoPath = undefined;
}
}
const session: RecordingSession = webcamVideoPath
? { screenVideoPath, webcamVideoPath, createdAt: recordingId }
: { screenVideoPath, createdAt: recordingId };
setCurrentRecordingSessionState(session);
currentProjectPath = null;
const sessionManifestPath = path.join(
RECORDINGS_DIR,
`${path.parse(screenVideoPath).name}${RECORDING_SESSION_SUFFIX}`,
);
await fs.writeFile(sessionManifestPath, JSON.stringify(session, null, 2), "utf-8");
return {
success: true,
path: screenVideoPath,
session,
message: "Native Windows recording session stored successfully",
};
} catch (error) {
console.error("Failed to stop native Windows recording:", error);
await stopCursorRecording();
return { success: false, error: String(error) };
} finally {
nativeWindowsCaptureProcess = null;
nativeWindowsCaptureTargetPath = null;
nativeWindowsCaptureWebcamTargetPath = null;
nativeWindowsCaptureRecordingId = null;
nativeWindowsCursorOffsetMs = 0;
const source = selectedSource || { name: "Screen" };
if (onRecordingStateChange) {
onRecordingStateChange(false, source.name);
}
}
});
ipcMain.handle("store-recorded-session", async (_, payload: StoreRecordedSessionInput) => {
try {
return await storeRecordedSessionFiles(payload);
@@ -586,15 +1081,7 @@ export function registerIpcHandlers(
setCurrentRecordingSessionState(session);
currentProjectPath = null;
const telemetryPath = `${screenVideoPath}.cursor.json`;
if (pendingCursorRecordingData && pendingCursorRecordingData.samples.length > 0) {
await fs.writeFile(
telemetryPath,
JSON.stringify(pendingCursorRecordingData, null, 2),
"utf-8",
);
}
pendingCursorRecordingData = null;
await writePendingCursorTelemetry(screenVideoPath);
const sessionManifestPath = path.join(
RECORDINGS_DIR,
@@ -653,39 +1140,9 @@ export function registerIpcHandlers(
ipcMain.handle("set-recording-state", async (_, recording: boolean, recordingId?: number) => {
if (recording) {
if (cursorRecordingSession) {
pendingCursorRecordingData = await cursorRecordingSession.stop();
cursorRecordingSession = null;
}
pendingCursorRecordingData = null;
cursorRecordingSession = createCursorRecordingSession({
getDisplayBounds: getSelectedSourceBounds,
maxSamples: MAX_CURSOR_SAMPLES,
platform: process.platform,
sampleIntervalMs: CURSOR_SAMPLE_INTERVAL_MS,
sourceId: getSelectedSourceId(),
startTimeMs:
typeof recordingId === "number" && Number.isFinite(recordingId) ? recordingId : undefined,
});
try {
await cursorRecordingSession.start();
} catch (error) {
console.error("Failed to start cursor recording session:", error);
cursorRecordingSession = null;
}
await startCursorRecording(recordingId);
} else {
if (cursorRecordingSession) {
try {
pendingCursorRecordingData = await cursorRecordingSession.stop();
} catch (error) {
console.error("Failed to stop cursor recording session:", error);
pendingCursorRecordingData = null;
} finally {
cursorRecordingSession = null;
}
}
await stopCursorRecording();
}
const source = selectedSource || { name: "Screen" };
+50
View File
@@ -0,0 +1,50 @@
# Native capture helpers
Windows native recording is resolved from one of these locations:
1. `OPENSCREEN_WGC_CAPTURE_EXE`, for local development and diagnostics.
2. `electron/native/wgc-capture/build/wgc-capture.exe`, for a locally built Ninja helper.
3. `electron/native/wgc-capture/build/Release/wgc-capture.exe`, for a locally built multi-config helper.
4. `electron/native/bin/win32-x64/wgc-capture.exe` or `electron/native/bin/win32-arm64/wgc-capture.exe`, for packaged prebuilt helpers.
Build the Windows helper with:
```powershell
npm run build:native:win
```
The build writes the CMake output to `electron/native/wgc-capture/build/wgc-capture.exe` and copies the redistributable binary to `electron/native/bin/win32-x64/wgc-capture.exe`.
The helper contract is process-based: the app starts the process with one JSON argument and sends commands on stdin. `stop\n` finalizes the recording. During migration the helper prints both newline-delimited JSON events and the legacy text messages `Recording started` / `Recording stopped. Output path: <path>`.
Current V2 JSON shape:
```json
{
"schemaVersion": 2,
"recordingId": 123,
"sourceType": "display",
"sourceId": "screen:0:0",
"displayId": 1,
"outputPath": "C:\\path\\recording-123.mp4",
"videoWidth": 1920,
"videoHeight": 1080,
"fps": 60,
"captureSystemAudio": false,
"captureMic": false,
"webcamEnabled": false,
"outputs": {
"screenPath": "C:\\path\\recording-123.mp4",
"webcamPath": "C:\\path\\recording-123-webcam.mp4"
}
}
```
The current helper implementation supports display video capture and system audio loopback. Microphone, webcam, and window capture now fail explicitly in the helper rather than silently falling back to Electron capture on Windows. See `docs/engineering/windows-native-recorder-roadmap.md` for the phased implementation plan.
Smoke-test the helper with:
```powershell
npm run test:wgc-helper:win
npm run test:wgc-audio:win
```
@@ -0,0 +1,45 @@
cmake_minimum_required(VERSION 3.20)
# The local Windows SDK image used by some contributors can miss gdi32.lib,
# while CMake's default MSVC console template links it unconditionally. This
# helper does not use GDI, so keep the standard library set minimal and explicit.
set(CMAKE_CXX_STANDARD_LIBRARIES
"kernel32.lib user32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib"
CACHE STRING "" FORCE)
project(openscreen-wgc-capture LANGUAGES CXX)
set(CMAKE_CXX_STANDARD 20)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_CXX_EXTENSIONS OFF)
add_executable(wgc-capture
src/main.cpp
src/mf_encoder.cpp
src/mf_encoder.h
src/monitor_utils.cpp
src/monitor_utils.h
src/wasapi_loopback_capture.cpp
src/wasapi_loopback_capture.h
src/wgc_session.cpp
src/wgc_session.h
)
target_compile_definitions(wgc-capture PRIVATE
NOMINMAX
WIN32_LEAN_AND_MEAN
_WIN32_WINNT=0x0A00
)
target_compile_options(wgc-capture PRIVATE /EHsc /W4 /utf-8)
target_link_libraries(wgc-capture PRIVATE
d3d11
dxgi
mf
mfplat
mfreadwrite
mfuuid
runtimeobject
windowsapp
)
+433
View File
@@ -0,0 +1,433 @@
#include "mf_encoder.h"
#include "monitor_utils.h"
#include "wasapi_loopback_capture.h"
#include "wgc_session.h"
#include <winrt/Windows.Foundation.h>
#include <algorithm>
#include <atomic>
#include <chrono>
#include <condition_variable>
#include <cctype>
#include <cstdint>
#include <iostream>
#include <mutex>
#include <string>
#include <thread>
namespace {
struct CaptureConfig {
int schemaVersion = 1;
int64_t displayId = 0;
int64_t recordingId = 0;
std::string sourceType = "display";
std::string sourceId;
std::string windowHandle;
std::string outputPath;
int fps = 60;
int width = 0;
int height = 0;
MonitorBounds bounds{};
bool hasDisplayBounds = false;
bool captureSystemAudio = false;
bool captureMic = false;
bool webcamEnabled = false;
std::string microphoneDeviceId;
double microphoneGain = 1.0;
std::string webcamDeviceId;
int webcamWidth = 0;
int webcamHeight = 0;
int webcamFps = 0;
};
std::wstring utf8ToWide(const std::string& value) {
if (value.empty()) {
return {};
}
const int size = MultiByteToWideChar(CP_UTF8, 0, value.data(), static_cast<int>(value.size()), nullptr, 0);
std::wstring result(static_cast<size_t>(size), L'\0');
MultiByteToWideChar(CP_UTF8, 0, value.data(), static_cast<int>(value.size()), result.data(), size);
return result;
}
std::string jsonEscape(const std::string& value) {
std::string result;
result.reserve(value.size());
for (const char c : value) {
switch (c) {
case '\\':
result += "\\\\";
break;
case '"':
result += "\\\"";
break;
case '\n':
result += "\\n";
break;
case '\r':
result += "\\r";
break;
case '\t':
result += "\\t";
break;
default:
result.push_back(c);
break;
}
}
return result;
}
bool findBool(const std::string& json, const std::string& key, bool fallback) {
auto pos = json.find("\"" + key + "\"");
if (pos == std::string::npos) {
return fallback;
}
pos = json.find(':', pos);
if (pos == std::string::npos) {
return fallback;
}
pos += 1;
while (pos < json.size() && std::isspace(static_cast<unsigned char>(json[pos]))) {
pos += 1;
}
if (json.compare(pos, 4, "true") == 0) {
return true;
}
if (json.compare(pos, 5, "false") == 0) {
return false;
}
return fallback;
}
int64_t findInt64(const std::string& json, const std::string& key, int64_t fallback) {
auto pos = json.find("\"" + key + "\"");
if (pos == std::string::npos) {
return fallback;
}
pos = json.find(':', pos);
if (pos == std::string::npos) {
return fallback;
}
pos += 1;
while (pos < json.size() && std::isspace(static_cast<unsigned char>(json[pos]))) {
pos += 1;
}
try {
return std::stoll(json.substr(pos));
} catch (...) {
return fallback;
}
}
int findInt(const std::string& json, const std::string& key, int fallback) {
return static_cast<int>(findInt64(json, key, fallback));
}
double findDouble(const std::string& json, const std::string& key, double fallback) {
auto pos = json.find("\"" + key + "\"");
if (pos == std::string::npos) {
return fallback;
}
pos = json.find(':', pos);
if (pos == std::string::npos) {
return fallback;
}
pos += 1;
while (pos < json.size() && std::isspace(static_cast<unsigned char>(json[pos]))) {
pos += 1;
}
try {
return std::stod(json.substr(pos));
} catch (...) {
return fallback;
}
}
std::string findString(const std::string& json, const std::string& key) {
auto pos = json.find("\"" + key + "\"");
if (pos == std::string::npos) {
return {};
}
pos = json.find(':', pos);
if (pos == std::string::npos) {
return {};
}
pos += 1;
while (pos < json.size() && std::isspace(static_cast<unsigned char>(json[pos]))) {
pos += 1;
}
if (pos >= json.size() || json[pos] != '"') {
return {};
}
pos += 1;
std::string result;
while (pos < json.size()) {
const char c = json[pos++];
if (c == '"') {
break;
}
if (c == '\\' && pos < json.size()) {
const char escaped = json[pos++];
switch (escaped) {
case '\\':
case '"':
case '/':
result.push_back(escaped);
break;
case 'n':
result.push_back('\n');
break;
case 'r':
result.push_back('\r');
break;
case 't':
result.push_back('\t');
break;
default:
result.push_back(escaped);
break;
}
continue;
}
result.push_back(c);
}
return result;
}
bool parseConfig(const std::string& json, CaptureConfig& config) {
config.schemaVersion = findInt(json, "schemaVersion", 1);
config.outputPath = findString(json, "screenPath");
if (config.outputPath.empty()) {
config.outputPath = findString(json, "outputPath");
}
if (config.outputPath.empty()) {
return false;
}
config.recordingId = findInt64(json, "recordingId", 0);
config.sourceType = findString(json, "sourceType");
if (config.sourceType.empty()) {
config.sourceType = "display";
}
config.sourceId = findString(json, "sourceId");
config.windowHandle = findString(json, "windowHandle");
config.displayId = findInt64(json, "displayId", 0);
config.fps = std::clamp(findInt(json, "fps", 60), 1, 120);
config.width = findInt(json, "videoWidth", findInt(json, "width", 0));
config.height = findInt(json, "videoHeight", findInt(json, "height", 0));
config.bounds.x = findInt(json, "displayX", 0);
config.bounds.y = findInt(json, "displayY", 0);
config.bounds.width = findInt(json, "displayW", 0);
config.bounds.height = findInt(json, "displayH", 0);
config.hasDisplayBounds = findBool(json, "hasDisplayBounds", false);
config.captureSystemAudio = findBool(json, "captureSystemAudio", false);
config.captureMic = findBool(json, "captureMic", false);
config.webcamEnabled = findBool(json, "webcamEnabled", false);
config.microphoneDeviceId = findString(json, "microphoneDeviceId");
config.microphoneGain = findDouble(json, "microphoneGain", 1.0);
config.webcamDeviceId = findString(json, "webcamDeviceId");
config.webcamWidth = findInt(json, "webcamWidth", 0);
config.webcamHeight = findInt(json, "webcamHeight", 0);
config.webcamFps = findInt(json, "webcamFps", 0);
return true;
}
void readStopCommands(std::atomic<bool>& stopRequested, std::condition_variable& cv) {
std::string line;
while (std::getline(std::cin, line)) {
if (line == "stop" || line == "q" || line == "quit") {
stopRequested = true;
cv.notify_all();
return;
}
}
stopRequested = true;
cv.notify_all();
}
} // namespace
int main(int argc, char* argv[]) {
if (argc < 2) {
std::cerr << "ERROR: Missing JSON config argument" << std::endl;
return 1;
}
winrt::init_apartment(winrt::apartment_type::multi_threaded);
CaptureConfig config;
if (!parseConfig(argv[1], config)) {
std::cerr << "ERROR: Failed to parse config JSON" << std::endl;
return 1;
}
std::cout << "{\"event\":\"ready\",\"schemaVersion\":2}" << std::endl;
if (config.sourceType != "display") {
std::cerr << "ERROR: Native window capture is not implemented yet" << std::endl;
return 1;
}
if (config.captureMic) {
std::cerr << "ERROR: Microphone capture is not implemented in this helper yet" << std::endl;
return 1;
}
if (config.webcamEnabled) {
std::cerr << "ERROR: Native webcam capture is not implemented in this helper yet" << std::endl;
return 1;
}
HMONITOR monitor = findMonitorForCapture(
config.displayId,
config.hasDisplayBounds ? &config.bounds : nullptr);
if (!monitor) {
std::cerr << "ERROR: Could not resolve monitor" << std::endl;
return 1;
}
WgcSession session;
if (!session.initialize(monitor, config.fps)) {
std::cerr << "ERROR: Failed to initialize WGC session" << std::endl;
return 1;
}
// WGC owns the captured texture size. Encoding must use that exact size
// until a dedicated GPU scaling pass is introduced; CopyResource requires
// matching resource dimensions.
int width = session.captureWidth();
int height = session.captureHeight();
width = (std::max(2, width) / 2) * 2;
height = (std::max(2, height) / 2) * 2;
const int pixels = width * height;
const int bitrate = pixels >= 3840 * 2160 ? 45'000'000 : pixels >= 2560 * 1440 ? 28'000'000 : 18'000'000;
WasapiLoopbackCapture loopbackCapture;
const AudioInputFormat* audioFormat = nullptr;
if (config.captureSystemAudio) {
if (!loopbackCapture.initialize()) {
std::cerr << "ERROR: Failed to initialize WASAPI loopback capture" << std::endl;
return 1;
}
audioFormat = &loopbackCapture.inputFormat();
std::cout << "{\"event\":\"audio-format\",\"schemaVersion\":2,\"sampleRate\":"
<< audioFormat->sampleRate << ",\"channels\":" << audioFormat->channels
<< ",\"bitsPerSample\":" << audioFormat->bitsPerSample << "}" << std::endl;
}
MFEncoder encoder;
if (!encoder.initialize(
utf8ToWide(config.outputPath),
width,
height,
config.fps,
bitrate,
session.device(),
session.context(),
audioFormat)) {
std::cerr << "ERROR: Failed to initialize Media Foundation encoder" << std::endl;
return 1;
}
std::mutex mutex;
std::condition_variable cv;
std::atomic<bool> stopRequested = false;
std::atomic<bool> firstFrameWritten = false;
std::atomic<bool> encodeFailed = false;
session.setFrameCallback([&](ID3D11Texture2D* texture, int64_t timestampHns) {
if (stopRequested) {
return;
}
std::scoped_lock lock(mutex);
if (!encoder.writeFrame(texture, timestampHns)) {
encodeFailed = true;
stopRequested = true;
cv.notify_all();
return;
}
if (!firstFrameWritten.exchange(true)) {
cv.notify_all();
}
});
if (config.captureSystemAudio) {
if (!loopbackCapture.start([&](const BYTE* data, DWORD byteCount, int64_t timestampHns, int64_t durationHns) {
if (stopRequested) {
return;
}
if (!encoder.writeAudio(data, byteCount, timestampHns, durationHns)) {
encodeFailed = true;
stopRequested = true;
cv.notify_all();
}
})) {
std::cerr << "ERROR: Failed to start WASAPI loopback capture" << std::endl;
return 1;
}
}
if (!session.start()) {
loopbackCapture.stop();
std::cerr << "ERROR: Failed to start WGC session" << std::endl;
return 1;
}
std::thread stdinThread(readStopCommands, std::ref(stopRequested), std::ref(cv));
{
std::unique_lock lock(mutex);
const bool started = cv.wait_for(lock, std::chrono::seconds(10), [&] {
return firstFrameWritten.load() || stopRequested.load();
});
if (!started || !firstFrameWritten) {
stopRequested = true;
cv.notify_all();
if (stdinThread.joinable()) {
stdinThread.detach();
}
loopbackCapture.stop();
std::cerr << "ERROR: Timed out waiting for first WGC frame" << std::endl;
return 1;
}
}
std::cout << "{\"event\":\"recording-started\",\"schemaVersion\":2}" << std::endl;
std::cout << "Recording started" << std::endl;
{
std::unique_lock lock(mutex);
cv.wait(lock, [&] {
return stopRequested.load();
});
}
loopbackCapture.stop();
session.stop();
{
std::scoped_lock lock(mutex);
encoder.finalize();
}
if (stdinThread.joinable()) {
stdinThread.join();
}
if (encodeFailed) {
std::cerr << "ERROR: Failed to encode WGC frame" << std::endl;
return 1;
}
std::cout << "{\"event\":\"recording-stopped\",\"schemaVersion\":2,\"screenPath\":\""
<< jsonEscape(config.outputPath) << "\"}" << std::endl;
std::cout << "Recording stopped. Output path: " << config.outputPath << std::endl;
return 0;
}
@@ -0,0 +1,317 @@
#include "mf_encoder.h"
#include <mfapi.h>
#include <mferror.h>
#include <propvarutil.h>
#include <algorithm>
#include <cstring>
#include <iostream>
namespace {
bool succeeded(HRESULT hr, const char* label) {
if (SUCCEEDED(hr)) {
return true;
}
std::cerr << "ERROR: " << label << " failed (hr=0x" << std::hex << hr << std::dec << ")"
<< std::endl;
return false;
}
void setFrameSize(IMFMediaType* type, UINT32 width, UINT32 height) {
MFSetAttributeSize(type, MF_MT_FRAME_SIZE, width, height);
}
void setFrameRate(IMFMediaType* type, UINT32 fps) {
MFSetAttributeRatio(type, MF_MT_FRAME_RATE, fps, 1);
}
void setPixelAspectRatio(IMFMediaType* type) {
MFSetAttributeRatio(type, MF_MT_PIXEL_ASPECT_RATIO, 1, 1);
}
void setAudioFormat(IMFMediaType* type, UINT32 channels, UINT32 sampleRate, UINT32 bitsPerSample) {
type->SetUINT32(MF_MT_AUDIO_NUM_CHANNELS, channels);
type->SetUINT32(MF_MT_AUDIO_SAMPLES_PER_SECOND, sampleRate);
type->SetUINT32(MF_MT_AUDIO_BITS_PER_SAMPLE, bitsPerSample);
}
} // namespace
MFEncoder::~MFEncoder() {
finalize();
}
bool MFEncoder::initialize(
const std::wstring& outputPath,
int width,
int height,
int fps,
int bitrate,
ID3D11Device* device,
ID3D11DeviceContext* context,
const AudioInputFormat* audioFormat) {
width_ = (std::max(2, width) / 2) * 2;
height_ = (std::max(2, height) / 2) * 2;
fps_ = std::max(1, fps);
device_ = device;
context_ = context;
if (!succeeded(MFStartup(MF_VERSION), "MFStartup")) {
return false;
}
Microsoft::WRL::ComPtr<IMFMediaType> outputType;
if (!succeeded(MFCreateMediaType(&outputType), "MFCreateMediaType(output)")) {
return false;
}
outputType->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video);
outputType->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_H264);
outputType->SetUINT32(MF_MT_AVG_BITRATE, static_cast<UINT32>(std::max(1, bitrate)));
outputType->SetUINT32(MF_MT_INTERLACE_MODE, MFVideoInterlace_Progressive);
setFrameSize(outputType.Get(), static_cast<UINT32>(width_), static_cast<UINT32>(height_));
setFrameRate(outputType.Get(), static_cast<UINT32>(fps_));
setPixelAspectRatio(outputType.Get());
if (!succeeded(MFCreateSinkWriterFromURL(outputPath.c_str(), nullptr, nullptr, &sinkWriter_),
"MFCreateSinkWriterFromURL")) {
return false;
}
if (!succeeded(sinkWriter_->AddStream(outputType.Get(), &videoStreamIndex_), "AddStream")) {
return false;
}
if (audioFormat && !configureAudioStream(*audioFormat)) {
return false;
}
Microsoft::WRL::ComPtr<IMFMediaType> inputType;
if (!succeeded(MFCreateMediaType(&inputType), "MFCreateMediaType(input)")) {
return false;
}
inputType->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video);
inputType->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_RGB32);
inputType->SetUINT32(MF_MT_INTERLACE_MODE, MFVideoInterlace_Progressive);
inputType->SetUINT32(MF_MT_DEFAULT_STRIDE, static_cast<UINT32>(width_ * 4));
setFrameSize(inputType.Get(), static_cast<UINT32>(width_), static_cast<UINT32>(height_));
setFrameRate(inputType.Get(), static_cast<UINT32>(fps_));
setPixelAspectRatio(inputType.Get());
if (!succeeded(sinkWriter_->SetInputMediaType(videoStreamIndex_, inputType.Get(), nullptr),
"SetInputMediaType")) {
return false;
}
if (!succeeded(sinkWriter_->BeginWriting(), "BeginWriting")) {
return false;
}
return true;
}
bool MFEncoder::configureAudioStream(const AudioInputFormat& audioFormat) {
if (!sinkWriter_) {
return false;
}
if (audioFormat.sampleRate == 0 || audioFormat.channels == 0 || audioFormat.blockAlign == 0) {
std::cerr << "ERROR: Invalid audio input format" << std::endl;
return false;
}
const UINT32 bitsPerSample = std::max<UINT32>(8, audioFormat.bitsPerSample);
const UINT32 aacBytesPerSecond = 24'000;
Microsoft::WRL::ComPtr<IMFMediaType> outputType;
if (!succeeded(MFCreateMediaType(&outputType), "MFCreateMediaType(audio output)")) {
return false;
}
outputType->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Audio);
outputType->SetGUID(MF_MT_SUBTYPE, MFAudioFormat_AAC);
setAudioFormat(outputType.Get(), audioFormat.channels, audioFormat.sampleRate, 16);
outputType->SetUINT32(MF_MT_AUDIO_AVG_BYTES_PER_SECOND, aacBytesPerSecond);
outputType->SetUINT32(MF_MT_AAC_PAYLOAD_TYPE, 0);
if (!succeeded(sinkWriter_->AddStream(outputType.Get(), &audioStreamIndex_), "AddStream(audio)")) {
return false;
}
Microsoft::WRL::ComPtr<IMFMediaType> inputType;
if (!succeeded(MFCreateMediaType(&inputType), "MFCreateMediaType(audio input)")) {
return false;
}
inputType->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Audio);
inputType->SetGUID(MF_MT_SUBTYPE, audioFormat.subtype);
setAudioFormat(inputType.Get(), audioFormat.channels, audioFormat.sampleRate, bitsPerSample);
inputType->SetUINT32(MF_MT_AUDIO_BLOCK_ALIGNMENT, audioFormat.blockAlign);
inputType->SetUINT32(MF_MT_AUDIO_AVG_BYTES_PER_SECOND, audioFormat.avgBytesPerSec);
inputType->SetUINT32(MF_MT_ALL_SAMPLES_INDEPENDENT, TRUE);
if (!succeeded(sinkWriter_->SetInputMediaType(audioStreamIndex_, inputType.Get(), nullptr),
"SetInputMediaType(audio)")) {
return false;
}
hasAudioStream_ = true;
return true;
}
bool MFEncoder::ensureStagingTexture(ID3D11Texture2D* texture) {
if (stagingTexture_) {
return true;
}
D3D11_TEXTURE2D_DESC desc{};
texture->GetDesc(&desc);
desc.Width = static_cast<UINT>(width_);
desc.Height = static_cast<UINT>(height_);
desc.MipLevels = 1;
desc.ArraySize = 1;
desc.Format = DXGI_FORMAT_B8G8R8A8_UNORM;
desc.SampleDesc.Count = 1;
desc.SampleDesc.Quality = 0;
desc.Usage = D3D11_USAGE_STAGING;
desc.BindFlags = 0;
desc.CPUAccessFlags = D3D11_CPU_ACCESS_READ;
desc.MiscFlags = 0;
return succeeded(device_->CreateTexture2D(&desc, nullptr, &stagingTexture_),
"CreateTexture2D(staging)");
}
bool MFEncoder::copyFrameToBuffer(ID3D11Texture2D* texture, BYTE* destination, DWORD destinationSize) {
if (!ensureStagingTexture(texture)) {
return false;
}
context_->CopyResource(stagingTexture_.Get(), texture);
D3D11_MAPPED_SUBRESOURCE mapped{};
if (!succeeded(context_->Map(stagingTexture_.Get(), 0, D3D11_MAP_READ, 0, &mapped), "Map")) {
return false;
}
const DWORD rowBytes = static_cast<DWORD>(width_ * 4);
const DWORD requiredBytes = rowBytes * static_cast<DWORD>(height_);
if (destinationSize < requiredBytes) {
context_->Unmap(stagingTexture_.Get(), 0);
std::cerr << "ERROR: Media Foundation buffer is too small" << std::endl;
return false;
}
auto* source = static_cast<const BYTE*>(mapped.pData);
for (int y = 0; y < height_; y += 1) {
std::memcpy(destination + rowBytes * y, source + mapped.RowPitch * y, rowBytes);
}
context_->Unmap(stagingTexture_.Get(), 0);
return true;
}
bool MFEncoder::writeFrame(ID3D11Texture2D* texture, int64_t timestampHns) {
std::scoped_lock writerLock(writerMutex_);
if (!sinkWriter_ || finalized_) {
return false;
}
if (firstTimestampHns_ < 0) {
firstTimestampHns_ = timestampHns;
}
int64_t sampleTime = timestampHns - firstTimestampHns_;
if (sampleTime <= lastTimestampHns_) {
sampleTime = lastTimestampHns_ + (10'000'000LL / fps_);
}
const int64_t sampleDuration = 10'000'000LL / fps_;
lastTimestampHns_ = sampleTime;
Microsoft::WRL::ComPtr<IMFMediaBuffer> buffer;
const DWORD frameBytes = static_cast<DWORD>(width_ * height_ * 4);
if (!succeeded(MFCreateMemoryBuffer(frameBytes, &buffer), "MFCreateMemoryBuffer")) {
return false;
}
BYTE* data = nullptr;
DWORD maxLength = 0;
DWORD currentLength = 0;
if (!succeeded(buffer->Lock(&data, &maxLength, &currentLength), "IMFMediaBuffer::Lock")) {
return false;
}
const bool copied = copyFrameToBuffer(texture, data, maxLength);
buffer->Unlock();
if (!copied) {
return false;
}
buffer->SetCurrentLength(frameBytes);
Microsoft::WRL::ComPtr<IMFSample> sample;
if (!succeeded(MFCreateSample(&sample), "MFCreateSample")) {
return false;
}
sample->AddBuffer(buffer.Get());
sample->SetSampleTime(sampleTime);
sample->SetSampleDuration(sampleDuration);
return succeeded(sinkWriter_->WriteSample(videoStreamIndex_, sample.Get()), "WriteSample");
}
bool MFEncoder::writeAudio(const BYTE* data, DWORD byteCount, int64_t timestampHns, int64_t durationHns) {
std::scoped_lock writerLock(writerMutex_);
if (!sinkWriter_ || finalized_ || !hasAudioStream_) {
return false;
}
if (!data || byteCount == 0 || durationHns <= 0) {
return true;
}
Microsoft::WRL::ComPtr<IMFMediaBuffer> buffer;
if (!succeeded(MFCreateMemoryBuffer(byteCount, &buffer), "MFCreateMemoryBuffer(audio)")) {
return false;
}
BYTE* destination = nullptr;
DWORD maxLength = 0;
DWORD currentLength = 0;
if (!succeeded(buffer->Lock(&destination, &maxLength, &currentLength),
"IMFMediaBuffer::Lock(audio)")) {
return false;
}
if (maxLength < byteCount) {
buffer->Unlock();
std::cerr << "ERROR: Media Foundation audio buffer is too small" << std::endl;
return false;
}
std::memcpy(destination, data, byteCount);
buffer->Unlock();
buffer->SetCurrentLength(byteCount);
Microsoft::WRL::ComPtr<IMFSample> sample;
if (!succeeded(MFCreateSample(&sample), "MFCreateSample(audio)")) {
return false;
}
sample->AddBuffer(buffer.Get());
sample->SetSampleTime(std::max<int64_t>(0, timestampHns));
sample->SetSampleDuration(durationHns);
return succeeded(sinkWriter_->WriteSample(audioStreamIndex_, sample.Get()), "WriteSample(audio)");
}
bool MFEncoder::finalize() {
std::scoped_lock writerLock(writerMutex_);
if (finalized_) {
return true;
}
finalized_ = true;
bool ok = true;
if (sinkWriter_) {
ok = succeeded(sinkWriter_->Finalize(), "SinkWriter::Finalize");
sinkWriter_.Reset();
}
stagingTexture_.Reset();
context_.Reset();
device_.Reset();
MFShutdown();
return ok;
}
@@ -0,0 +1,63 @@
#pragma once
#include <Windows.h>
#include <d3d11.h>
#include <mfapi.h>
#include <mfidl.h>
#include <mfreadwrite.h>
#include <wrl/client.h>
#include <cstdint>
#include <mutex>
#include <string>
struct AudioInputFormat {
GUID subtype = MFAudioFormat_PCM;
UINT32 sampleRate = 0;
UINT32 channels = 0;
UINT32 bitsPerSample = 0;
UINT32 blockAlign = 0;
UINT32 avgBytesPerSec = 0;
};
class MFEncoder {
public:
MFEncoder() = default;
~MFEncoder();
MFEncoder(const MFEncoder&) = delete;
MFEncoder& operator=(const MFEncoder&) = delete;
bool initialize(
const std::wstring& outputPath,
int width,
int height,
int fps,
int bitrate,
ID3D11Device* device,
ID3D11DeviceContext* context,
const AudioInputFormat* audioFormat = nullptr);
bool writeFrame(ID3D11Texture2D* texture, int64_t timestampHns);
bool writeAudio(const BYTE* data, DWORD byteCount, int64_t timestampHns, int64_t durationHns);
bool finalize();
private:
bool ensureStagingTexture(ID3D11Texture2D* texture);
bool copyFrameToBuffer(ID3D11Texture2D* texture, BYTE* destination, DWORD destinationSize);
bool configureAudioStream(const AudioInputFormat& audioFormat);
Microsoft::WRL::ComPtr<IMFSinkWriter> sinkWriter_;
Microsoft::WRL::ComPtr<ID3D11Device> device_;
Microsoft::WRL::ComPtr<ID3D11DeviceContext> context_;
Microsoft::WRL::ComPtr<ID3D11Texture2D> stagingTexture_;
std::mutex writerMutex_;
DWORD videoStreamIndex_ = 0;
DWORD audioStreamIndex_ = 0;
bool hasAudioStream_ = false;
int width_ = 0;
int height_ = 0;
int fps_ = 60;
int64_t firstTimestampHns_ = -1;
int64_t lastTimestampHns_ = -1;
bool finalized_ = false;
};
@@ -0,0 +1,88 @@
#include "monitor_utils.h"
#include <algorithm>
#include <cmath>
#include <vector>
namespace {
struct MonitorCandidate {
HMONITOR monitor = nullptr;
RECT rect{};
};
std::vector<MonitorCandidate> enumerateMonitors() {
std::vector<MonitorCandidate> monitors;
EnumDisplayMonitors(
nullptr,
nullptr,
[](HMONITOR monitor, HDC, LPRECT rect, LPARAM userData) -> BOOL {
auto* result = reinterpret_cast<std::vector<MonitorCandidate>*>(userData);
result->push_back({monitor, *rect});
return TRUE;
},
reinterpret_cast<LPARAM>(&monitors));
return monitors;
}
bool rectMatchesBounds(const RECT& rect, const MonitorBounds& bounds) {
return rect.left == bounds.x &&
rect.top == bounds.y &&
(rect.right - rect.left) == bounds.width &&
(rect.bottom - rect.top) == bounds.height;
}
int64_t overlapArea(const RECT& rect, const MonitorBounds& bounds) {
const LONG left = std::max<LONG>(rect.left, bounds.x);
const LONG top = std::max<LONG>(rect.top, bounds.y);
const LONG right = std::min<LONG>(rect.right, bounds.x + bounds.width);
const LONG bottom = std::min<LONG>(rect.bottom, bounds.y + bounds.height);
if (right <= left || bottom <= top) {
return 0;
}
return static_cast<int64_t>(right - left) * static_cast<int64_t>(bottom - top);
}
} // namespace
HMONITOR findMonitorForCapture(int64_t displayId, const MonitorBounds* bounds) {
const auto monitors = enumerateMonitors();
if (monitors.empty()) {
return MonitorFromPoint({0, 0}, MONITOR_DEFAULTTOPRIMARY);
}
// Electron's display_id is not stable across all Windows capture backends.
// Bounds are the most reliable contract because they come from Electron's
// selected display and match the WGC monitor coordinate space.
if (bounds && bounds->width > 0 && bounds->height > 0) {
for (const auto& candidate : monitors) {
if (rectMatchesBounds(candidate.rect, *bounds)) {
return candidate.monitor;
}
}
HMONITOR bestMonitor = nullptr;
int64_t bestArea = 0;
for (const auto& candidate : monitors) {
const int64_t area = overlapArea(candidate.rect, *bounds);
if (area > bestArea) {
bestArea = area;
bestMonitor = candidate.monitor;
}
}
if (bestMonitor) {
return bestMonitor;
}
}
// Best-effort fallback for helpers invoked without bounds. Some callers pass
// zero-based ids while Win32 monitor handles are pointer values, so only use
// this when it exactly matches the HMONITOR value.
for (const auto& candidate : monitors) {
if (reinterpret_cast<int64_t>(candidate.monitor) == displayId) {
return candidate.monitor;
}
}
return MonitorFromPoint({0, 0}, MONITOR_DEFAULTTOPRIMARY);
}
@@ -0,0 +1,14 @@
#pragma once
#include <Windows.h>
#include <cstdint>
struct MonitorBounds {
int x = 0;
int y = 0;
int width = 0;
int height = 0;
};
HMONITOR findMonitorForCapture(int64_t displayId, const MonitorBounds* bounds);
@@ -0,0 +1,205 @@
#include "wasapi_loopback_capture.h"
#include <ksmedia.h>
#include <algorithm>
#include <chrono>
#include <iostream>
namespace {
constexpr REFERENCE_TIME BufferDurationHns = 10'000'000;
constexpr int64_t HnsPerSecond = 10'000'000;
bool succeeded(HRESULT hr, const char* label) {
if (SUCCEEDED(hr)) {
return true;
}
std::cerr << "ERROR: " << label << " failed (hr=0x" << std::hex << hr << std::dec << ")"
<< std::endl;
return false;
}
GUID audioSubtypeFromFormat(WAVEFORMATEX* format) {
if (format->wFormatTag == WAVE_FORMAT_IEEE_FLOAT) {
return MFAudioFormat_Float;
}
if (format->wFormatTag == WAVE_FORMAT_PCM) {
return MFAudioFormat_PCM;
}
if (format->wFormatTag == WAVE_FORMAT_EXTENSIBLE &&
format->cbSize >= sizeof(WAVEFORMATEXTENSIBLE) - sizeof(WAVEFORMATEX)) {
auto* extensible = reinterpret_cast<WAVEFORMATEXTENSIBLE*>(format);
if (extensible->SubFormat == KSDATAFORMAT_SUBTYPE_IEEE_FLOAT) {
return MFAudioFormat_Float;
}
if (extensible->SubFormat == KSDATAFORMAT_SUBTYPE_PCM) {
return MFAudioFormat_PCM;
}
}
return GUID_NULL;
}
} // namespace
WasapiLoopbackCapture::~WasapiLoopbackCapture() {
stop();
if (mixFormat_) {
CoTaskMemFree(mixFormat_);
mixFormat_ = nullptr;
}
}
bool WasapiLoopbackCapture::initialize() {
HRESULT hr = CoCreateInstance(
__uuidof(MMDeviceEnumerator),
nullptr,
CLSCTX_ALL,
IID_PPV_ARGS(&deviceEnumerator_));
if (!succeeded(hr, "CoCreateInstance(MMDeviceEnumerator)")) {
return false;
}
hr = deviceEnumerator_->GetDefaultAudioEndpoint(eRender, eConsole, &device_);
if (!succeeded(hr, "GetDefaultAudioEndpoint(render)")) {
return false;
}
hr = device_->Activate(__uuidof(IAudioClient), CLSCTX_ALL, nullptr, &audioClient_);
if (!succeeded(hr, "IMMDevice::Activate(IAudioClient)")) {
return false;
}
hr = audioClient_->GetMixFormat(&mixFormat_);
if (!succeeded(hr, "IAudioClient::GetMixFormat") || !mixFormat_) {
return false;
}
if (!resolveInputFormat(mixFormat_)) {
std::cerr << "ERROR: Unsupported WASAPI loopback mix format" << std::endl;
return false;
}
hr = audioClient_->Initialize(
AUDCLNT_SHAREMODE_SHARED,
AUDCLNT_STREAMFLAGS_LOOPBACK,
BufferDurationHns,
0,
mixFormat_,
nullptr);
if (!succeeded(hr, "IAudioClient::Initialize(loopback)")) {
return false;
}
hr = audioClient_->GetService(IID_PPV_ARGS(&captureClient_));
if (!succeeded(hr, "IAudioClient::GetService(IAudioCaptureClient)")) {
return false;
}
return true;
}
bool WasapiLoopbackCapture::resolveInputFormat(WAVEFORMATEX* mixFormat) {
const GUID subtype = audioSubtypeFromFormat(mixFormat);
if (subtype == GUID_NULL) {
return false;
}
inputFormat_.subtype = subtype;
inputFormat_.sampleRate = mixFormat->nSamplesPerSec;
inputFormat_.channels = mixFormat->nChannels;
inputFormat_.bitsPerSample = mixFormat->wBitsPerSample;
inputFormat_.blockAlign = mixFormat->nBlockAlign;
inputFormat_.avgBytesPerSec = mixFormat->nAvgBytesPerSec;
return inputFormat_.sampleRate > 0 && inputFormat_.channels > 0 && inputFormat_.blockAlign > 0;
}
bool WasapiLoopbackCapture::start(AudioCallback callback) {
if (!audioClient_ || !captureClient_ || !callback) {
return false;
}
callback_ = std::move(callback);
stopRequested_ = false;
writtenFrames_ = 0;
HRESULT hr = audioClient_->Start();
if (!succeeded(hr, "IAudioClient::Start")) {
return false;
}
thread_ = std::thread([this] {
captureLoop();
});
return true;
}
void WasapiLoopbackCapture::stop() {
stopRequested_ = true;
if (thread_.joinable()) {
thread_.join();
}
if (audioClient_) {
audioClient_->Stop();
}
}
const AudioInputFormat& WasapiLoopbackCapture::inputFormat() const {
return inputFormat_;
}
void WasapiLoopbackCapture::captureLoop() {
while (!stopRequested_) {
UINT32 packetFrames = 0;
HRESULT hr = captureClient_->GetNextPacketSize(&packetFrames);
if (FAILED(hr)) {
std::cerr << "ERROR: IAudioCaptureClient::GetNextPacketSize failed (hr=0x" << std::hex
<< hr << std::dec << ")" << std::endl;
break;
}
while (packetFrames > 0 && !stopRequested_) {
BYTE* data = nullptr;
UINT32 framesAvailable = 0;
DWORD flags = 0;
hr = captureClient_->GetBuffer(&data, &framesAvailable, &flags, nullptr, nullptr);
if (FAILED(hr)) {
std::cerr << "ERROR: IAudioCaptureClient::GetBuffer failed (hr=0x" << std::hex
<< hr << std::dec << ")" << std::endl;
break;
}
const DWORD byteCount = framesAvailable * inputFormat_.blockAlign;
const int64_t timestampHns =
static_cast<int64_t>((writtenFrames_ * HnsPerSecond) / inputFormat_.sampleRate);
const int64_t durationHns =
static_cast<int64_t>((static_cast<uint64_t>(framesAvailable) * HnsPerSecond) /
inputFormat_.sampleRate);
if (byteCount > 0) {
if ((flags & AUDCLNT_BUFFERFLAGS_SILENT) != 0 || !data) {
silenceBuffer_.assign(byteCount, 0);
callback_(silenceBuffer_.data(), byteCount, timestampHns, durationHns);
} else {
callback_(data, byteCount, timestampHns, durationHns);
}
}
writtenFrames_ += framesAvailable;
captureClient_->ReleaseBuffer(framesAvailable);
hr = captureClient_->GetNextPacketSize(&packetFrames);
if (FAILED(hr)) {
std::cerr << "ERROR: IAudioCaptureClient::GetNextPacketSize failed (hr=0x"
<< std::hex << hr << std::dec << ")" << std::endl;
packetFrames = 0;
break;
}
}
std::this_thread::sleep_for(std::chrono::milliseconds(5));
}
}
@@ -0,0 +1,47 @@
#pragma once
#include "mf_encoder.h"
#include <Windows.h>
#include <audioclient.h>
#include <mmdeviceapi.h>
#include <wrl/client.h>
#include <atomic>
#include <cstdint>
#include <functional>
#include <thread>
#include <vector>
class WasapiLoopbackCapture {
public:
using AudioCallback = std::function<void(const BYTE* data, DWORD byteCount, int64_t timestampHns, int64_t durationHns)>;
WasapiLoopbackCapture() = default;
~WasapiLoopbackCapture();
WasapiLoopbackCapture(const WasapiLoopbackCapture&) = delete;
WasapiLoopbackCapture& operator=(const WasapiLoopbackCapture&) = delete;
bool initialize();
bool start(AudioCallback callback);
void stop();
const AudioInputFormat& inputFormat() const;
private:
void captureLoop();
bool resolveInputFormat(WAVEFORMATEX* mixFormat);
Microsoft::WRL::ComPtr<IMMDeviceEnumerator> deviceEnumerator_;
Microsoft::WRL::ComPtr<IMMDevice> device_;
Microsoft::WRL::ComPtr<IAudioClient> audioClient_;
Microsoft::WRL::ComPtr<IAudioCaptureClient> captureClient_;
WAVEFORMATEX* mixFormat_ = nullptr;
AudioInputFormat inputFormat_{};
AudioCallback callback_;
std::thread thread_;
std::atomic<bool> stopRequested_ = false;
std::vector<BYTE> silenceBuffer_;
uint64_t writtenFrames_ = 0;
};
@@ -0,0 +1,223 @@
#include "wgc_session.h"
#include <Windows.Graphics.Capture.Interop.h>
#include <dxgi1_2.h>
#include <inspectable.h>
#include <winrt/base.h>
#include <iostream>
namespace wf = winrt::Windows::Foundation;
namespace wgcap = winrt::Windows::Graphics::Capture;
namespace wgdx = winrt::Windows::Graphics::DirectX;
namespace wgd3d = winrt::Windows::Graphics::DirectX::Direct3D11;
extern "C" HRESULT __stdcall CreateDirect3D11DeviceFromDXGIDevice(
::IDXGIDevice* dxgiDevice,
::IInspectable** graphicsDevice);
namespace {
bool succeeded(HRESULT hr, const char* label) {
if (SUCCEEDED(hr)) {
return true;
}
std::cerr << "ERROR: " << label << " failed (hr=0x" << std::hex << hr << std::dec << ")"
<< std::endl;
return false;
}
int64_t timeSpanToHns(wf::TimeSpan const& value) {
return value.count();
}
} // namespace
WgcSession::~WgcSession() {
stop();
}
bool WgcSession::createD3DDevice() {
UINT flags = D3D11_CREATE_DEVICE_BGRA_SUPPORT;
#if defined(_DEBUG)
flags |= D3D11_CREATE_DEVICE_DEBUG;
#endif
D3D_FEATURE_LEVEL featureLevels[] = {
D3D_FEATURE_LEVEL_11_1,
D3D_FEATURE_LEVEL_11_0,
D3D_FEATURE_LEVEL_10_1,
D3D_FEATURE_LEVEL_10_0,
};
D3D_FEATURE_LEVEL featureLevel{};
HRESULT hr = D3D11CreateDevice(
nullptr,
D3D_DRIVER_TYPE_HARDWARE,
nullptr,
flags,
featureLevels,
ARRAYSIZE(featureLevels),
D3D11_SDK_VERSION,
&d3dDevice_,
&featureLevel,
&d3dContext_);
#if defined(_DEBUG)
if (FAILED(hr)) {
flags &= ~D3D11_CREATE_DEVICE_DEBUG;
hr = D3D11CreateDevice(
nullptr,
D3D_DRIVER_TYPE_HARDWARE,
nullptr,
flags,
featureLevels,
ARRAYSIZE(featureLevels),
D3D11_SDK_VERSION,
&d3dDevice_,
&featureLevel,
&d3dContext_);
}
#endif
if (!succeeded(hr, "D3D11CreateDevice")) {
return false;
}
Microsoft::WRL::ComPtr<IDXGIDevice> dxgiDevice;
if (!succeeded(d3dDevice_.As(&dxgiDevice), "Query IDXGIDevice")) {
return false;
}
winrt::com_ptr<::IInspectable> inspectableDevice;
if (!succeeded(CreateDirect3D11DeviceFromDXGIDevice(dxgiDevice.Get(), inspectableDevice.put()),
"CreateDirect3D11DeviceFromDXGIDevice")) {
return false;
}
winrtDevice_ = inspectableDevice.as<wgd3d::IDirect3DDevice>();
return true;
}
bool WgcSession::createCaptureItem(HMONITOR monitor) {
auto factory = winrt::get_activation_factory<wgcap::GraphicsCaptureItem>();
auto interop = factory.as<IGraphicsCaptureItemInterop>();
wgcap::GraphicsCaptureItem item{nullptr};
HRESULT hr = interop->CreateForMonitor(
monitor,
winrt::guid_of<wgcap::GraphicsCaptureItem>(),
reinterpret_cast<void**>(winrt::put_abi(item)));
if (!succeeded(hr, "CreateForMonitor")) {
return false;
}
item_ = item;
const auto size = item_.Size();
width_ = static_cast<int>(size.Width);
height_ = static_cast<int>(size.Height);
return width_ > 0 && height_ > 0;
}
bool WgcSession::initialize(HMONITOR monitor, int fps) {
fps_ = fps > 0 ? fps : 60;
if (!createD3DDevice()) {
return false;
}
if (!createCaptureItem(monitor)) {
return false;
}
framePool_ = wgcap::Direct3D11CaptureFramePool::CreateFreeThreaded(
winrtDevice_,
wgdx::DirectXPixelFormat::B8G8R8A8UIntNormalized,
2,
item_.Size());
session_ = framePool_.CreateCaptureSession(item_);
try {
session_.IsCursorCaptureEnabled(false);
} catch (...) {
// Older WGC builds can omit this property; callers still overlay their own cursor.
}
frameArrivedToken_ = framePool_.FrameArrived({this, &WgcSession::onFrameArrived});
return true;
}
void WgcSession::setFrameCallback(FrameCallback callback) {
std::scoped_lock lock(callbackMutex_);
frameCallback_ = std::move(callback);
}
bool WgcSession::start() {
if (!session_) {
return false;
}
session_.StartCapture();
started_ = true;
return true;
}
void WgcSession::stop() {
if (framePool_) {
framePool_.FrameArrived(frameArrivedToken_);
}
if (session_) {
session_.Close();
session_ = nullptr;
}
if (framePool_) {
framePool_.Close();
framePool_ = nullptr;
}
item_ = nullptr;
winrtDevice_ = nullptr;
d3dContext_.Reset();
d3dDevice_.Reset();
started_ = false;
}
void WgcSession::onFrameArrived(
wgcap::Direct3D11CaptureFramePool const& sender,
wf::IInspectable const&) {
auto frame = sender.TryGetNextFrame();
if (!frame) {
return;
}
auto surface = frame.Surface();
auto access = surface.as<::Windows::Graphics::DirectX::Direct3D11::IDirect3DDxgiInterfaceAccess>();
Microsoft::WRL::ComPtr<ID3D11Texture2D> texture;
HRESULT hr = access->GetInterface(__uuidof(ID3D11Texture2D), reinterpret_cast<void**>(texture.GetAddressOf()));
if (FAILED(hr) || !texture) {
return;
}
FrameCallback callback;
{
std::scoped_lock lock(callbackMutex_);
callback = frameCallback_;
}
if (callback) {
callback(texture.Get(), timeSpanToHns(frame.SystemRelativeTime()));
}
}
int WgcSession::captureWidth() const {
return width_;
}
int WgcSession::captureHeight() const {
return height_;
}
ID3D11Device* WgcSession::device() const {
return d3dDevice_.Get();
}
ID3D11DeviceContext* WgcSession::context() const {
return d3dContext_.Get();
}
@@ -0,0 +1,55 @@
#pragma once
#include <Windows.h>
#include <d3d11.h>
#include <windows.graphics.capture.h>
#include <windows.graphics.directx.direct3d11.interop.h>
#include <winrt/Windows.Foundation.h>
#include <winrt/Windows.Graphics.Capture.h>
#include <winrt/Windows.Graphics.DirectX.Direct3D11.h>
#include <wrl/client.h>
#include <functional>
#include <mutex>
class WgcSession {
public:
using FrameCallback = std::function<void(ID3D11Texture2D*, int64_t)>;
WgcSession() = default;
~WgcSession();
WgcSession(const WgcSession&) = delete;
WgcSession& operator=(const WgcSession&) = delete;
bool initialize(HMONITOR monitor, int fps);
void setFrameCallback(FrameCallback callback);
bool start();
void stop();
int captureWidth() const;
int captureHeight() const;
ID3D11Device* device() const;
ID3D11DeviceContext* context() const;
private:
bool createD3DDevice();
bool createCaptureItem(HMONITOR monitor);
void onFrameArrived(
winrt::Windows::Graphics::Capture::Direct3D11CaptureFramePool const& sender,
winrt::Windows::Foundation::IInspectable const&);
Microsoft::WRL::ComPtr<ID3D11Device> d3dDevice_;
Microsoft::WRL::ComPtr<ID3D11DeviceContext> d3dContext_;
winrt::Windows::Graphics::DirectX::Direct3D11::IDirect3DDevice winrtDevice_{nullptr};
winrt::Windows::Graphics::Capture::GraphicsCaptureItem item_{nullptr};
winrt::Windows::Graphics::Capture::Direct3D11CaptureFramePool framePool_{nullptr};
winrt::Windows::Graphics::Capture::GraphicsCaptureSession session_{nullptr};
winrt::event_token frameArrivedToken_{};
FrameCallback frameCallback_;
std::mutex callbackMutex_;
int width_ = 0;
int height_ = 0;
int fps_ = 60;
bool started_ = false;
};
+10
View File
@@ -1,4 +1,5 @@
import { contextBridge, ipcRenderer } from "electron";
import type { NativeWindowsRecordingRequest } from "../src/lib/nativeWindowsRecording";
import type { RecordingSession, StoreRecordedSessionInput } from "../src/lib/recordingSession";
import { NATIVE_BRIDGE_CHANNEL, type NativeBridgeRequest } from "../src/native/contracts";
@@ -64,6 +65,15 @@ contextBridge.exposeInMainWorld("electronAPI", {
setRecordingState: (recording: boolean, recordingId?: number) => {
return ipcRenderer.invoke("set-recording-state", recording, recordingId);
},
isNativeWindowsCaptureAvailable: () => {
return ipcRenderer.invoke("is-native-windows-capture-available");
},
startNativeWindowsRecording: (request: NativeWindowsRecordingRequest) => {
return ipcRenderer.invoke("start-native-windows-recording", request);
},
stopNativeWindowsRecording: (discard?: boolean) => {
return ipcRenderer.invoke("stop-native-windows-recording", discard);
},
getCursorTelemetry: (videoPath?: string) => {
return ipcRenderer.invoke("get-cursor-telemetry", videoPath);
},
+8 -5
View File
@@ -19,13 +19,16 @@
"lint:fix": "biome check --write .",
"format": "biome format --write .",
"i18n:check": "node scripts/i18n-check.mjs",
"preview": "vite preview",
"build:mac": "tsc && vite build && electron-builder --mac",
"build:win": "tsc && vite build && electron-builder --win --config.npmRebuild=false",
"build:linux": "tsc && vite build && electron-builder --linux AppImage deb pacman --config.npmRebuild=false",
"test": "vitest --run",
"preview": "vite preview",
"build:mac": "tsc && vite build && electron-builder --mac",
"build:native:win": "node scripts/build-windows-wgc-helper.mjs",
"build:win": "npm run build:native:win && tsc && vite build && electron-builder --win --config.npmRebuild=false",
"build:linux": "tsc && vite build && electron-builder --linux AppImage deb pacman --config.npmRebuild=false",
"test": "vitest --run",
"test:watch": "vitest",
"test:cursor-native:win": "node scripts/test-windows-native-cursor.mjs",
"test:wgc-helper:win": "node scripts/test-windows-wgc-helper.mjs",
"test:wgc-audio:win": "node scripts/test-windows-wgc-helper.mjs --system-audio",
"capture:openscreen-preview": "node scripts/capture-openscreen-preview.mjs",
"build-vite": "tsc && vite build",
"test:browser": "vitest --config vitest.browser.config.ts --run",
+112
View File
@@ -0,0 +1,112 @@
import { spawn } from "node:child_process";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import { fileURLToPath } from "node:url";
const __dirname = path.dirname(fileURLToPath(import.meta.url));
const ROOT = path.join(__dirname, "..");
const SOURCE_DIR = path.join(ROOT, "electron", "native", "wgc-capture");
const BUILD_DIR = path.join(SOURCE_DIR, "build");
const COMPAT_LIB_DIR = path.join(BUILD_DIR, "compat-libs");
const BIN_DIR = path.join(ROOT, "electron", "native", "bin", "win32-x64");
const CMAKE = process.env.CMAKE_EXE ?? "cmake";
function findVcVarsAll() {
const explicit = process.env.VCVARSALL;
if (explicit && fs.existsSync(explicit)) {
return explicit;
}
const roots = [
process.env.VSINSTALLDIR,
"C:\\Program Files\\Microsoft Visual Studio\\2022\\Community",
"C:\\Program Files\\Microsoft Visual Studio\\2022\\Professional",
"C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise",
"C:\\Program Files (x86)\\Microsoft Visual Studio\\2022\\BuildTools",
"C:\\Program Files (x86)\\Microsoft Visual Studio\\2022\\Community",
];
for (const root of roots.filter(Boolean)) {
const candidate = path.join(root, "VC", "Auxiliary", "Build", "vcvarsall.bat");
if (fs.existsSync(candidate)) {
return candidate;
}
}
return null;
}
function run(command, args, options = {}) {
return new Promise((resolve, reject) => {
const child = spawn(command, args, {
cwd: ROOT,
stdio: "inherit",
windowsHide: true,
...options,
});
child.once("error", reject);
child.once("exit", (code) => {
if (code === 0) {
resolve();
} else {
reject(new Error(`${command} ${args.join(" ")} failed with code ${code}`));
}
});
});
}
async function runInVsEnv(command) {
const vcvarsAll = findVcVarsAll();
if (!vcvarsAll) {
throw new Error(
"Could not find Visual Studio vcvarsall.bat. Install Visual Studio Build Tools with C++.",
);
}
const cmdPath = path.join(os.tmpdir(), `openscreen-build-wgc-${process.pid}-${Date.now()}.cmd`);
fs.writeFileSync(
cmdPath,
[
"@echo off",
`call "${vcvarsAll}" x64`,
"if errorlevel 1 exit /b %errorlevel%",
`if not exist "${COMPAT_LIB_DIR}" mkdir "${COMPAT_LIB_DIR}"`,
`for %%L in (gdi32.lib winspool.lib shell32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib) do if not exist "%WindowsSdkDir%Lib\\%WindowsSDKLibVersion%um\\x64\\%%L" copy /Y "%WindowsSdkDir%Lib\\%WindowsSDKLibVersion%um\\x64\\kernel32.Lib" "${COMPAT_LIB_DIR}\\%%L" >nul`,
"if errorlevel 1 exit /b %errorlevel%",
`set "LIB=${COMPAT_LIB_DIR};%LIB%"`,
command,
"exit /b %errorlevel%",
"",
].join("\r\n"),
);
try {
await run("cmd.exe", ["/d", "/c", cmdPath]);
} finally {
fs.rmSync(cmdPath, { force: true });
}
}
if (process.platform !== "win32") {
console.log("Skipping WGC helper build: Windows-only.");
process.exit(0);
}
fs.mkdirSync(BUILD_DIR, { recursive: true });
await runInVsEnv(
`"${CMAKE}" -S "${SOURCE_DIR}" -B "${BUILD_DIR}" -G Ninja -DCMAKE_BUILD_TYPE=Release`,
);
await runInVsEnv(`"${CMAKE}" --build "${BUILD_DIR}" --config Release`);
const outputPath = path.join(BUILD_DIR, "wgc-capture.exe");
if (!fs.existsSync(outputPath)) {
throw new Error(`WGC helper build completed but ${outputPath} was not found.`);
}
fs.mkdirSync(BIN_DIR, { recursive: true });
const distributablePath = path.join(BIN_DIR, "wgc-capture.exe");
fs.copyFileSync(outputPath, distributablePath);
console.log(`Built ${outputPath}`);
console.log(`Copied ${distributablePath}`);
+167
View File
@@ -0,0 +1,167 @@
import { spawn, spawnSync } from "node:child_process";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import { fileURLToPath } from "node:url";
const __dirname = path.dirname(fileURLToPath(import.meta.url));
const ROOT = path.join(__dirname, "..");
const HELPER_PATH =
process.env.OPENSCREEN_WGC_CAPTURE_EXE ??
path.join(ROOT, "electron", "native", "bin", "win32-x64", "wgc-capture.exe");
const DURATION_MS = Number(process.env.OPENSCREEN_WGC_TEST_DURATION_MS ?? 5000);
const WITH_SYSTEM_AUDIO =
process.env.OPENSCREEN_WGC_TEST_SYSTEM_AUDIO === "true" ||
process.argv.includes("--system-audio");
function runHelper(config) {
return new Promise((resolve, reject) => {
const child = spawn(HELPER_PATH, [JSON.stringify(config)], {
stdio: ["pipe", "pipe", "pipe"],
windowsHide: true,
});
let stdout = "";
let stderr = "";
child.stdout.on("data", (chunk) => {
stdout += chunk.toString();
});
child.stderr.on("data", (chunk) => {
stderr += chunk.toString();
});
child.once("error", reject);
child.once("exit", (code) => {
resolve({ code, stdout, stderr });
});
setTimeout(() => {
child.stdin.write("stop\n");
}, DURATION_MS);
});
}
function probeStreams(outputPath) {
const ffprobe = spawnSync(
"ffprobe",
["-v", "error", "-show_streams", "-of", "json", outputPath],
{ encoding: "utf8", windowsHide: true },
);
if (ffprobe.status !== 0) {
throw new Error(`ffprobe failed: ${ffprobe.stderr || ffprobe.stdout}`);
}
return JSON.parse(ffprobe.stdout).streams ?? [];
}
function measureFirstFrameLuma(outputPath) {
const ffmpeg = spawnSync(
"ffmpeg",
[
"-v",
"error",
"-i",
outputPath,
"-frames:v",
"1",
"-f",
"rawvideo",
"-pix_fmt",
"gray",
"pipe:1",
],
{ windowsHide: true, maxBuffer: 64 * 1024 * 1024 },
);
if (ffmpeg.status !== 0) {
throw new Error(`ffmpeg frame extraction failed: ${ffmpeg.stderr?.toString() ?? ""}`);
}
const data = ffmpeg.stdout;
if (!data || data.length === 0) {
throw new Error(`ffmpeg did not return frame data for ${outputPath}`);
}
let sum = 0;
let max = 0;
for (const value of data) {
sum += value;
if (value > max) {
max = value;
}
}
return { average: sum / data.length, max };
}
if (process.platform !== "win32") {
console.log("Skipping WGC helper smoke test: Windows-only.");
process.exit(0);
}
if (!fs.existsSync(HELPER_PATH)) {
throw new Error(`WGC helper not found at ${HELPER_PATH}. Run npm run build:native:win first.`);
}
const outputPath = path.join(
os.tmpdir(),
`openscreen-wgc-helper-${WITH_SYSTEM_AUDIO ? "audio" : "video"}-${Date.now()}.mp4`,
);
const config = {
schemaVersion: 2,
recordingId: Date.now(),
outputPath,
sourceType: "display",
sourceId: "screen:0:0",
displayId: 0,
fps: 30,
videoWidth: 1280,
videoHeight: 720,
displayX: 0,
displayY: 0,
displayW: 1920,
displayH: 1080,
hasDisplayBounds: true,
captureSystemAudio: WITH_SYSTEM_AUDIO,
captureMic: false,
webcamEnabled: false,
outputs: { screenPath: outputPath },
};
const result = await runHelper(config);
if (result.code !== 0) {
throw new Error(`WGC helper exited with ${result.code}\n${result.stdout}\n${result.stderr}`);
}
if (!fs.existsSync(outputPath) || fs.statSync(outputPath).size === 0) {
throw new Error(`WGC helper did not produce a video at ${outputPath}`);
}
const streams = probeStreams(outputPath);
const hasVideo = streams.some((stream) => stream.codec_type === "video");
const hasAudio = streams.some((stream) => stream.codec_type === "audio");
if (!hasVideo) {
throw new Error(`WGC helper output has no video stream: ${outputPath}`);
}
if (WITH_SYSTEM_AUDIO && !hasAudio) {
throw new Error(`WGC helper output has no audio stream: ${outputPath}`);
}
const frameLuma = measureFirstFrameLuma(outputPath);
if (frameLuma.average < 1 && frameLuma.max < 5) {
throw new Error(`WGC helper output first frame is black: ${outputPath}`);
}
console.log(
JSON.stringify(
{
success: true,
outputPath,
bytes: fs.statSync(outputPath).size,
streams: streams.map((stream) => ({
index: stream.index,
codecType: stream.codec_type,
codecName: stream.codec_name,
duration: stream.duration,
})),
firstFrameLuma: frameLuma,
},
null,
2,
),
);
@@ -1832,7 +1832,6 @@ const VideoPlayback = forwardRef<VideoPlaybackRef, VideoPlaybackProps>(
src={videoPath}
className="hidden"
preload="auto"
muted
playsInline
onLoadedMetadata={handleLoadedMetadata}
onDurationChange={(e) => {
+166 -1
View File
@@ -2,6 +2,7 @@ import { fixWebmDuration } from "@fix-webm-duration/fix";
import { useCallback, useEffect, useRef, useState } from "react";
import { toast } from "sonner";
import { useScopedT } from "@/contexts/I18nContext";
import type { NativeWindowsRecordingRequest } from "@/lib/nativeWindowsRecording";
import { requestCameraAccess } from "@/lib/requestCameraAccess";
const TARGET_FRAME_RATE = 60;
@@ -62,6 +63,11 @@ type RecorderHandle = {
recordedBlobPromise: Promise<Blob>;
};
type NativeWindowsRecordingHandle = {
recordingId: number;
finalizing: boolean;
};
function createRecorderHandle(stream: MediaStream, options: MediaRecorderOptions): RecorderHandle {
const recorder = new MediaRecorder(stream, options);
const chunks: Blob[] = [];
@@ -96,6 +102,7 @@ export function useScreenRecorder(): UseScreenRecorderReturn {
const [webcamEnabled, setWebcamEnabledState] = useState(false);
const screenRecorder = useRef<RecorderHandle | null>(null);
const webcamRecorder = useRef<RecorderHandle | null>(null);
const nativeWindowsRecording = useRef<NativeWindowsRecordingHandle | null>(null);
const stream = useRef<MediaStream | null>(null);
const screenStream = useRef<MediaStream | null>(null);
const microphoneStream = useRef<MediaStream | null>(null);
@@ -365,7 +372,58 @@ export function useScreenRecorder(): UseScreenRecorderReturn {
[teardownMedia],
);
const finalizeNativeWindowsRecording = useCallback(async (discard = false) => {
const activeNativeRecording = nativeWindowsRecording.current;
if (!activeNativeRecording || activeNativeRecording.finalizing) {
return false;
}
activeNativeRecording.finalizing = true;
nativeWindowsRecording.current = null;
setRecording(false);
setPaused(false);
setElapsedSeconds(0);
accumulatedDurationMs.current = 0;
segmentStartedAt.current = null;
try {
const result = await window.electronAPI.stopNativeWindowsRecording(discard);
if (discard || result.discarded) {
return true;
}
if (!result.success) {
console.error("Failed to stop native Windows recording:", result.error);
toast.error(result.error ?? "Failed to stop native Windows recording");
return true;
}
if (result.session) {
await window.electronAPI.setCurrentRecordingSession(result.session);
} else if (result.path) {
await window.electronAPI.setCurrentVideoPath(result.path);
}
await window.electronAPI.switchToEditor();
return true;
} catch (error) {
console.error("Error saving native Windows recording:", error);
toast.error(
error instanceof Error ? error.message : "Failed to save native Windows recording",
);
return true;
} finally {
if (discardRecordingId.current === activeNativeRecording.recordingId) {
discardRecordingId.current = null;
}
}
}, []);
const stopRecording = useRef(() => {
if (nativeWindowsRecording.current) {
void finalizeNativeWindowsRecording(false);
return;
}
const activeScreenRecorder = screenRecorder.current;
if (!activeScreenRecorder) {
return;
@@ -431,6 +489,9 @@ export function useScreenRecorder(): UseScreenRecorderReturn {
allowAutoFinalize.current = false;
restarting.current = false;
discardRecordingId.current = null;
if (nativeWindowsRecording.current) {
void finalizeNativeWindowsRecording(true);
}
if (
screenRecorder.current?.recorder.state === "recording" ||
@@ -456,7 +517,7 @@ export function useScreenRecorder(): UseScreenRecorderReturn {
webcamRecorder.current = null;
teardownMedia();
};
}, [teardownMedia, safeHideCountdownOverlay]);
}, [teardownMedia, safeHideCountdownOverlay, finalizeNativeWindowsRecording]);
const safeShowCountdownOverlay = async (value: number, runId: number) => {
try {
@@ -486,6 +547,85 @@ export function useScreenRecorder(): UseScreenRecorderReturn {
const isCountdownRunActive = (runId?: number) =>
runId === undefined || countdownRunId.current === runId;
const startNativeWindowsRecordingIfAvailable = async (
selectedSource: ProcessedDesktopSource,
countdownRunToken?: number,
) => {
try {
const platform = await window.electronAPI.getPlatform();
if (platform !== "win32") {
return false;
}
const availability = await window.electronAPI.isNativeWindowsCaptureAvailable();
if (!availability.success || !availability.available) {
throw new Error(
availability.reason === "missing-helper"
? "Native Windows capture helper is not available."
: (availability.error ?? "Native Windows capture is not available."),
);
}
if (!isCountdownRunActive(countdownRunToken)) {
return true;
}
const activeRecordingId = Date.now();
const displayId = Number(selectedSource.display_id);
const sourceType = selectedSource.id.startsWith("window:") ? "window" : "display";
const request: NativeWindowsRecordingRequest = {
recordingId: activeRecordingId,
source: {
type: sourceType,
sourceId: selectedSource.id,
...(Number.isFinite(displayId) ? { displayId } : {}),
},
video: {
fps: TARGET_FRAME_RATE,
width: TARGET_WIDTH,
height: TARGET_HEIGHT,
},
audio: {
system: {
enabled: systemAudioEnabled,
},
microphone: {
enabled: microphoneEnabled,
deviceId: microphoneDeviceId,
gain: MIC_GAIN_BOOST,
},
},
webcam: {
enabled: webcamEnabled,
deviceId: webcamDeviceId,
width: WEBCAM_TARGET_WIDTH,
height: WEBCAM_TARGET_HEIGHT,
fps: WEBCAM_TARGET_FRAME_RATE,
},
};
const result = await window.electronAPI.startNativeWindowsRecording(request);
if (!result.success || !result.recordingId) {
throw new Error(result.error ?? "Native Windows capture failed.");
}
recordingId.current = result.recordingId;
nativeWindowsRecording.current = {
recordingId: result.recordingId,
finalizing: false,
};
accumulatedDurationMs.current = 0;
segmentStartedAt.current = result.recordingId;
allowAutoFinalize.current = true;
setRecording(true);
setPaused(false);
setElapsedSeconds(0);
return true;
} catch (error) {
console.error("Native Windows capture failed:", error);
throw error;
}
};
const startRecordCountdown = async () => {
if (countdownActive || recording) {
return;
@@ -573,6 +713,10 @@ export function useScreenRecorder(): UseScreenRecorderReturn {
return;
}
if (await startNativeWindowsRecordingIfAvailable(selectedSource, countdownRunToken)) {
return;
}
let screenMediaStream: MediaStream;
// getDisplayMedia + setDisplayMediaRequestHandler (main.ts) supplies the
@@ -846,6 +990,19 @@ export function useScreenRecorder(): UseScreenRecorderReturn {
const restartRecording = async () => {
if (restarting.current) return;
if (nativeWindowsRecording.current) {
const activeRecordingId = recordingId.current;
restarting.current = true;
discardRecordingId.current = activeRecordingId;
try {
await finalizeNativeWindowsRecording(true);
await startRecording();
} finally {
restarting.current = false;
}
return;
}
const activeScreenRecorder = screenRecorder.current;
if (!activeScreenRecorder || activeScreenRecorder.recorder.state === "inactive") return;
@@ -903,6 +1060,14 @@ export function useScreenRecorder(): UseScreenRecorderReturn {
}, [getRecordingDurationMs, paused, recording]);
const cancelRecording = () => {
if (nativeWindowsRecording.current) {
const activeRecordingId = recordingId.current;
discardRecordingId.current = activeRecordingId;
allowAutoFinalize.current = false;
void finalizeNativeWindowsRecording(true);
return;
}
const activeScreenRecorder = screenRecorder.current;
if (
activeScreenRecorder?.recorder.state === "recording" ||
+9 -9
View File
@@ -54,8 +54,8 @@ const PRETTY_NATIVE_CURSOR_ASSETS: Partial<Record<NativeCursorType, PrettyNative
imageDataUrl: arrowUrl,
width: 32,
height: 32,
hotspotX: 5.8,
hotspotY: 3.2,
hotspotX: 16.25,
hotspotY: 15.03,
},
text: {
imageDataUrl: textUrl,
@@ -67,9 +67,9 @@ const PRETTY_NATIVE_CURSOR_ASSETS: Partial<Record<NativeCursorType, PrettyNative
pointer: {
imageDataUrl: pointerUrl,
width: 32,
height: 32,
hotspotX: 11.8,
hotspotY: 2.6,
height: 33,
hotspotX: 16.65,
hotspotY: 14.24,
},
crosshair: {
imageDataUrl: crosshairUrl,
@@ -131,15 +131,15 @@ const PRETTY_NATIVE_CURSOR_ASSETS: Partial<Record<NativeCursorType, PrettyNative
imageDataUrl: appStartingUrl,
width: 32,
height: 32,
hotspotX: 5.8,
hotspotY: 3.2,
hotspotX: 7.25,
hotspotY: 4.03,
},
help: {
imageDataUrl: helpUrl,
width: 32,
height: 32,
hotspotX: 5.8,
hotspotY: 3.2,
hotspotX: 7.25,
hotspotY: 4.03,
},
"up-arrow": {
imageDataUrl: upArrowUrl,
+4 -24
View File
@@ -3,6 +3,7 @@ import type { SpeedRegion, TrimRegion } from "@/components/video-editor/types";
import type { VideoMuxer } from "./muxer";
const AUDIO_BITRATE = 128_000;
const EXPORT_AUDIO_CODEC = "mp4a.40.2";
const DECODE_BACKPRESSURE_LIMIT = 20;
const MIN_SPEED_REGION_DELTA_MS = 0.0001;
const SEEK_TIMEOUT_MS = 5_000;
@@ -138,7 +139,7 @@ export class AudioProcessor {
const channels = audioConfig.numberOfChannels || 2;
const encodeConfig: AudioEncoderConfig = {
codec: "opus",
codec: EXPORT_AUDIO_CODEC,
sampleRate,
numberOfChannels: channels,
bitrate: AUDIO_BITRATE,
@@ -146,7 +147,7 @@ export class AudioProcessor {
const encodeSupport = await AudioEncoder.isConfigSupported(encodeConfig);
if (!encodeSupport.supported) {
console.warn("[AudioProcessor] Opus encoding not supported, skipping audio");
console.warn("[AudioProcessor] AAC encoding not supported, skipping audio");
for (const frame of decodedFrames) frame.close();
return;
}
@@ -397,28 +398,7 @@ export class AudioProcessor {
try {
await demuxer.load(file);
const audioConfig = await demuxer.getDecoderConfig("audio");
const reader = demuxer.read("audio").getReader();
let isFirstChunk = true;
try {
while (!this.cancelled) {
const { done, value: chunk } = await reader.read();
if (done || !chunk) break;
if (isFirstChunk) {
await muxer.addAudioChunk(chunk, { decoderConfig: audioConfig });
isFirstChunk = false;
} else {
await muxer.addAudioChunk(chunk);
}
}
} finally {
try {
await reader.cancel();
} catch {
/* reader already closed */
}
}
await this.processTrimOnlyAudio(demuxer, muxer, []);
} finally {
try {
demuxer.destroy();
+1 -1
View File
@@ -40,7 +40,7 @@ export class VideoMuxer {
// Create audio source if needed
if (this.hasAudio) {
this.audioSource = new EncodedAudioPacketSource("opus");
this.audioSource = new EncodedAudioPacketSource("aac");
this.output.addAudioTrack(this.audioSource);
}
+41
View File
@@ -0,0 +1,41 @@
export type NativeWindowsSourceType = "display" | "window";
export type NativeWindowsRecordingRequest = {
recordingId?: number;
source: {
type: NativeWindowsSourceType;
sourceId: string;
displayId?: number;
windowHandle?: string;
};
video: {
fps: number;
width: number;
height: number;
};
audio: {
system: {
enabled: boolean;
};
microphone: {
enabled: boolean;
deviceId?: string;
gain: number;
};
};
webcam: {
enabled: boolean;
deviceId?: string;
width: number;
height: number;
fps: number;
};
};
export type NativeWindowsRecordingStartResult = {
success: boolean;
recordingId?: number;
path?: string;
helperPath?: string;
error?: string;
};