Capturing sources and actions

The Capture tab brings screen regions, windows, monitors, cameras, URL streams and local media into one timeline. You can record a timed frame series or a sequence of user actions, review the material and only then hand it to markup and CV.

1. Four source groups

  • Screen — a virtual-desktop region, physical monitor or a specific window.
  • Camera — a live source with a selected device index and OpenCV backend.
  • Files / folders — one manager for one or more images, videos and directories, rather than separate “video file” and “frame folder” modes.
  • Stream / URL — RTSP, HTTP and other addresses supported by the selected OpenCV backend.

Boundary: recording is available for screen, camera and stream sources. Local files and folders are attached for navigation and playback; Screph does not record them again.

Network and credentials. For Stream / URL, the selected OpenCV/FFmpeg/GStreamer backend connects directly to the endpoint; this is not an upload to Screph Cloud. The URL is stored in shared Capture settings. Avoid embedding a long-lived password/token in the URL and treat the settings file as sensitive when the endpoint contains credentials.

2. What to capture on screen

For the Screen source, choose the capture scope before recording:

  • Selected region — a rectangle in virtual-desktop coordinates; it may span multiple monitors.
  • Selected monitor — a specific physical output from the list.
  • Active monitor — the monitor containing the active window at capture time.
  • Selected window — a window bound to its HWND, process and class; cursor capture can be enabled or disabled.
  • Follow active window — follow the active window while excluding the Screph application itself.

Regions and monitors use DXcam/DXGI; windows use Windows Graphics Capture. A closed or minimized window, missing monitor or backend error is reported explicitly: the application does not silently replace the selected target with another source.

3. Files and folders manager

  1. Select Files / folders and press Select….
  2. Add multiple files and directories or drag them in from Explorer.
  3. Enable recursive folder scanning when needed.
  4. Wait for asynchronous validation of each row and apply only valid sources.

Images are supported as PNG, JPG/JPEG and BMP. Video formats include MKV, AVI, MP4, MOV/M4V, WEBM and MPEG/MPG. Selection history is project-scoped and can reload a previous set.

Offline video is read using exact PTS/time-base through PyAV. The separate OpenCV path accepts only explicitly validated CFR material; unverified VFR is not hidden behind an approximate FPS value.

4. Two recording strategies

timed — timed capture

Screph acquires frames at the configured FPS. Use it for screen motion, video analysis and an evenly sampled time series.

action_sequence — action sequence

Screph records mouse and keyboard activity and associates each action with state frames before, during and after it. This mode is intended for reproducible GUI workflows: the timeline stores an Action Trace as well as images.

Pause and stop affect the active strategy. Action-sequence settings control delays, frames around an action, key-repeat grouping and whether mouse-move frames are included.

5. Timeline, navigation and editing

  • The scrubber, frame stepping, playback, speed and loop controls work for recorded and imported media.
  • A timeline range can be selected and deleted; related action records are remapped to the remaining frames instead of retaining stale indexes.
  • Persisted links use durable source/frame references and timeline placements. A global frame number alone is not treated as sufficient identity.
  • An unavailable position is returned as a diagnosable gap; the temporal reader does not skip it silently.

The Preview button opens a separate Qt/PySide window. Frame, duration, memory and current-path information remains on the main tab.

6. Action labeling with CV and VLM

Action CV is available only for the Screen source and the action_sequence strategy. Select an input-event provider first: the disabled default blocks recording. Automatic processing after recording stops requires both Delegate labeling to VLM and Run after recording stops. The Label with VLM button starts a full manual rerun independently of those toggles.

The safe defaults neither send data nor create a CV candidate. The block_unredacted policy produces a blocking diagnostic before any provider call. After explicitly selecting roi_only, Screph sends a full-size frame with everything outside the ROI painted over; allow_unredacted permits the original frame. The default object_descriptor contract returns a description and future-search recommendation but does not execute CV. To obtain candidates, select the CV plan or hybrid contract and separately enable Execute VLM CV plans.

What the selected CV model receives

  • The explicit cv LLM-role assignment is used; there is no hidden fallback to another profile or model. The profile may target a local runtime or an external provider.
  • The prompt contains the normalized action—type, time, source, key or button and coordinates when present—plus the target point and ROI hint, prepared-frame metadata and hashes, the selected response contract and the allowed CV-method catalogue.
  • By default, at most three frames are selected in action, after, before order. A blocking privacy diagnostic stops the request before the client call.
  • Successful shared LLM calls leave usage metadata, while Action CV also stores requests, raw response payloads and a response cache inside the recording directory by default.

Review, exports and the project

  1. The timeline and review panel show status, overlay/action/before/after/mask views, trace, confidence and diagnostics. Failed actions can be retried individually or as a group; cancel is a cooperative request.
  2. You can accept or reject a candidate, choose an alternative or correct its bounding box. The decision is appended to review JSONL.
  3. Accept immediately rebuilds derived screen-element, target and project-patch artifacts and enriches existing automation-context steps when present. It still does not mutate the canonical Canvas.
  4. Into project is a separate mutation command: it imports the current patch into the Canvas through the undo/history gateway and updates existing elements by candidate ID. Save the canonical project afterwards.

Verify export and import status. The current review panel does not surface a derived-export rebuild failure, so an older patch may remain after a disk problem. A full manual rerun removes the action_cv directory, including prior requests, responses, decisions and exports, but does not roll back enrichment already written into the adjacent automation_context. Preserve anything you need before rerunning, then re-check accepted targets and project import.

Configure an LLM connection and data-transfer rules →

7. Storage, settings and export

  • ram_images — frames in memory with buffer controls.
  • disk_images — frames on project storage for longer sessions.
  • PNG export runs in a background worker; interval-based auto-save is available.
  • FPS, recording strategy, scope, buffer, playback, export and source settings are stored in Screph settings. The same contract is editable under Settings → CV settings → Capture settings.

8. If a source does not work

  • Window is blank or stopped: check whether it was minimized or closed, then refresh targets and select the window again.
  • Camera or URL does not open: verify the index/address and try another explicit backend: Auto, MSMF, DirectShow, FFmpeg or GStreamer.
  • File is marked invalid: do not apply the row before validation finishes; for video, check the container, codec and timebase.
  • High memory usage: reduce FPS/session length or switch from ram_images to disk_images.
  • VLM labeling is blocked: check the CV role in LLM settings, model availability and privacy diagnostics; there is no hidden provider fallback.
Next: markup and project CV GUI and pipeline