Modules

Screph modules and their boundaries

Screph is an environment for creating AI tasks on visual data. It consists of modules for source acquisition, markup, computer-vision methods, project storage, work with code and scenario execution. Inputs, outputs and current limitations are listed for each module.

ImplementedExperimentalPlanned
Input

Capture

Implemented

Acquires and indexes visual sources but does not create markup by itself.

InputsRegion/window/monitor, camera, URL stream or a validated set of files and folders.
OperationsTimed or action-sequence recording, playback, timeline editing, memory or disk storage, source validation and preview.
OutputFrames, source references, timeline positions, PNG exports and files associated with actions and frames.
BoundaryMedia files are played back, not recorded; a high-load live source may reduce interface responsiveness.
Authoring

Selector

Implemented

Owns the main editable project and connects the visual source with domain markup.

  • Elements, typed visual features and selection areas with automatic or manual membership. Markup groups are part of the main project format, but Selector does not yet have a separate free-form group editor.
  • Feature operations: copies use separate image files; patterns, grids and derived features can be created; target points, owner and source links, crop, mask and preview are checked.
  • The relation graph shows hierarchy, owner-to-feature links, element and feature relations, visibility and properties of the selected relation.
  • Canvas layouts A/B/C/D show different materials from one project. Zoom, pan and cursor can be linked; the layout is stored locally and is not part of the project package.
  • The active image provides pixel inspection, a magnifier, histogram, line profile, layers, a navigator and result comparison between revisions.
  • Text and voice descriptions can be linked to elements, features, matches, relations and image series. Semantic processing is configured separately and may automatically apply a high-confidence typed rule.
  • Tree modules store the selected branch in a separate compact JSON file and check whether it matches the current project state.
Analysis

Computer vision

Implemented

Computer-vision methods create results and may propose markup changes. The main markup changes only when such a change is applied separately.

28 panel methods

Edges, HSV/K-Means, GrabCut, Watershed, superpixels, contours and components, MSER/Hough, thresholding, matching, comparison and alignment, KLT, OCR, layout analysis and model-backed segmentation.

14 sequence nodes

Grayscale, tone and contrast, blur, CLAHE, resize, crop, rotate and deskew, mask composition, inversion, morphology and contour extraction.

Execution and dependencies

A single method, a linear sequence or a method graph. OCR requires ocr.runtime; YOLO, SAM and OmniParser require ml.runtime and model files.

Full catalog and mode boundaries →

Data

Projects and exports

Implemented

The main project package uses versioned JSON and retains references to result files.

  • gui_elements, image_features, feature_matches, markup_groups, edges и regions.
  • Source images, computer-vision result manifests, masks, contours, metrics and save diagnostics.
  • An automatically generated for_ai_agent file; a supported automation target separately prepares main.py.
  • Schema and file-integrity validation during load and save.
Execution

GUI Automation

Implemented

Automation Runtime reads the main project and executes a separate Python scenario. The for_ai_agent file is not its direct input, and execution does not start automatically after markup.

Find and wait

Image and text matching, condition waits and recorded capture for repeatable debugging.

Input

Click, move, text input, hotkey, drag and scroll through Arduino HID, legacy serial or FakerInput Virtual HID.

Automation Manager

Scenario launch, EVT telemetry, screen and match preview, progress and logs.

Assistance

AI Assistant

Experimental

The assistant is off by default. It builds context with restricted data redacted and invokes only allowed typed actions. Images and files require explicit permission; changes are previewed and require confirmation.

Open the Assistant guide →
Recorded actions

Action CV

Experimental

Connects frames before and after an action, the action itself, an LLM plan, a local computer-vision run, proposed targets, review decisions and export. The original action trace remains unchanged for verification.

Code and speech

Screph Code and voice

Implemented
  • Screph Code uses the Monaco editor, the OpenHands runtime and a separate Pro Agent process with a versioned JSON-RPC protocol.
  • Builder can modify files in the selected code folder immediately. Changes and Undo do not cover every agent edit, so file changes require inspection with Git or another external tool; use a separate copy or branch before running it.
  • IDE transfer supports the VS Code/Codex bridge, Trae, PearAI, and Window Commander. Other IDE extension packages remain internal scaffolds.
  • Speech recognition uses Vosk, Yandex, Google and OpenAI. Available modes depend on the selected service; voice descriptions are stored in the project.
  • Voice2Text is a separate system-tray application for editable dictation, text insertion into Windows applications, bindings and optional screenshot transformation through an LLM. The main application is implemented; the current legacy ZIP requires replacement after publication review.
Module status

An available mode does not always mean a complete process

Status and limitations