Screph - an environment for creating tasks for AI using visual data. Capture receives the raw materials, Selector stores the editable markup, computer vision techniques generate additional results, and the main project passes selected data to tools for code manipulation and script execution.
Capture works with screen area, window, monitor, camera, URL stream, images, videos and folders. Frames remain connected to their source and position on the timeline.
Selector associates a visual source with elements, features, regions, annotations, and relationships. The main markup is edited on the canvas and saved in the project with a history of changes.
A single method, sequence, or graph of methods can create masks, outlines, detections, recognized text, metrics, and an execution log. The result of the method is stored separately from the proposed markup change.
Boundaries, thresholding, segmentation, contours, mapping, comparison, alignment and motion analysis.
YOLO, SAM and OmniParser require a prepared runtime environment and suitable model files. OCR with Tesseract requires local installation of Tesseract.
A proposed change becomes part of the main markup only after a user decision in a supported review process.
The main project stores the format version, source images, markup, connections, groups, descriptions and links to attached method results. When loading and saving, the project structure and associated files are checked.
A separate view of the selected data is created for Screph Code and supported external tools. GUI Automation uses the main project and a separately prepared script.
Automation Runtime loads the main project, looks for images and text, waits for the given conditions and performs input actions through the configured Arduino HID or FakerInput Virtual HID. Markup itself does not trigger automation.
Before launching, you need a script, a suitable input method, verification of coordinates and observation of the target application.
Screph Code receives the selected project data through the installed Pro Agent component and works with the code folder. Builder can change files in this folder, so you need a separate copy or branch before working, and then check the changes through Git or another diff tool.
The assistant is disabled by default. Images and files are shared with permission only, and actions that change data are shown for review and require confirmation.
Agent setup and limitations →Action CV associates an action with frames before and after it, runs selected methods locally, and stores suggested goals along with the user's decision. The result requires human verification.