Use cases

Supported Screph workflows

Four workflows: preparing interface automation, evaluating computer vision operations, finding targets in an action recording, and experimental Industrial and UAV tasks.

ImplementedGeneral / GUI Automation

Preparing a GUI automation script

Purpose

Describe interface elements, their features, states and relationships for later automation.

Initial data

A Windows screen or window, a single screenshot or an action recording.

Work in Screph

Annotate elements and relationships, add descriptions, use OCR and image comparison if needed, then check the result.

Saved result

Main project and separately prepared script for Automation Runtime; selected data can also be transferred to Screph Code.

Boundary: To run automation, prepare a script, check the coordinate system and configure the input method. The project file itself does not execute actions.

ImplementedGeneral

Evaluating computer vision operations

Purpose

Choose operations and parameters for a region of interest, inspect the result and apply it to annotations.

Initial data

An image, screenshot, video, or media folder.

Work in Screph

Run a single method, sequence, or graph of methods and compare the source data with masks, outlines, recognized text, and metrics.

Saved result

Parameters, execution log and result files. Changes to the main markup are saved only after confirmation.

Boundary: Methods with models require suitable model files and dependencies; Tesseract OCR requires local installation.

ExperimentalAction CV

Finding targets in recorded actions

Purpose

Determine the visual target of a recorded keystroke or mouse button.

Initial data

Action recording and associated footage before, during and after the action.

Work in Screph

LLM can propose a plan, computer vision techniques are executed locally, and the result is accepted, modified, or rejected by the user.

Saved result

The proposed target, its source information and the user’s decision, saved alongside the original recording.

Boundary: The result requires human verification. The original action record remains unchanged.

ExperimentalIndustrial / UAV

Industrial and UAV workflows

Purpose

Explore images with domain data types for defects, measurements, changes, and trajectories.

Initial data

An industrial image or UAV frames with available temporal and spatial data.

Work in Screph

Selecting a mode, running methods, comparing the result with the original data, and confirming a supported markup change.

Saved result

Basic markup and additional materials of the selected mode.

Boundary: These processes are not yet complete and do not control equipment, production, missions, or autopilot.