Use cases

Supported Screph workflows

The current version supports the four workflows below. Each one lists its input, the work performed in Screph, the saved output, and its limitations.

ImplementedGeneral / GUI Automation

Preparing a GUI automation script

Purpose

Describe interface elements, their characteristics, states and relationships for subsequent automation.

Initial data

Screen or window Windows, a single snapshot or recording of actions.

Work in Screph

Marking of elements and connections, descriptions, if necessary OCR and comparison of images, then checking the result.

Saved result

Main project and separately prepared script for Automation Runtime; selected data can also be transferred to Screph Code.

Boundary: The project is not executed automatically. To run, you need a separate script, a coordinate context, and a customized input method.

ImplementedGeneral

Purpose

Link the area of interest, method parameters, visual result and decision to apply to the markup.

Initial data

An image, screenshot, video, or media folder.

Work in Screph

Run a single method, sequence, or graph of methods and compare the source data with masks, outlines, recognized text, and metrics.

Saved result

Parameters, execution log and result files. Changes to the main markup are saved only after confirmation.

Boundary: Methods with models require suitable model files and dependencies; Tesseract OCR requires local installation.

ExperimentallyAction CV

Purpose

Determine the visual target of a recorded keystroke or mouse button.

Initial data

Action recording and associated footage before, during and after the action.

Work in Screph

LLM can propose a plan, computer vision techniques are executed locally, and the result is accepted, modified, or rejected by the user.

Saved result

The proposed goal, information about its origin and the user's solution next to the original entry.

Boundary: The result requires human verification. The original action record remains unchanged.

ExperimentallyIndustrial / UAV

Purpose

Explore images with domain data types for defects, measurements, changes, and trajectories.

Initial data

Industrial image or footage UAV with available time and space data.

Work in Screph

Selecting a mode, running methods, comparing the result with the original data, and confirming a supported markup change.

Saved result

Basic markup and additional materials of the selected mode.

Boundary: These processes are not yet complete and do not control equipment, production, missions, or autopilot.