Computer vision in Screph
In Screph, you can configure processing using example images: annotate objects, choose operations and parameters, and check the results. You can use the prepared materials when working on the program’s code.
Available operations, markup types, and save targets depend on the program mode. Statuses and limitations are listed under “Program modes”.
From annotating an example to running a program
In Selector, use Canvas to select objects and add distinctive features, descriptions and relations. The main Computer vision panel receives the image of the selected area from the active canvas view. Use this example to configure how to find or analyze the objects of interest. The entire image can also serve as input.
The visual builder for sequences of computer vision methods lets you choose a classical algorithm or an available model-based operation, change parameters, and build a chain or graph. You can configure processing yourself or with a connected agent. Python code operations can include your own logic. Dataset-based model training takes place in the Training section.
Saved objects, descriptions and selected materials are passed to Screph Code. There, the user or coding agent connects the configured computer vision workflow to program logic: acquiring new frames, calling the saved workflow and using its outputs. The editor supports working with the selected operations’ code and extending the solution with custom logic.
- Open an example image, annotate the objects of interest and select the region to configure.
- Select one operation, a processing chain, or a processing graph.
- Configure settings and run preview.
- Compare source material, result, mask, contours and metrics.
- Attach the desired result to the project or confirm a supported markup change.
- Test the workflow on other frames, save the project and use the prepared materials in Screph Code.
Annotations describe what to look for; the configured workflow defines how to find it. Testing on new images helps refine the solution before using it in a program.
Source and area of interest
An operation receives the active image or selected area. Verify that the area belongs to the intended frame and that its scale and coordinates match the canvas display.
The result can be shown next to the source, on top of it, or as a separate mask. Selecting a view does not change the saved markup.
Operation categories
| Category | Examples |
|---|---|
| Preparing the image | Brightness, contrast, blur, resize, crop, rotate and straighten. |
| Edges and regions | Thresholding, boundaries, contours, components, morphology and segmentation. |
| Matching and motion | Pattern search, comparison, alignment, optical flow and tracking. |
| OCR | Text recognition using a local Tesseract installation. |
| Model-based operations | YOLO, SAM, OmniParser |
The actual availability of an operation depends on the installed components, model files, and selected program mode.
One operation
Suitable for one transformation or analysis. After changing a parameter, refresh the preview and compare the result with the source. Saved parameters belong to the selected operation and material.
Processing chain
A linear sequence passes the result of each step to the next. It is suitable for predictable, branch-free processing such as cropping, contrast enhancement, thresholding, and edge extraction.
Check intermediate steps: an error in early processing affects all subsequent results.
Processing graph
A graph lets you branch processing, combine compatible results and select individual outputs for viewing. Connections can only link compatible data types.
- Give the steps clear names and test each output separately.
- For a complex graph, start with a small working part and add branches gradually.
- The experimental graph may differ in the available operations from the linear sequence.
Preview, save and apply
| Action | Output |
|---|---|
| Preview | Shows the result of the current parameters without changing the main markup. |
| Attach result | Saves the selected mask, image, or other result next to the project and associates it with an object. |
| Apply change | After review, adds or changes a supported object in the main annotations. |
Reviewing the proposed change
An operation proposal can be accepted, edited, rejected, or recalculated. Before applying it, check the source frame, object boundary, markup type, and absence of duplicates.
Industrial and UAV support a limited set of domain-specific changes. They remain experimental and do not provide a production-ready review and export workflow.
Video and temporal results
For videos and frame sequences, operations can produce observations and trajectories over time. Check these results across the entire selected range.
Hiding a result from the interface does not necessarily delete its files. Before moving a project, check its linked sequence material and save the required results with the project.
Models and dependencies
YOLO, SAM, and OmniParser require suitable model files and prepared dependencies. An installed model is not selected automatically for all operations.
Typical problems
- Empty result: check the source, area of interest and parameter range.
- Invalid coordinates: check scale, cropping and selected view.
- Operation unavailable: check the dependencies and model files.
- The project has become large: remove unnecessary temporary results and attach only necessary materials.
Operation selection and result freshness
The catalog depends on the task, inputs, and program mode. An operation card shows required parameters, the model, and any unavailability reason. Choosing an algorithm, configuring a step, and starting a run are separate actions; an installed model does not start processing by itself.
Changing the source, region, or parameters requires a new run before applying a result. Stop cancels the current work; a late response from an earlier run cannot change the current annotation. A saved result can be inspected separately, but viewing it does not establish its suitability for a new frame.
To compare variants, select the source material, operation, and parameters to vary. Compare results on the same input, then check the chosen variant with a regular run. Evaluate it on independent data too; a recommendation or one good image is insufficient.
Frame sequences: runs and results
- Select the source and range in Capture, then open Frame sequences. Set the direction and frame stride. The playhead position does not replace the start of the selected input.
- Selecting an operation reveals its settings within the workspace. Set an initial box for a tracker, or select a compatible calibration for visual odometry. Run becomes available after the listed blockers are resolved.
- A run fixes its inputs and settings. Progress and Stop are shown during execution. Switching projects cancels current work; a result from the previous source is not transferred to a new selection.
Processing uses a frozen frame range. Direct execution on a live stream, automatic retry, and synchronized processing of multiple streams are unavailable in this workflow. Open or record the source in Capture first.
Results may contain tables, plots, masks, trajectories, and transforms. Selecting a frame, pair, or window links the view to the timeline. Image coordinates and spatial pose are different; unscaled monocular visual odometry does not provide distances in metres.
Sequence history and A/B comparison
Select a completed current run as A, run variant B, and compare them on the same frozen input. Result fields, units, coordinates, and time must match. An unavailable comparison reports its reason; incompatible outputs are not converted into a numeric difference.
Save experiment links two runs and the selected frame, pair, or window. After the project is saved, the record is available in history with its verified material. Viewing an old record does not rerun processing or automatically change editor settings.
After parameters change, historical data remains readable but is no longer the current result. Returning to previous values does not activate it automatically. Applying it requires explicit restoration of a compatible context and fresh preparation; missing or damaged material blocks reading.