CV GUI and Pipeline Guide

1. What the CV GUI Does

The CV GUI helps you detect regions in an image, build masks and contours, read text, combine multiple processing steps, and turn the result into canonical project markup. The selected program mode determines the available task and candidate types.

• Simple mode: choose one method and tune a small set of parameters.
• Linear pipeline: build a straight sequence of steps.
• Advanced pipeline: build a graph with methods, branching, merges, and runtime inspection.

2. Basic Workflow

1. Open an image or select an element/ROI on the canvas.
2. Open the CV/AI panel and choose a processing mode.
3. Enable live preview for quick iteration or keep manual Apply for heavier methods.
4. Check the preview and the overlay on the canvas.
5. Choose the required boundary: archive the technical run, attach a mask/result to an element's image series, or explicitly create canonical elements.

Practical tip: if you are unsure where to start, first reduce the area of interest (ROI), then run color or segmentation processing. It is faster and usually produces less noise.

Multiple presentations of one project

The layout control in the menu-bar toolbar can show one A pane, A/B side by side or stacked, three panes with A on the left and B/C on the right, or a 2×2 A/B/C/D grid. These are not project copies: canonical markup, selection and project undo/redo are shared, while visual material and pane-local display settings may differ.

• Each pane's settings let you change its label and select Source, a working CV result, a mask or diagnostic artifact, its display mode and crop source.
• The Activate control chooses the pane used by Canvas tools and, by default, the ROI pixel source. Activation of a stale, unmapped or read-only presentation is rejected; if the current active pane becomes incompatible, ownership returns to A.
• Linked panes synchronize zoom, viewport center and the canvas-space cursor. A always belongs to the linked group; B/C/D can be unlinked. A read-only pane may drive navigation without owning project interaction.
• The Crop source field explicitly chooses the current instance or the original Source.
• A mask, diff/composite, display-only transform or stale/unmapped result is not a valid ROI input. The operation is blocked instead of silently falling back to Source.
• Pane A cannot be closed. B/C/D are closed explicitly without deleting a presentation; a smaller layout never removes them silently. Global Reset assigns Source to A, activates it, closes the other panes and returns to the single layout.

The Result, Grayscale, Invert and Pseudocolor options change display only, not the material or recipe. Keep Source in an adjacent linked pane for comparison.

Full-image preprocessing

1. Open Source or an existing working presentation in the target pane.
2. Press Configure CV; for Source, a working processed_raster presentation is created.
3. Under Image analysis → Presentation processing, choose a method or linear pipeline and press Update presentation.

Each working presentation has its own parameters. Its catalog always operates on the Full image; methods requiring an ROI, click, timeline, reference image or a project-side mutation are excluded.

3. Main Parts of the Panel

• Processing Mode: choose a method or a pipeline mode.
• Method Parameters: dynamic form with settings for the selected method.
• Preview: shows what the method or pipeline produced.
• Overlay: overlays masks and contours on the canvas so you can compare them with the original image.
• Presets: save useful parameter combinations.
• Result Saving: separate actions for the runtime archive, the selected element's image series and canonical markup.
• Live preview / Auto / AI: Live drawing recalculates after parameter changes; 🔮 Auto tunes parameters from the image; AI selection lets the LLM propose an algorithm. For large frames and model-backed methods, prefer manual Refresh preview.
• Session and actions: Apply runs analysis and, in the main panel, normally attaches or replaces a cv_result entry in the selected element's image series; Save writes a technical run to the named CV session; Add as elements creates canonical elements, and Add mask to results appends a separate mask entry.
• HSV eyedropper: samples a preview color and fills H/S/V ranges with dH/dS/dV tolerances.

Text / OCR uses the local Tesseract supplied by ocr.runtime. YOLO, the three SAM backends and OmniParser separately require a ready ml.runtime and suitable local model assets. A method appearing in the list means its UI is available, not that the runtime or weights are already ready. See model and weight management.

The LLM settings button opens the CV role profile. Model-backed and OCR methods require their matching runtime components. LLM connections · Dependencies

4. CV Result Lifecycle

CV candidates, canonical markup and FeatureMatch

A CV method first emits a result/evidence: preview, detections, contours, a mask and metrics. This is not yet a Selector object. Review binds the candidate to a markup_candidate_id and creates an accepted decision; a patch and apply preview follow. Only an explicit apply, after target, revision and duplicate checks, mutates the canonical project.

Candidate review/apply and FeatureMatch creation are different operations: the first promotes evidence into canonical markup, while the second records a concrete occurrence of an existing feature. They are separate project-JSON records and separate undo/redo mutations.

A verified bundle stores the source snapshot when available, execution recipe, config snapshot, metrics, standard and extra outputs, SHA-256/size metadata, coordinate contracts and a write report. Its manifest is published only after required references pass write and readback.

A named CV session is a runtime archive under ScrephData/screen_selector/cv/results, not a second project format. The recent-session list keeps at most ten IDs, but this is not file retention: old session directories are not automatically pruned here. When the canonical project is saved, Selector imports only CV entries attached to element image series, copies their artifacts into <name>_artifacts, and writes a project-local result manifest. An unattached technical archive is not included in the package.

How CV artifacts enter the project package →

5. Video CV and Temporal Tracking

The Video CV workspace analyzes a Capture frame sequence with durable source and timeline identity. It can process a selected range, the current clip, the complete timeline, an 11-frame window around the playhead, or a live stream; direction is selected separately. Pre-roll and post-roll expand offline bounds, frame stride subsamples the input, and 100/75/50/25% processing scales reduce load while results are normalized back to source coordinates.

• The analysis ROI and exclusion areas are bound to the selection context. If selection changes, execution is blocked until the context is refreshed; a stale ROI is not reused silently.
• The live reader has a four-frame queue and an explicit reject, drop-oldest or drop-newest backpressure policy. Stop/cancel closes the reader; offline gaps are diagnosed and reset the temporal processor.
• Overlay layers include motion, tracks, accepted markup, detections, optical flow, events and candidates. Run history can restore parameters, rerun an analysis and navigate to an issue.

Temporal providers

The built-in registry includes UAV Sparse Camera Stabilization, OpenCV MOG2 Motion, MIL/CSRT/KCF trackers, Farneback Optical Flow, Activity Events, Native Scene Difference and conditional PySceneDetect Content. An unavailable provider is disabled with a reason and repair action: CSRT/KCF require a suitable opencv-contrib build, while PySceneDetect needs its dependency. SAM 2.1 Prompt Mask Propagation is currently an unavailable internal target for a future isolated worker, not a normal working backend.

Runs, track apply and editing

1. Run a provider and wait for completed, partial, cancelled or failed status. The input snapshot, configuration, provenance, coverage, issues and observations are stored in the temporal run store.
2. For a tracker run, select a track and prepare the apply preview. The plan reports observations, keyframes, lost states and the expected revision while preserving manual or protected observations from an existing track.
3. Confirm apply. The project must already be saved or opened because the operation writes the Selector-owned <project>.temporal-markup.json sidecar with a revision guard and project undo/redo.

After apply, you can add a manual keyframe from the current ROI, remove a keyframe, trim or terminate, split, mark occluded/outside, undo/redo, safely merge without ambiguous overlaps, reassign a track ID and retrack from a manual keyframe. This is an explicit prepare/apply boundary for temporal markup, not automatic acceptance of every observation.

Exporting the active run

• Neutral JSON contains run/provider/status, hashes, coverage, provenance, configuration, observations and optional codegen IR from the persisted recipe. It is data, not an executable script.
• Annotated preview creates an MJPG AVI using the current start/end fields and timeline mapping, drawing track and motion bounding boxes. It requires a positive Capture FPS, every requested frame and a working OpenCV VideoWriter.

Both exports are artifacts of the selected run, not a dataset export or project save. Temporal data files and boundaries →

6. Actual Method Catalog

The main panel registers 28 user-facing single methods. These are actual selectable entries, not a count of unique mathematical algorithms: MobileSAM, FastSAM and EfficientSAM, for example, are three backends in one promptable-segmentation family.

• Filters and color: Edges (Canny), Color (HSV), Color Mask (K-Means).
• Segmentation and masks: GrabCut, Watershed, SLIC Superpixels, YOLO segmentation, MobileSAM, FastSAM, EfficientSAM.
• Text and structure: Text / OCR, OmniParser, Hough Lines, Hough Circles, MSER, Connected Components, Contours, Adaptive Threshold.
• GUI, reference and temporal: Template Match, Point Region Grow, Shape Analyze, Line/Rect Layout, Feature Match, Before/After Diff, Align To Reference, Target Track (KLT), UI Layout (Local), Visual ReID.

Linear pipelines and method-flow graphs additionally register 14 utility nodes: Grayscale, Brightness/Contrast, Gamma, Gaussian Blur, Median Blur, CLAHE, Resize, Crop/Pad, Rotate, Deskew, Apply Mask to Image, Invert Mask, Morphology and Extract Contours. They exist to build data flows and therefore are not duplicated in the single-method list.

What the program mode changes

• General: the 28 shared methods and 14 pipeline nodes form the neutral image-analysis catalog.
• GUI Automation: uses the shared CV capabilities and adds GUI types, markup refs, a review draft and a promotion patch. This is a semantic/authoring layer, not another 42 separate methods.
• Industrial / UAV: have mode-specific pipeline operations, but their catalogs mix CV, preprocessing, orchestration, decisions and export nodes. These workspaces are experimental, so the registered-step count must not be read as a count of production-ready algorithms.

7. Linear Pipeline

The linear pipeline is the best option when you need a predictable flow without branching: for example, detect edges, clean the mask, then extract contours. It is easier to learn and easier to reuse across projects.

• Each step runs from top to bottom.
• You edit one step at a time.
• Step previews help you see where noise appeared or where an object was lost.
• If the chain stays linear, it is easy to maintain and later move into advanced mode.

8. Advanced Pipeline: Detailed Guide

The advanced pipeline is designed for scenarios where a simple linear chain is no longer enough. It lets you build a full processing graph: split into branches, merge results, keep a dedicated preview output, and inspect the runtime trace step by step.

When to Switch to Advanced Mode

• You need to compare two different branches for the same image.
• You need to choose between variants conditionally.
• You want to control the final result and the preview separately.
• You need the runtime inspector to see the payload of each node and connection.

Important: the advanced mode is still experimental. It keeps its own draft state, while the linear pipeline remains the main format until you explicitly apply an exportable graph back.

Advanced Pipeline Screen Layout

• Top Action Bar: quick actions like Quick Add, Snippets, Auto Layout, Repair Graph, Apply to Linear.
• Graph Canvas: the main editing area with nodes, connections, comments, and groups.
• Right-side Inspectors: information about the selected item, payload, pinned inspector, and selection preview.
• Runtime Debug Area: execution history, step scrubber, diff, pinned runtime step, and preview history.
• Validation / Search / Recommendations: jump to issues, search the graph, and follow suggested fixes.

Node Types

• Source: input image, point prompt, constants.
• Method: actual CV methods: edge detection, segmentation, OCR, contour extraction, and so on.
• Control: conditional nodes and merge nodes used to choose or combine branches.
• Utility: helper nodes such as reroute nodes to keep the graph readable.
• Sink: final output targets for result and preview.

A typical graph always includes at least Input Image and Result Output. They define the start and the explicit end of the flow. If your graph has multiple branches and no explicit result output, the runtime may treat it as ambiguous.

How to Build a Graph in Practice

1. Open the Advanced tab next to the linear editor.
2. Add nodes via Quick Add or the Add Node button.
3. Connect ports by dragging from an output to a compatible input.
4. If the graph becomes large, use reroute nodes and comment/group frames.
5. For branching logic, insert If and Merge nodes.
6. After editing, check the validation panel and summary on the right.

Recommended starter pattern: `Input Image -> one or two method nodes -> Result Output`. First make sure this skeleton works, then add branching, merge nodes, and a separate preview sink.

Editing and Navigation

• Delete removes selected nodes, connections, or groups.
• F2 renames the selected node.
• Ctrl+D duplicates the current selection.
• Tab or Space opens Quick Add.
• A runs auto layout.
• Ctrl+0 fits the graph into view.

Runtime Inspector: How to Read Graph Execution

After execution, the advanced pipeline stores a runtime snapshot. It shows not only the final image but also what happened in each node: which inputs were available, what the node returned, which branch was selected, and where warnings or errors appeared.

• Runtime scrubber: switch between the full view and individual runtime steps.
• Current / Pinned step: useful for comparing two states or two execution branches.
• Selection preview: shows the preview for the selected node or edge.
• Payload inspector: helps you understand what kind of payload flows through a connection: image, mask, contours, detections, or text.

Runtime shortcuts: `Alt+Home` jumps to the beginning of the trace, `Alt+Left/Right` moves backward/forward, and `Alt+End` returns to the full view.

Drafts, Sync, and Applying Back to Linear

Advanced mode has its own editable draft state. As long as the graph contains branching, utility/control nodes, or other non-linear structures, it may remain an advanced-only draft. If the configuration stays linearly exportable, you can press Apply to Linear and convert it back to the regular linear pipeline.

• Resync from Linear rebuilds the graph from the current linear pipeline.
• Apply to Linear writes the graph back into the linear format when possible.
• Repair Graph suggests safe fixes when the structure is incomplete or broken.

9. Performance Tips

• For heavy graphs, turn off live preview and run Apply manually.
• Limit the ROI before running heavy processing if you do not need the full screen.
• Use reroute nodes and comment frames to avoid turning the graph into unreadable spaghetti.
• After major changes, run Auto Layout and check the validation summary.

10. Common Problems and What to Do

• Preview is empty: make sure the graph ends in Result Output or a Preview sink and that nodes receive compatible inputs.
• Graph cannot be applied back to linear: the usual reason is branching, merge nodes, utility/control nodes, or another non-linear fragment.
• Too much noise: reduce the ROI, add mask cleanup, or split the task into several steps instead of using one generic method.
• The result is unstable across frames: lock the area, reduce the number of branches, and inspect the runtime trace step by step instead of looking only at the final preview.