Automation Runtime executes prepared GUI workflows from a reviewed Screph project. It loads elements and templates, detects visual or text states, waits for defined conditions and sends explicit commands through the selected input backend.
1. Where execution begins
Project JSON describes a target and its context; it is not a self-running bot. Execution begins only when you start a script or an Automation Manager command. The target application must first be open, visible and in the expected display configuration.
Save and review the project in Screph.
Under Settings → Dependencies, prepare the script/CV runtime and the OCR runtime when text is involved.
Under Settings → Input emulation, select and test a backend.
Run the workflow against a safe test target first and watch steps/screenshots in Automation Manager.
2. Loading and validating a project
validate_project(path) returns an (ok, errors) pair; calling it without checking the result is not sufficient. load_project(path) does not run that validation automatically: it reads elements, projects coordinates, registers the first image_path of each element as a template and returns a GuiBotController plus a list of GuiElement.
from automation_runtime import (
load_project,
validate_project,
wait_and_click_image,
wait_until_gone_image,
)
project_path = "projects/my_ui_project.json"
ok, errors = validate_project(project_path)
if not ok:
raise RuntimeError("Project is not runnable:\n" + "\n".join(errors))
bot, elements = load_project(project_path)
target = next(item for item in elements if item.id == "start_button")
if not wait_and_click_image(bot, target.id, timeout=20, desc="Start"):
raise RuntimeError("start_button was not found")
if not wait_until_gone_image(bot, target.id, timeout=10):
raise RuntimeError("Start action did not reach the expected next state")
Using stable element IDs and screen matching is generally more robust than a fixed sequence of absolute clicks. Templates still depend on scale, theme, window state and source-image quality. The runtime does not verify that a visible match belongs to the expected process or window.
3. Main API groups
Images and waits
wait_and_click_image, double_click_image, right_click_image — find a template and request an action; the workflow verifies the postcondition.
wait_until_gone_image — confirm that a state is absent before timeout.
find_all_images, wait_for_any_image, wait_for_all_images — work with lists and alternative states.
Text and OCR
wait_and_click_text — wait for text and click its detected region.
wait_for_text_change, ensure_text_contains — verify a change or expected content.
retry, take_screenshot and action logging for diagnostics and repeatability.
wait_for_all_images accumulates templates found at different polling moments; true does not prove that all of them were present in the same frame.
wait_until_gone_image confirms absence during polling but does not require the template to have been observed first.
wait_and_click_image and wait_and_click_text first detect the target and then search again for the click. Their true reflects the first detection, while the current wrapper does not check the second search/click return value. Verify the expected postcondition with a separate helper.
4. Coordinates and selected monitor
Public input helpers receive global physical coordinates on the virtual desktop. Projection of project geometry depends on the stored coordinate space:
Source
Loader behavior
What it does not prove
screen_physical_px
Accepts bounding boxes as already global.
The capture target is not resolved again; only the input guard checks topology when a pointer action occurs.
screenshot_raw_px + monitor
Finds the current monitor by stable device name and scales the raw screenshot into its current rectangle.
A resolution change causes rescaling rather than requiring identical geometry.
screenshot_raw_px + window
Resolves the current HWND, checks stored PID/class when present and scales into the current window rectangle.
Does not verify foreground state, visibility, occlusion, title or window contents.
screenshot_raw_px + region
Uses the stored physical region rectangle and scales raw coordinates.
A region has no live window/monitor identity to revalidate.
The monitor selected in input settings is a separate pointer-target boundary, not an origin offset. all permits any connected monitor but still rejects points outside outputs or in virtual-desktop gaps; a specific monitor rejects points outside its rectangle. This check runs during the actual move/click and may reject a project that passed validate_project.
5. Input backend
The user selects the backend in settings. The runtime does not silently switch to another input method.
Arduino Leonardo HID — the primary hardware mode using a control HID channel and matching firmware. The flash action puts a Leonardo/ATmega32U4 into its bootloader and writes the selected .hex through avrdude; this modifies an external device rather than merely testing a connection.
Legacy Serial — compatibility with the older Leonardo COM firmware.
FakerInput Virtual HID — a software HID keyboard and mouse on Windows. Selecting the mode for the first time immediately verifies the bundled MSI's SHA256, Authenticode signature and signer, then installs a system driver through one UAC request; a manual restart may be required afterward.
Arduino and FakerInput move the pointer using relative HID deltas. The runtime reads system speed and acceleration and warns about risk; it does not change them automatically. A separate explicit --apply-recommended CLI command persists neutral settings in the current user's profile and does not restore the previous values itself. Always run the input test before a real workflow.
Tesseract and ocr.runtime are required only for reading, finding and validating text. Image matching, coordinate actions and ordinary text input do not require OCR. If OCR is disabled or broken, text helpers return an empty/false/none result and EVT progress where the helper emits it rather than switching to image matching.
find_text/wait_and_click_text compare the query with one OCR token, so a multi-word phrase is not assembled across adjacent tokens.
wait_for_text_change returns a new non-empty value only after three consecutive identical OCR readings.
ensure_text_contains joins the region's tokens into a string and performs a case-sensitive substring check.
Automation Manager runs the selected workflow as a child Python process with the current Windows user's permissions and the script directory as its working directory. The runtime emits structured EVT events to stdout: current step, progress, screen snapshot, template and detected match. The interface displays them alongside the execution log.
This is not a sandbox. The process inherits Screph's environment, including credentials available there, receives parameters as AM_* variables and can use files, network and devices available to the user. Review the code, its dependencies, environment and any external processes it creates before running it.
Before starting, review the script, working directory and parameters.
Do not manually interact with the target window during a coordinate-driven workflow.
If state diverges, stop the process, keep the log/screenshot and fix the project or precondition; do not blindly increase timeouts.
Stop first terminates the direct child, waits up to five seconds and then kills that process. Processes spawned by the workflow itself may continue running and must be checked separately.
8. Troubleshooting
Template not found: check scale, theme, window state, ROI and threshold.
Text not found: check ocr.runtime, Tesseract language and region quality.
Pointer moves incorrectly: check monitor topology, physical coordinates, the monitor guard and Windows acceleration settings.
Input does not start: open the input-emulation test and repair the selected backend; there is no hidden fallback.
