LLM connections and speech recognition

1. LLM connections

Open Settings → LLM settings. Each connection stores its provider type, endpoint, selected model and access parameters. Supported connections are:

• OpenAI, Anthropic, Gemini, OpenRouter;
• Ollama and LM Studio;
• a custom OpenAI-compatible endpoint;
• LiteLLM Proxy — shown in advanced mode.

Requests run through the shared LiteLLM runtime. Saving a predefined connection row immediately starts an access/catalog check. In the generic profile editor, Check first saves the current form and API key and then runs the probe; it is not read-only. A credential, endpoint or capability error does not reroute the request to another profile.

Verified access/model state lives only in the current connection-service instance. After a new launch, a configured model may be shown as access not verified. The profile's manual model is appended to a successful catalogue even when the provider did not list it. For the cv role, the list uses a name/metadata heuristic but falls back to the full list when no vision candidate is found; presence in the combo and a general test do not prove image or structured-response support.

2. Assigning profiles to roles

One connection does not have to serve the whole product. A role selects a profile/model for a specific path; the role itself does not create a separate capability or safety policy:

• assistant — the built-in AI Assistant;
• cv — CV auto-tune, pipeline planning and Action CV/VLM;
• codegen — Screph Code;
• semantic_selection — semantic element selection;
• test — connection and model testing.

When the first profile is saved, empty cv, test and semantic_selection roles are assigned to it automatically; codegen is included only when that UI is enabled. The assistant role is not filled automatically. This is initial assignment, not runtime fallback. Deleting a profile clears every role that references it.

If a role profile is disabled, deleted or unavailable, the related operation fails clearly. Resolution order is explicit request profile/model, then role profile/model, then the profile's default model. Test the role separately even when the base connection passes its general check.

3. Keys, local operation and data transfer

• LLM and current voice API keys are stored through the system keyring; in the Windows build this is Windows Credential Manager under service name Screph. The ordinary profile payload in ScrephData/settings_modules/settings.json contains provider kind, model, endpoint, timeout and roles, but not the API-key value.
• If the keyring is unavailable, saving a secret fails: there is no plaintext fallback to ordinary settings JSON. Environment variables remain an explicit runtime source and may be inherited by child processes.
• Ollama and LM Studio can keep requests on the local machine when the service and model are actually running locally.
• A cloud profile may send the prompt, selected context and images to its provider. The exact payload depends on the capability and its privacy/attachment settings.
• In AI Assistant, image and artifact sharing is off by default. Action CV defaults to block_unredacted, which blocks the provider call; roi_only paints over everything outside the ROI, while allow_unredacted explicitly permits the original frame. Its prompt also includes action type, time and source, key or button and coordinates when present, the ROI hint, frame metadata and hashes, the response contract and the allowed method catalogue.

When a shared LLM request actually receives an image_path, Screph reads the file, converts it to RGB JPEG, downsizes it to at most 1024×1024 at quality 85 and sends a base64 data URL with detail=low. For a CV/image request, the profile timeout is replaced by a 120-minute transport timeout; retry count still comes from the request/profile setting and is passed to LiteLLM.

Successful shared-runtime responses append a local row to ScrephData/llm/usage_events.jsonl containing feature, profile, model, tokens, calculated cost, latency and status. Prompt/response text is not written there. There is no automatic retention or clear button, and exceptions raised before a response exists are generally absent, so it is not a complete billing or error ledger.

Where secrets are stored and how to remove them →

4. Speech recognition

Open Settings → Speech recognition. The section includes settings and test tabs; voice annotations use the selected recognizer. Provider-aware adapters are implemented for:

• Vosk — local batch and streaming modes; requires voice.local, a microphone and a matching language model.
• Yandex — SDK, streaming or async depending on the available configuration.
• Google Cloud — explicitly selected cloud recognition and its credentials/capabilities.
• OpenAI — recognition through its configured provider path.

Not every provider supports every mode. The loader checks the selected capability and reports it as unavailable when a streaming/async/backend path is not implemented or available for that configuration; it does not replace the provider with local Vosk.

5. Configure and test voice

1. Select a recognizer and mode. For Vosk, prepare voice.local on the Dependencies tab first.
2. Enter the model, region, endpoint or credentials required by that provider.
3. Allow microphone access in Windows and select the correct input device.
4. On the Recognition test tab, record a short phrase and verify transcript, language and latency.
5. Enable voice annotation in a working project only after that test.

6. Quick diagnostics

• Model is not listed: check the endpoint and connection test; for a local service, verify that it is running.
• CV uses another connection: check the cv role assignment, not just the last edited profile.
• Vosk does not start: re-probe voice.local, the language-model path and microphone permission.
• Cloud speech returns a capability error: select a supported mode for the provider; the error does not mean Screph switched to batch or another service.