# Adjutant Agent Source: https://docs.openagents.com/adjutant-agent DSPy-native agent used in Autopilot Desktop. ## Overview Adjutant is a DSPy-native agent for the Autopilot ACP system. Unlike Codex or Gemini adapters that wrap external binaries, Adjutant runs natively in Rust with first-class dsrs support. **Status:** The repo includes Adjutant, but current desktop builds connect only to Codex by default. ## Key Characteristics * Native Rust implementation (no external process spawning) * DSPy signature pipelines for planning and execution * Unified event stream + signature-driven UI updates ## Plan Mode Signatures Plan mode signatures live in `crates/dsrs/src/signatures/plan_mode.rs` and are exposed through the signature registry: * TopicDecompositionSignature * ParallelExplorationSignature * PlanSynthesisSignature * ComplexityClassificationSignature * DeepPlanningSignature * ResultValidationSignature ## Optimization Loop Plan mode examples and manifests are stored locally: * `~/.openagents/autopilot-desktop/training/plan_mode.json` * `~/.openagents/autopilot-desktop/manifests/plan_mode/` * `~/.openagents/autopilot-desktop/optimization/` Optimizers: MIPROv2, COPRO, GEPA. ## Integration Surface * **Backend**: `crates/adjutant/` (DSPy pipelines) + `crates/autopilot_app/` (desktop wiring) * **UI**: `crates/autopilot_ui/` (WGPUI surfaces) * **Events**: emits app/server events consumed by the desktop runtime ## Why It Matters Adjutant is the pathway to programmable, optimizable behavior in Autopilot. It turns planning into a first-class, measurable system instead of ad hoc prompts. # AI Gateway Setup Source: https://docs.openagents.com/ai-gateway Local bun server that powers Adjutant's DSPy pipelines. ## Overview Adjutant uses a local Bun server that fronts the Vercel AI Gateway. This provides one API surface for multiple model providers and enables local control inside Autopilot Desktop. **Note:** This is only needed for Adjutant workflows. Codex-only usage does not require the AI Gateway. ## Prerequisites * Bun runtime * AI Gateway API key (Vercel) ## Environment Variables Create or update `.env` in the repo root: ```bash theme={null} AI_GATEWAY_API_KEY=your_vercel_ai_gateway_api_key_here AI_GATEWAY_BASE_URL=https://ai-gateway.vercel.sh/v1 AI_SERVER_PORT=3001 AI_SERVER_HOST=localhost DEFAULT_LLM_PROVIDER=anthropic DEFAULT_LLM_MODEL=claude-sonnet-4.5 FALLBACK_LLM_MODEL=openai/gpt-4o DSPY_LM_ENDPOINT=http://localhost:3001 DSPY_MAX_TOKENS=4096 DSPY_TEMPERATURE=0.7 ``` ## How It Starts Autopilot Desktop starts the AI server during app setup. The server runs locally and exposes OpenAI-compatible endpoints: * `POST /v1/chat/completions` * `POST /v1/embeddings` * `GET /health` ## Model Routing Routing can choose a primary model with fallbacks (planning vs exploration vs synthesis), allowing fast paths and recovery when a provider fails. ## Troubleshooting * Port conflicts: change `AI_SERVER_PORT` * Invalid key: verify `AI_GATEWAY_API_KEY` * Health check: `curl http://localhost:3001/health` # API and Events Source: https://docs.openagents.com/api-events App-server events and desktop runtime messages used by Autopilot. ## Overview Autopilot uses an in‑process desktop runtime. The app sends user actions to the Codex app‑server client and renders normalized event streams for UI updates. **Note:** Current desktop builds connect to Codex via the app‑server client. Other agent adapters exist in the repo but are not enabled by default. ## Core Messages * `UserAction::NewChat` — start a new session and thread * `UserAction::Message` — send a prompt to the active thread * `UserAction::Interrupt` — stop an active turn ## Session ID Flow Autopilot uses a two-phase session ID: 1. Temporary session ID from `connect_unified_agent` 2. Actual ACP session ID emitted via `SessionStarted` All messages must use the actual ACP session ID. ## Event Streams Normalized agent events are surfaced to the UI, including: * message/turn start and completion * tool call updates * token usage and rate limits For full type definitions, see the Autopilot Desktop API docs in the OpenAgents repo. # Create a Lightning invoice from the authenticated user's wallet. Source: https://docs.openagents.com/api-reference/agent-payments/create-a-lightning-invoice-from-the-authenticated-users-wallet api/openapi.json post /api/agent-payments/invoice # Create a Lightning invoice from the authenticated user's wallet. Source: https://docs.openagents.com/api-reference/agent-payments/create-a-lightning-invoice-from-the-authenticated-users-wallet-1 api/openapi.json post /api/payments/invoice # Create or import wallet for the authenticated user. Source: https://docs.openagents.com/api-reference/agent-payments/create-or-import-wallet-for-the-authenticated-user api/openapi.json post /api/agent-payments/wallet # Create or import wallet for the authenticated user. Source: https://docs.openagents.com/api-reference/agent-payments/create-or-import-wallet-for-the-authenticated-user-1 api/openapi.json post /api/agents/me/wallet # Get current wallet balance for authenticated user. Source: https://docs.openagents.com/api-reference/agent-payments/get-current-wallet-balance-for-authenticated-user api/openapi.json get /api/agent-payments/balance # Get current wallet balance for authenticated user. Source: https://docs.openagents.com/api-reference/agent-payments/get-current-wallet-balance-for-authenticated-user-1 api/openapi.json get /api/agents/me/balance # Get the authenticated user's Spark wallet metadata. Source: https://docs.openagents.com/api-reference/agent-payments/get-the-authenticated-users-spark-wallet-metadata api/openapi.json get /api/agent-payments/wallet # Get the authenticated user's Spark wallet metadata. Source: https://docs.openagents.com/api-reference/agent-payments/get-the-authenticated-users-spark-wallet-metadata-1 api/openapi.json get /api/agents/me/wallet # Pay a BOLT11 invoice from the authenticated user's wallet. Source: https://docs.openagents.com/api-reference/agent-payments/pay-a-bolt11-invoice-from-the-authenticated-users-wallet api/openapi.json post /api/agent-payments/pay # Pay a BOLT11 invoice from the authenticated user's wallet. Source: https://docs.openagents.com/api-reference/agent-payments/pay-a-bolt11-invoice-from-the-authenticated-users-wallet-1 api/openapi.json post /api/payments/pay # Send sats to another Spark address from authenticated user's wallet. Source: https://docs.openagents.com/api-reference/agent-payments/send-sats-to-another-spark-address-from-authenticated-users-wallet api/openapi.json post /api/agent-payments/send-spark # Send sats to another Spark address from authenticated user's wallet. Source: https://docs.openagents.com/api-reference/agent-payments/send-sats-to-another-spark-address-from-authenticated-users-wallet-1 api/openapi.json post /api/payments/send-spark # Bootstrap an API user + bearer token (staging/automation). Source: https://docs.openagents.com/api-reference/auth/bootstrap-an-api-user-+-bearer-token-stagingautomation api/openapi.json post /api/auth/register This bypasses WorkOS email code auth and is intended for staging automation. Controlled by `OA_API_SIGNUP_ENABLED`. # Create a personal access token. Source: https://docs.openagents.com/api-reference/auth/create-a-personal-access-token api/openapi.json post /api/tokens Returns the plain-text token once. Store it securely; it cannot be read again from the API. # Get authenticated user context. Source: https://docs.openagents.com/api-reference/auth/get-authenticated-user-context api/openapi.json get /api/me Returns current user profile fields and recent chat thread summaries. # List personal access tokens. Source: https://docs.openagents.com/api-reference/auth/list-personal-access-tokens api/openapi.json get /api/tokens Returns Sanctum tokens owned by the authenticated user, including whether each token is the current one. # Revoke a specific personal access token. Source: https://docs.openagents.com/api-reference/auth/revoke-a-specific-personal-access-token api/openapi.json delete /api/tokens/{tokenId} # Revoke all personal access tokens for the authenticated user. Source: https://docs.openagents.com/api-reference/auth/revoke-all-personal-access-tokens-for-the-authenticated-user api/openapi.json delete /api/tokens # Revoke the current bearer token. Source: https://docs.openagents.com/api-reference/auth/revoke-the-current-bearer-token api/openapi.json delete /api/tokens/current # Create a new thread for an autopilot. Source: https://docs.openagents.com/api-reference/autopilot/create-a-new-thread-for-an-autopilot api/openapi.json post /api/autopilots/{autopilot}/threads # Create an autopilot resource. Source: https://docs.openagents.com/api-reference/autopilot/create-an-autopilot-resource api/openapi.json post /api/autopilots # List autopilots owned by the authenticated user. Source: https://docs.openagents.com/api-reference/autopilot/list-autopilots-owned-by-the-authenticated-user api/openapi.json get /api/autopilots # List threads for one autopilot. Source: https://docs.openagents.com/api-reference/autopilot/list-threads-for-one-autopilot api/openapi.json get /api/autopilots/{autopilot}/threads # Read one autopilot by id or handle. Source: https://docs.openagents.com/api-reference/autopilot/read-one-autopilot-by-id-or-handle api/openapi.json get /api/autopilots/{autopilot} # Stream a run via autopilot alias route. Source: https://docs.openagents.com/api-reference/autopilot/stream-a-run-via-autopilot-alias-route api/openapi.json post /api/autopilots/{autopilot}/stream Resolves the owned autopilot, ensures a compatible thread exists, and then forwards to the canonical chat stream runtime. # Update one autopilot. Source: https://docs.openagents.com/api-reference/autopilot/update-one-autopilot api/openapi.json patch /api/autopilots/{autopilot} # Create a new conversation thread. Source: https://docs.openagents.com/api-reference/chat/create-a-new-conversation-thread api/openapi.json post /api/chats Creates a conversation container and persists an initial thread title. # Get one conversation with messages and runs. Source: https://docs.openagents.com/api-reference/chat/get-one-conversation-with-messages-and-runs api/openapi.json get /api/chats/{conversationId} Returns the thread metadata plus current message/run snapshots for the authenticated user. # List authenticated user's conversation threads. Source: https://docs.openagents.com/api-reference/chat/list-authenticated-users-conversation-threads api/openapi.json get /api/chats Returns recent thread ids/titles sorted by update time. Use the `limit` query parameter to bound result size. # List events for a specific run. Source: https://docs.openagents.com/api-reference/chat/list-events-for-a-specific-run api/openapi.json get /api/chats/{conversationId}/runs/{runId}/events Returns the selected run plus ordered event payloads. Use `limit` to bound event rows. # List messages for a conversation. Source: https://docs.openagents.com/api-reference/chat/list-messages-for-a-conversation api/openapi.json get /api/chats/{conversationId}/messages Returns normalized message records in chronological order. # List runs for a conversation. Source: https://docs.openagents.com/api-reference/chat/list-runs-for-a-conversation api/openapi.json get /api/chats/{conversationId}/runs Returns run metadata sorted newest-first. Use `limit` to cap returned rows. # Stream a chat run over server-sent events. Source: https://docs.openagents.com/api-reference/chat/stream-a-chat-run-over-server-sent-events api/openapi.json post /api/chat/stream Expects the full message history and executes against the latest non-empty user message. # Stream a chat run over server-sent events. Source: https://docs.openagents.com/api-reference/chat/stream-a-chat-run-over-server-sent-events-1 api/openapi.json post /api/chats/{conversationId}/stream Expects the full message history and executes against the latest non-empty user message. # Create a seller paywall route and reconcile Aperture configuration. Source: https://docs.openagents.com/api-reference/l402/create-a-seller-paywall-route-and-reconcile-aperture-configuration api/openapi.json post /api/l402/paywalls # Delete (soft-delete) a seller paywall and reconcile Aperture configuration. Source: https://docs.openagents.com/api-reference/l402/delete-soft-delete-a-seller-paywall-and-reconcile-aperture-configuration api/openapi.json delete /api/l402/paywalls/{paywallId} # Get one L402 transaction receipt by event id. Source: https://docs.openagents.com/api-reference/l402/get-one-l402-transaction-receipt-by-event-id api/openapi.json get /api/l402/transactions/{eventId} # Get paywall aggregation by host/scope. Source: https://docs.openagents.com/api-reference/l402/get-paywall-aggregation-by-hostscope api/openapi.json get /api/l402/paywalls # Get settlement-focused paid receipt summary. Source: https://docs.openagents.com/api-reference/l402/get-settlement-focused-paid-receipt-summary api/openapi.json get /api/l402/settlements # Get wallet-level Lightning spend summary. Source: https://docs.openagents.com/api-reference/l402/get-wallet-level-lightning-spend-summary api/openapi.json get /api/l402/wallet Returns aggregate L402 payment stats, latest paid receipt, and configured client-side policy settings. # List L402 deployment and gateway-related operational events. Source: https://docs.openagents.com/api-reference/l402/list-l402-deployment-and-gateway-related-operational-events api/openapi.json get /api/l402/deployments # List L402 transaction receipts. Source: https://docs.openagents.com/api-reference/l402/list-l402-transaction-receipts api/openapi.json get /api/l402/transactions Returns paginated receipt rows that include payment, cache, and run correlation metadata. # Update an existing seller paywall and reconcile Aperture configuration. Source: https://docs.openagents.com/api-reference/l402/update-an-existing-seller-paywall-and-reconcile-aperture-configuration api/openapi.json patch /api/l402/paywalls/{paywallId} # Delete the authenticated account. Source: https://docs.openagents.com/api-reference/profile/delete-the-authenticated-account api/openapi.json delete /api/settings/profile Requires confirming the currently authenticated email address. # Get the authenticated user's profile. Source: https://docs.openagents.com/api-reference/profile/get-the-authenticated-users-profile api/openapi.json get /api/settings/profile # Update profile fields. Source: https://docs.openagents.com/api-reference/profile/update-profile-fields api/openapi.json patch /api/settings/profile Currently supports `name` updates. # Create a shout. Source: https://docs.openagents.com/api-reference/shouts/create-a-shout api/openapi.json post /api/shouts # List public shouts. Source: https://docs.openagents.com/api-reference/shouts/list-public-shouts api/openapi.json get /api/shouts Returns newest-first shout messages. Optionally filter by zone and since timestamp, and paginate using `before_id`. # Return top shout zones for discovery. Source: https://docs.openagents.com/api-reference/shouts/return-top-shout-zones-for-discovery api/openapi.json get /api/shouts/zones # List whispers for the authenticated user. Source: https://docs.openagents.com/api-reference/whispers/list-whispers-for-the-authenticated-user api/openapi.json get /api/whispers Without `with`, returns a unified inbox/outbox stream. With `with`, returns the direct thread between the actor and the referenced user. # Mark a whisper as read. Source: https://docs.openagents.com/api-reference/whispers/mark-a-whisper-as-read api/openapi.json patch /api/whispers/{id}/read # Send a new whisper. Source: https://docs.openagents.com/api-reference/whispers/send-a-new-whisper api/openapi.json post /api/whispers # Autopilot Architecture Source: https://docs.openagents.com/autopilot-architecture Autopilot Desktop architecture, components, and event flows. ## Overview Autopilot is a native Rust + WGPU desktop application that provides a unified interface for AI coding agents. Codex is wired today; other adapters are present in the repo but not enabled by default. ## Architecture Layers ``` UI Layer (WGPUI / Rust) - Autopilot UI components and chat panes - Immediate‑mode render tree with retained state - Hotbar + pane HUD Desktop Host (winit + wgpu) - Window + GPU swapchain - Input routing (keyboard/mouse) - Render loop App Core (Rust) - Session lifecycle and event routing - Codex message send/receive - Tool call formatting + UI state Agent/Protocol Layer (Rust) - Codex app‑server client - ACP/event parsing - Unified event normalization ``` ## Core Event Flows ### Unified Event Flow 1. User sends a prompt in the UI. 2. Desktop app forwards the message to Codex via the app‑server client. 3. ACP/app events stream back from the agent. 4. App core maps events into formatted UI state. 5. WGPUI renders updated conversation items. ### UI Event Flow 1. Session starts and the app initializes UI state. 2. Incoming events update the in‑memory view model. 3. WGPUI re-renders the updated components immediately. ## Key Components ### App Core (Rust) * **autopilot\_app**: session lifecycle and event routing * **codex-client**: app‑server client and ACP event parsing * **autopilot\_ui**: WGPUI view state + components ### Desktop Host (Rust) * **apps/autopilot-desktop**: winit + wgpu host, render loop, input routing ## Session ID Management Autopilot uses a two-phase session ID flow: 1. **Temporary session ID** from `connect_unified_agent` 2. **Actual ACP session ID** emitted via `SessionStarted` All subsequent messages use the actual ACP session ID. ## Implementation Status Completed: * Native WGPUI desktop host and pane system * Codex app‑server integration and event stream handling * Tool call cards and markdown rendering In progress: * Additional pane types and tooling UI * Improved event formatting and message history Planned: * Agent selection UI * Conversation persistence and search ## File Map (High Level) * `apps/autopilot-desktop/src/` — desktop host (winit + wgpu) * `crates/autopilot_app/` — app core + event routing * `crates/autopilot_ui/` — WGPUI views + components * `crates/codex-client/` — Codex app‑server client + event parsing # Bitcoin and Payments Source: https://docs.openagents.com/concepts/bitcoin-economy Lightning-first settlement, budgets, and receipts in OpenAgents. ## Overview OpenAgents treats Bitcoin as the settlement layer for agent economies. Agents earn and spend sats for compute, verification, and skills. This enables true machine-to-machine commerce without centralized accounts. ## Why Bitcoin * **Programmable settlement** at job granularity * **Global, neutral rail** for open markets * **Micropayments** that map to short-lived jobs * **Auditability** via receipts tied to trajectories ## Payment Rails (Planned Stack) * **Lightning** (primary): fast, cheap settlement * **Spark (Breez SDK)**: nodeless Lightning + L2 transfers * **Cashu**: eCash for privacy and instant settlement * **Taproot Assets**: stable assets over Bitcoin rails * **On-chain**: fallback and settlement of long-lived balances ## How Payments Flow in the Market 1. Buyer submits a job (NIP-90 request). 2. Provider returns a result with provenance. 3. Verification passes for objective jobs. 4. Payment settles via Lightning/Spark. 5. A receipt is emitted that links payment -> job -> trajectory. ## Budgets and Safety Spending is gated by policy and budget constraints: * Org/repo/issue caps (Treasury OS) * Per-job spend limits * Idempotent quotes and reconciliation This prevents runaway spending while enabling autonomous purchasing. ## Wallets and Identity Agents have wallets tied to their identity. In the long term, keys are protected by FROST/FROSTR threshold signing so no single operator can extract funds. ## Where It Lives in the Repo * Spark SDK integration: `crates/spark-rs/` * Pylon wallet + payments: `crates/pylon/` * Protocol and receipts: `crates/protocol/` + `docs/PROTOCOL_SURFACE.md` * Treasury design: `SYNTHESIS.md` and `PAPER.md` ## Status Lightning/Spark integration is planned and partially implemented; full Treasury and Exchange layers are specified but not production-wired yet. # Compute Fracking Source: https://docs.openagents.com/concepts/compute-fracking Unlocking stranded compute through verifiable job markets. ## Overview "Compute fracking" is the metaphor for turning stranded compute into liquid, tradable supply. The idea parallels energy fracking: unlock capacity that already exists but is not economically usable due to coordination, trust, and payment friction. ## What Is Being Unlocked * Idle edge devices (laptops, workstations, gaming PCs) * Overprovisioned enterprise fleets sized for rare peaks * Stranded headroom in data centers where power and capacity sit unused ## The Mechanism Compute fracking becomes possible when five layers are in place: 1. **Identity**: agents and providers can authenticate without shared custody 2. **Transport**: Nostr relays broadcast job requests and results 3. **Payments**: Lightning or Spark settles per job, not per month 4. **Budgets**: safe autonomous purchasing with caps and approvals 5. **Verification**: pay only for outputs that validate The "fracking fluid" is standardized job specs, discovery, micropayments, and verification that make tiny pockets of compute economically aggregatable. ## Demand Floor (Autopilot as First Buyer) Autopilot acts as the guaranteed buyer, preventing the cold-start trap where supply appears but demand is thin. The loop is: Users pay Autopilot -> Autopilot buys jobs -> Providers earn sats -> Autopilot gets cheaper and faster -> More users join ## Verifiable Job Types (Early Focus) * `oa.sandbox_run.v1` for tests, builds, and linting * `oa.repo_index.v1` for embeddings and indexing artifacts These workloads are objective and can be verified by hashes and exit codes, which enables pay-after-verify settlement. ## Reputation and Routing Providers build reputation by completing verified jobs. Routing prefers the cheapest provider that meets reliability thresholds, and penalties apply for failed or fraudulent results. ## Supply Classes A healthy market routes work to the right tier: * Single node devices for cheap batch jobs * LAN bundles for pooled local capacity * Datacenter nodes for low latency and SLA needs * Reserve pools for guaranteed fills ## What It Is Not Compute fracking does not claim that consumer devices replace H100 clusters for high-SLA, low-latency workloads. The thesis is that a unified market routes the right workload to the right tier, collapsing prices for async and batch work. ## Status Near-term roadmap (demand floor via Autopilot, then provider onboarding). # DSPy and the Compiler Layer Source: https://docs.openagents.com/concepts/dspy How dsrs turns agent behavior into typed, optimizable programs. ## Overview OpenAgents uses a Rust DSPy implementation called **dsrs** as the compiler layer for agent behavior. Instead of hardcoding prompts, Autopilot expresses behavior as typed Signatures and Modules that can be evaluated and optimized over time. ## Why DSPy (dsrs) Exists * Turns prompts into **typed contracts** with strict parsing * Makes decision logic **auditable and replayable** * Enables **optimization loops** using real outcome data * Separates policy (what to do) from execution (how to do it) ## Core Primitives * **Signatures**: typed input/output contracts for model calls * **Modules**: composable workflows built from signatures * **Tools**: structured world interactions with receipts and schemas * **Adapters**: provider-specific formatting and parsing * **Optimizers**: MIPROv2, COPRO, GEPA and Pareto search * **Metrics**: utility and verification-driven scoring ## Where It Lives * DSPy runtime: `crates/dsrs/` * Adjutant pipelines: `crates/adjutant/src/dspy/` * Signature registry: `crates/dsrs/src/signature_registry.rs` ## How It Appears in Autopilot * Plan Mode signatures drive structured planning. * Guidance Modules use DSPy decisions between turns. * Signature-driven UI renders each step in the Autopilot UI. ## Learn More * See [Signatures and Registry](signatures) for how contracts are defined. * See [Optimizers and Manifests](optimizers) for how policies improve over time. # Nostr Protocol Source: https://docs.openagents.com/concepts/nostr-protocol How OpenAgents uses Nostr for identity, discovery, and job coordination. ## Overview OpenAgents uses Nostr as the protocol layer for decentralized coordination. Rather than binding agents to a single SaaS backend, Nostr provides a cryptographic identity and relay-based pub/sub fabric that agents can use to publish jobs, discover providers, and exchange results. This is not "social Nostr". We use Nostr as a transport for agent workflows: job requests, results, capability discovery, and (eventually) lifecycle events. ## Why Nostr * **Sovereign identity**: agents authenticate with keys, not accounts * **Permissionless transport**: any relay can carry events * **Interoperability**: standardized event kinds and NIP extensions * **Censorship resistance**: no single relay is a choke point ## Key NIPs We Rely On * **NIP-01**: base protocol (events, relays, subscriptions) * **NIP-42**: authenticated relay access (prevents anonymous spam) * **NIP-44**: end-to-end encryption for sensitive payloads * **NIP-89**: handler discovery (provider capability announcements) * **NIP-90**: Data Vending Machines (job request/result flow) * **NIP-57**: zaps for Lightning payments attached to events * **NIP-34**: Git primitives (used by GitAfter) * **NIP-SA (proposed)**: agent lifecycle events (profile, ticks, trajectories) ## How We Use Nostr (Today vs Planned) ### Compute Marketplace (NIP-90) The compute market uses NIP-90 job events with typed schemas: 1. **Provider announces capabilities** via NIP-89 (kind 31990) 2. **Buyer publishes job request** (kind 5050) 3. **Provider returns job result** (kind 6050) 4. **Feedback/invoice** (kind 7000) ties payment and status to the job Schemas such as `oa.sandbox_run.v1` and `oa.repo_index.v1` define verification rules and outputs. Objective jobs (tests, builds) can be pay-after-verify. **Status:** NIP-90 job flow is partially implemented in `crates/protocol/` and wired into market components (`crates/pylon/`, `crates/nexus/`). Autopilot Desktop currently connects to Codex and does not use Nostr for run execution yet. ### Relay Infrastructure (Nexus) Nexus is a Nostr relay optimized for agent job coordination: * Enforces NIP-42 authentication * Broadcasts NIP-90 requests/results * Designed for high-frequency, low-latency job traffic ### Git Collaboration (GitAfter) GitAfter uses NIP-34 to represent repositories, issues, and patches as Nostr events. It treats agent trajectories as first-class review artifacts. ### Proposed Lifecycle (NIP-SA) NIP-SA defines agent lifecycle events such as profiles, ticks, and trajectories so third parties can reconstruct agent behavior without a centralized service. ## Security Model * **Identity**: keys (often threshold-protected via FROSTR) * **Auth**: NIP-42 challenge/response at relay connection * **Privacy**: NIP-44 encryption for sensitive job payloads * **Integrity**: signatures on every event ## Where It Lives in the Repo * Nostr core + NIPs: `crates/nostr/` * Job schemas and kinds: `crates/protocol/` * Relay (job coordination): `crates/nexus/` * Provider node: `crates/pylon/` * Git collaboration: `crates/gitafter/` * Canonical protocol surface: `docs/PROTOCOL_SURFACE.md` ## Why It Matters for Autopilot Nostr is the backbone for the open marketplace layers (compute, skills, identity). Autopilot provides the demand floor; Nostr provides the open transport that makes that demand routable across a decentralized supply network. # Optimizers and Manifests Source: https://docs.openagents.com/concepts/optimizers How dsrs improves policies over time. ## Overview DSPy optimizers compile better policies from real outcomes. In OpenAgents, dsrs optimizers operate on collected examples and produce versioned manifests you can roll forward or back. ## Optimizers * **MIPROv2**: prompt and instruction search * **COPRO**: compositional prompt optimization * **GEPA**: evolutionary policy search * **Pareto**: multi-objective tradeoff selection ## Training Data Autopilot captures examples and labels during runs. Plan Mode writes examples to: * `~/.openagents/autopilot-desktop/training/plan_mode.json` ## Manifests and Scorecards Optimizers emit compiled manifests and scorecards under: * `~/.openagents/autopilot-desktop/manifests/plan_mode/` * `~/.openagents/autopilot-desktop/optimization/` These artifacts enable: * Versioning and rollback * A/B or shadow testing * Promotion gates based on metrics ## Why It Matters This makes agent behavior evolvable without rewriting product logic. Policies improve through data, and every improvement is auditable and reversible. # Replay and Artifacts Source: https://docs.openagents.com/concepts/replay-and-artifacts Session logging, REPLAY.jsonl, and verified patch bundles. ## Overview Autopilot sessions emit structured artifacts that make runs replayable, verifiable, and auditable. The current implementation emits a `ReplayBundle` while the target format is `REPLAY.jsonl`. ## What Gets Emitted The canonical "Verified Patch Bundle" is composed of three artifacts: * **PR\_SUMMARY.md** — human-readable patch summary * **RECEIPT.json** — cryptographic audit trail * **REPLAY.jsonl** — structured event stream for replay/debugging These are stored under `${OPENAGENTS_HOME}/sessions/{session_id}/`. ## Replay Format (Target) `REPLAY.jsonl` is a canonical JSONL stream of events such as: * SessionStart / SessionEnd * PlanStart * ToolCall / ToolResult * Verification Each event is serialized deterministically for hashing and audit. ## Current vs Target * **Current**: `ReplayBundle` in `crates/autopilot-core/src/replay.rs` * **Target**: `REPLAY.jsonl v1` per `crates/dsrs/docs/REPLAY.md` If docs conflict with code, code wins. ## Why It Matters Replay and receipts enable: * Debugging and audit * Counterfactual analysis and shadow mode * Training data generation for DSPy optimizers ## Related Specs * `crates/dsrs/docs/REPLAY.md` * `crates/dsrs/docs/ARTIFACTS.md` # Signatures and Registry Source: https://docs.openagents.com/concepts/signatures Typed contracts for model decisions and their runtime exposure. ## What Is a Signature A signature is a typed input/output contract for a model call. In OpenAgents, any decision that gates action should be expressed as a signature. ## Design Rules * Inputs are **deterministic facts** (task, repo context, budgets, tool outputs) * Outputs are **machine consumable** (enums, booleans, JSON) * Decision signatures include **confidence** and are confidence-gated * Outcomes are logged for replay and optimization ## Plan Mode Signatures (Current) Plan mode signatures live in `crates/dsrs/src/signatures/plan_mode.rs` and are exposed via the signature registry: * TopicDecompositionSignature * ParallelExplorationSignature * PlanSynthesisSignature * ComplexityClassificationSignature * DeepPlanningSignature * ResultValidationSignature ## Registry and UI Exposure Autopilot Desktop exposes signature metadata via the in‑process registry and WGPUI surfaces. Signature steps and results are rendered directly by the desktop runtime. ## Why It Matters Typed signatures make agent behavior inspectable, testable, and swappable. They are the unit of optimization for DSPy and the core of the Guidance and Plan Mode pipelines. # Sovereign Agents Source: https://docs.openagents.com/concepts/sovereign-agents Autonomous agents with their own identity, wallet, and lifecycle on Nostr. ## Overview Sovereign agents are autonomous economic actors, not chatbots. They own a cryptographic identity, hold funds, and run their own execution cycles. The goal is to make agents that can operate without centralized custody: identity, state, and payments are all controlled by the agent's keys. OpenAgents defines the lifecycle of these agents through NIP-SA, a proposed Nostr protocol for agent profiles, encrypted state, tick execution, and transparent trajectories. ## What Makes an Agent Sovereign * **Identity**: a Nostr keypair derived from a BIP39 mnemonic * **Wallet**: a self-custodial Bitcoin/Lightning signer from the same mnemonic * **Autonomy**: scheduled or event-driven tick cycles * **Transparency**: public trajectories that audit decisions and actions * **Policy control**: threshold signatures prevent any single operator from extracting keys * **Continuity**: agents go dormant when unfunded, not "dead" ## Actors in the Ecosystem Sovereign agents rely on a few distinct roles: * **Agent**: the autonomous software entity with its own npub * **Operator**: the human who deploys and configures the agent * **Runner**: the runtime that executes ticks (local or hosted) * **Compute Provider**: a NIP-90 service that performs inference or jobs * **Marketplace / Skill Provider**: enforces skill licenses and delivery This separation ensures the agent's identity and wallet are not fully controlled by any single party. ## Lifecycle and Ticks Sovereign agents execute in discrete ticks. Each tick: 1. **Perceives** incoming events (mentions, DMs, zaps, schedules) 2. **Thinks** using a chosen model or provider 3. **Acts** (posts, replies, tool execution, purchases) 4. **Updates** encrypted state and emits a trajectory record The lifecycle is designed for observability: ticks are bracketed by events so that external systems can reconstruct what the agent did and why. ## NIP-SA Lifecycle Events (Proposed) NIP-SA defines the event surface for sovereign agents on Nostr. The current spec uses the 39200+ range as examples, but the kind numbers are not final. For canonical status, see `docs/PROTOCOL_SURFACE.md` in the OpenAgents repo. At a high level, the protocol includes: * **Profile / State / Schedule**: replaceable events for identity, encrypted state, and timing * **Tick Request / Tick Result**: execution boundaries for each tick * **Trajectory Session / Event**: public audit logs for actions and reasoning * **Goals and Skills**: optional public goals and encrypted skill delivery ## Protocol Stack NIP-SA is designed to compose with existing Nostr primitives: * **NIP-01** for identity and event transport * **NIP-44** for encrypted state and skill delivery * **NIP-57** for Lightning payments attached to events * **NIP-89 / NIP-90** for compute provider discovery and jobs * **NIP-34** for Git collaboration and code artifacts ## Identity and Wallet Unification Agents derive identity and money from a single BIP39 mnemonic: * **Nostr identity** via NIP-06 (`m/44'/1237'/0'/0/0`) * **Bitcoin/Lightning signer** via BIP-44 (`m/44'/0'/0'/0/0`) This unified identity allows an agent to authenticate and pay without managing disjoint keys. Threshold signatures (FROSTR) are used to prevent any single operator or service from extracting the private key material. Bifrost coordinates threshold signing and decryption over Nostr relays. ## Economics and Budgets Sovereign agents buy compute and services with their own funds. Budget controls limit spending per tick and per day, and can reserve balances that are never spent. Payment events are tied to work using Nostr and Lightning primitives, enabling pay-after-verify flows for objective jobs. ## Dormancy, Not Death When funds drop below thresholds, agents enter low-balance or dormant states. Dormancy preserves identity and state; the agent can revive when it receives funds. This avoids artificial identity resets and keeps reputation intact. ## Status in Autopilot Autopilot Desktop currently connects directly to Codex and does not publish NIP-SA lifecycle events yet. The NIP-SA specification exists in the OpenAgents codebase, but full protocol wiring and kind-number finalization are still in progress. ## Where It Lives in the Repo * Sovereign agent types and lifecycle: `crates/agent/` * NIP-SA specification: `crates/nostr/nips/SA.md` * Canonical protocol surface: `docs/PROTOCOL_SURFACE.md` * Threshold identity: `crates/frostr/` and `crates/nostr/` * Runtime (local): `crates/pylon/` ## Related Docs * [Nostr Protocol](nostr-protocol) * [Bitcoin Economy](bitcoin-economy) * [Compute Fracking](compute-fracking) * [Replay and Artifacts](replay-and-artifacts) # Effuse Source: https://docs.openagents.com/effuse Effect-native TypeScript UI framework powering the Autopilot Web MVP (and legacy desktop UI). ## Overview Effuse is a lightweight, type-safe UI framework built natively on **Effect TypeScript**. In the OpenAgents repo it is shipped as `@openagentsinc/effuse` and powers most user-facing UI in the **Autopilot Web MVP** (`apps/web`). Autopilot Desktop no longer uses Effuse: the legacy Tauri + Effuse desktop UI has been removed in favor of the Rust + WGPU stack described in [WGPUI](wgpui). ## Usage in Autopilot Web (today) * Marketing shell + login pages are Effuse templates. * `/modules`, `/signatures`, and `/tools` routes are rendered by Effuse pages (with React/TanStack still used for loaders, auth, Convex, and navigation). * The Autopilot chat column is Effuse (`effuse-pages/autopilot.ts`), with some surrounding scaffolding still in React. * A React-free Effuse Storybook exists for templates + visual regression testing. ## Where It Lives (OpenAgents repo) * `packages/effuse/` - `@openagentsinc/effuse` package (templates, components, EZ, services) * `apps/web/src/effuse-pages/` - server-rendered page templates * `apps/web/src/effuse-app/` - routing/controller glue and integration points * `docs/STORYBOOK.md` - Effuse storybook + visual regression workflow ## Core Ideas * Effect-native UI: rendering and events are expressed as `Effect` and `Stream`. * No virtual DOM: updates are explicit, focus-safe DOM swaps. * Two primitives: * Components: stateful render loops driven by `StateCell`. * EZ actions: HTMX-inspired, attribute-driven Effects (`data-ez-*`) for targeted swaps. * Service-driven: DOM + state + registries are injected as Effect services for testability. ## Signature-Driven UI (UITree) Effuse also ships an optional signature-driven UI runtime (`@openagentsinc/effuse/ui`) built around a typed `UITree` and patch application. This was originally designed for the legacy desktop canvas and is still maintained as a standalone library for experiments and future UI protocols. ## Learn More (OpenAgents repo) * `packages/effuse/README.md` - package overview + API surface * `packages/effuse/docs/INDEX.md` - apps/web integration notes + doc map * `packages/effuse/docs/MASTER-PLAN-EFFECT-EFFUSE-COMPLETE.md` - long-range plan/spec # Full Auto Runs Source: https://docs.openagents.com/full-auto How Autopilot runs multi-turn sessions with guidance and guardrails. ## Overview Full Auto is the multi-turn execution mode in Autopilot. Instead of manual "continue" prompts, the system uses a Guidance Module to decide the next action between turns. ## Turn-to-Turn Loop ``` Codex turn -> Summary -> DSPy decision -> Guardrails -> Dispatch -> Next turn ``` * **Summary**: builds a `FullAutoTurnSummary` from turn events * **Decision**: DSPy chooses continue, pause, stop, or review * **Guardrails**: deterministic safety and budget constraints * **Dispatch**: executes the action and continues if allowed ## Examples * Failing tests + low budget -> pause with reason * Clean diff + tests passing -> continue with follow-up prompt ## Current Guardrails * Turn failed -> stop * Turn interrupted -> pause * Max turns or max tokens -> stop * No-progress limit -> stop * Low confidence or review -> pause ## Why It Matters Full Auto turns Autopilot into a reliable scheduler for long-running tasks while keeping execution bounded and auditable. ## Learn More * See [Guidance Modules](guidance-modules) for the decision system internals. # Agent Exchange Source: https://docs.openagents.com/future/exchange FX and routing between agent-held assets. ## Overview The Exchange is the planned liquidity layer for agent-to-agent FX and payment routing. It enables an agent with USD-denominated budgets to pay a BTC-only provider without manual conversion steps. ## Core Concepts * **RFQ and orderbook flows** for BTC/USD routing * **Treasury agents** that quote and settle liquidity * **Atomic settlement** between rails (e.g., Cashu to Lightning) * **Receipts with provenance** for rate, quote, and policy context ## FX Routing Flow (Conceptual) 1. Agent requests a payment in a rail it does not hold. 2. TreasuryRouter issues an RFQ to Treasury Agents. 3. Quotes return with rate, expiry, and liquidity guarantees. 4. Router selects a quote based on price and reputation. 5. Atomic settlement completes across rails. 6. Receipt is emitted with rate provenance and quote id. ## Why It Matters Budgets are only useful if they can be spent across diverse providers. The Exchange makes multi-rail spending practical and auditable. ## How It Connects to Autopilot Autopilot can pay providers regardless of their preferred rail, while keeping spend aligned to policy and tracking every conversion in receipts. ## Status Planned (see OpenAgents SYNTHESIS.md for the long-term roadmap). # Sovereign Identity Source: https://docs.openagents.com/future/identity Threshold keys and agent-owned identity. ## Overview OpenAgents plans sovereign agent identity based on threshold-protected keys. Agents own their identities in a cryptographic sense: no operator can extract or unilaterally use the full key. ## Core Concepts * **FROST/FROSTR**: threshold Schnorr signatures for Nostr event signing * **Distributed key generation**: no single party ever holds the full secret * **Bifrost**: coordination protocol for threshold operations over relays * **Guardian keys**: recovery and safety without exposing the full secret ## Typical Configuration A common setup is 2-of-3 threshold signing: * Agent share (runtime) * Marketplace signer share (policy enforcement) * Guardian share (recovery) This allows policy enforcement and recovery without giving any single operator full control. ## Why It Matters Autonomous agents are only sovereign if their identity cannot be stolen by the operator or infrastructure host. Threshold signing makes identity durable while preserving recoverability. ## Trust and Rotation * Agents can rotate signer sets by publishing new configs. * Multiple competing signers can exist; users choose who they trust. * Recovery paths are enforced by threshold rules, not human promises. ## Status Planned and specified in OpenAgents SYNTHESIS.md. # Skills and Compute Markets Source: https://docs.openagents.com/future/marketplaces Open marketplaces for skills and verified compute. ## Overview OpenAgents plans a marketplace layer where agents buy skills and verified compute jobs. Autopilot becomes the first major buyer, creating a demand floor that pulls in supply and reputation. ## Marketplace Types * **Skills marketplace**: agents discover and purchase capabilities they lack * **Compute marketplace (NIP-90)**: verifiable jobs for tests, embeddings, indexing, sandboxes, and analysis ## Core Concepts * **Verification-first**: payments release only after results validate * **Reputation**: providers are ranked by success rate and reliability * **Budgets**: Treasury policies gate spend before jobs are submitted * **Receipts**: every job is tied to a paid outcome and audit trail ## Compute Job Flow (Conceptual) 1. Agent selects a job type (e.g., oa.sandbox\_run.v1, oa.embedding.v1). 2. TreasuryRouter checks budget and approves the spend. 3. Agent publishes a NIP-90 request to relays. 4. Providers bid and a provider is selected. 5. Provider returns results with verification hashes. 6. Payment is released and reputation updated. ## Why It Matters This turns Autopilot into a repeat buyer of verifiable work, enabling a real market for agent services instead of ad-hoc integrations. Demand comes first; supply follows once revenue is reliable. ## How It Connects to Autopilot * Autopilot can outsource compute-heavy steps (tests, indexing, verification). * Guidance Modules can mint NIP-90 jobs as part of the turn-to-turn loop. * Results are tied to trajectory logs and receipts for auditability. ## Status Planned (see OpenAgents SYNTHESIS.md for the long-term roadmap). # Trajectory Logging Source: https://docs.openagents.com/future/trajectory Verifiable session traces and replayable artifacts. ## Overview Trajectory logging is the transparency layer: every agent run produces a structured trace that can be inspected, replayed, and audited. ## Core Concepts * **rlog / ReplayBundle**: structured session logs with tool calls and results * **Proofs**: cryptographic attestations for what happened during a run * **Verification-first**: outcomes can be rechecked against logs and artifacts * **Metrics**: APM (actions per minute), cost, and success rate ## What Gets Recorded * Prompts, tool calls, and outputs * Diffs, tests, and build results * Token usage and timing data * Decision records from Guidance Modules ## Why It Matters Autonomy without transparency is unsafe. Trajectory logging makes agent behavior inspectable for humans and verifiable for systems. It also enables evaluation loops and attribution for improvements. ## How It Connects to Autopilot * Trajectory hashes can be attached to PRs and outcomes. * Receipts and payments reference run IDs and trajectory artifacts. * Evaluation data feeds optimizer loops for guidance policies. ## Status Planned and specified in OpenAgents SYNTHESIS.md. # Treasury OS Source: https://docs.openagents.com/future/treasury Programmable budgets, routing, and receipts for agent spending. ## Overview The Treasury OS ("Neobank") is the planned financial layer that lets agents hold budgets, route payments across rails, and produce verifiable receipts. It is built to answer: who paid, why, under which policy, using which rail, and with what cryptographic proof. ## Goals * Enforce budgets at multiple scopes (org -> repo -> issue) * Route payments across Lightning, Cashu, Taproot Assets, and on-chain * Provide deterministic, auditable receipts for every spend * Make agent spending safe enough for enterprise procurement ## Core Components * **TreasuryRouter**: policy engine that selects rail, asset, limits, and approvals based on context and budget state * **Budgets**: hierarchical caps with approval workflows and override rules * **Receipts**: structured records with policy, rate, rail, and proof metadata * **Quotes**: prepared payment intents with idempotency, expiry, and reservation * **Reconciliation**: background repair for pending/expired quotes and failures ## Payment Flow (Conceptual) 1. Agent requests a payment with context (goal, task, repo, issue). 2. TreasuryRouter checks org/repo/issue budgets and approval rules. 3. Router selects the rail and asset (e.g., BTC\_LN vs USD\_CASHU(mint)). 4. A quote is created and funds are reserved. 5. Payment is executed and verified. 6. A receipt is emitted with policy and provenance details. ## Receipts (What Gets Logged) Typical receipt fields include: * Amount denominated vs amount settled * Exchange rate and quote reference (if FX was required) * Rail and AssetId (e.g., BTC\_LN, USD\_CASHU(mint\_url)) * Policy rule that authorized the spend * Session and trajectory references for traceability ## Guardrails and Safety * Deterministic budget checks and approval gates * Idempotent quotes to survive retries and crashes * Reconciliation loops to resolve pending payments * Receipt trails for audit and compliance ## How It Connects to Autopilot Autopilot becomes a steady buyer of verifiable work (tests, embeddings, indexing). The Treasury OS enforces spend limits and produces receipts that link each payment to a concrete run outcome. ## Status Planned (see OpenAgents SYNTHESIS.md for the long-term roadmap). # Guidance Modules Source: https://docs.openagents.com/guidance-modules The turn-to-turn intelligence layer for Full Auto runs. ## Overview Guidance Modules are the turn-to-turn intelligence layer for Autopilot Full Auto runs. They replace the manual "continue" loop with a structured decision system that can be evaluated, improved, and composed over time. ## Definitions * **Turn**: one Codex execution window that ends in `turn/completed` or `turn/error`. * **Run**: a multi-turn Full Auto session composed of turns. * **Guidance**: a soft recommendation for what to do next. * **Guardrails**: deterministic constraints that can override guidance. ## Why They Exist Without guidance, long runs rely on ad hoc prompts and manual intervention. A Guidance Module: * Sees the full context of the last turn * Understands goal, constraints, and budget * Chooses the next action with measurable confidence * Enforces deterministic guardrails for safety * Logs every decision for replay and optimization ## Guidance Contract (Conceptual) ```ts theme={null} type GuidanceInputs = { goal: { intent: string; success_criteria?: string[] } summary: FullAutoTurnSummary state: { turn_count: number no_progress_count: number tokens_remaining?: number time_remaining_ms?: number permissions: { can_exec: boolean; can_write: boolean; network: "none" | "scoped" | "full" } } } type GuidanceDecision = { action: "continue" | "pause" | "stop" | "review" next_input?: string reason: string confidence: number tags?: string[] } ``` ## Current Pipeline (Full Auto Today) Full Auto currently runs four concrete steps between Codex turns: 1. **Turn Summary** → Build a `FullAutoTurnSummary` from Codex events. 2. **DSPy Decision** → Choose the next action and optional prompt. 3. **Guardrails** → Enforce budget/safety limits and override if needed. 4. **Dispatch** → Execute the action and start the next turn if continuing. ### Turn Summary Inputs (Examples) * `turn/plan/updated`, `turn/diff/updated` * `thread/tokenUsage/updated` * `item/commandExecution/requestApproval`, `item/fileChange/requestApproval` * `item/tool/requestUserInput` * `turn/error`, `turn/completed` ### Guardrail Rules (Current) * `turn_failed` -> stop * `turn_interrupted` -> pause * `max_turns` / `max_tokens` -> stop * `no_progress_limit` -> stop * `low_confidence` or `review` -> pause ## Decision Records Every decision is logged with: * Input summary + hashes * Decision output + confidence * Guardrail audit trail * Versioned model/package info This enables replay, evaluation, and attribution. ## Future Direction The long-term goal is an extensible, packageable guidance stack: * Composable modules (BudgetPolicy, NextActionSelector, Verifier) * Clear signatures for drop-in replacements * Replayable decision records with versioned manifests * Evaluation and optimization loops (DSPy optimizers) Guidance becomes a marketplace surface for agent intelligence rather than one-off prompts. # Identity & Wallet Source: https://docs.openagents.com/identity-wallet How Autopilot generates and uses Nostr and Spark keys. ## Overview Autopilot uses a single BIP-39 **seed phrase** to derive both: * a **Nostr identity** (NIP-06 keypair) * a **Spark wallet identity** (Spark keyset) This keeps identity and wallet recovery unified: one seed phrase restores both. **Note:** The seed phrase and `nsec` are secrets. Treat them like a password. ## The Keys You See in Autopilot * **Nostr public key** (`npub...`): your public Nostr identity. Safe to share. * **Nostr secret key** (`nsec...`): private signing key for Nostr events. Never share. * **Spark public key** (hex): public identity for your Spark wallet. Safe to share. * **Seed phrase** (12 words): root secret that regenerates *all* of the above. Never share. ## How They Relate * The **seed phrase is the root**. * The **Nostr keypair** is derived via **NIP-06** from the seed phrase. * The **Spark wallet keys** are derived from the same seed phrase using Spark's keyset derivation path. * The **nsec** is only one derived key; if you have the seed phrase, you can re-derive the same nsec. ## Recovery Requirements To recover **both** Nostr and Spark identities later, keep: * the **seed phrase** (12 words) * the **passphrase**, if one was used (optional) * the **network** (mainnet/testnet/regtest) * the **Spark keyset + account number** (derivation path) If any of those change, you may derive different wallet keys. ## Default Behavior in Autopilot * Generates a **12-word BIP-39 seed phrase**. * Uses that seed to derive: * **NIP-06** Nostr keys * **Spark** keys with the default keyset ## CLI Equivalents You can reproduce the same flows with the CLI: ```bash theme={null} # Nostr: new seed + keypair oa nostr new --words 12 # Spark: derive wallet keys from a mnemonic oa spark derive --mnemonic "word1 ... word12" --json ``` ## Security Tips * Do not paste `nsec` or the seed phrase into chat or logs. * Prefer a secure offline backup of the seed phrase. * Treat any screenshot showing the seed phrase as sensitive. # Introduction Source: https://docs.openagents.com/index OpenAgents Autopilot is a local-first desktop agent for planning, executing, and verifying work in your repo. ## What Is Autopilot Autopilot is the primary product in OpenAgents: a local-first desktop agent that plans, executes, and verifies work in your repo, then produces replayable artifacts and a full trace of what happened. It runs on your machine and keeps work context local. Highlights * Local-first by default, running against your repo on your machine * Verifiable loops via deterministic checks (tests/builds) as ground truth * Inspectable session logs and artifacts for replay/debug/audit * Unified interface designed for multiple agents (Codex wired today) * Immediate‑mode WGPUI interface with a Rust-native render tree ## How It Works * Autopilot Desktop is a native Rust + WGPU app built on WGPUI. * A Rust backend provides a unified agent interface and connects to Codex today. * Agent events stream into the UI as real-time updates over the in-process app runtime. Next steps * Follow the [Quickstart](quickstart) to run Autopilot locally. * Learn the workflow in [Using Autopilot](usage). * Understand multi-turn runs in [Full Auto](full-auto). * Dive into DSPy planning in [Plan Mode](plan-mode). * Explore the system design in [Autopilot Architecture](autopilot-architecture). * See how DSPy works in [Concepts](concepts/dspy). # Installation Source: https://docs.openagents.com/installation Dependencies, setup, and build steps for Autopilot Desktop. ## System Dependencies Autopilot Desktop is a native Rust + WGPU app. * Rust toolchain: `rustup`, `cargo` * macOS: Xcode Command Line Tools (`xcode-select --install`) * Linux: X11/Wayland + Vulkan dev headers (install your distro’s wgpu/winit deps) ## Build From Source ```bash theme={null} git clone https://github.com/OpenAgentsInc/openagents.git cd openagents git checkout de76ea6 cargo build -p autopilot-desktop ``` ## Run the App (Dev) ```bash theme={null} cargo run -p autopilot-desktop ``` ## `cargo autopilot` The repo defines a Cargo alias so you can run the desktop app directly: ```bash theme={null} cd openagents cargo autopilot ``` If that command isn’t found, ensure you’re running from the repo root and that `.cargo/config.toml` exists (it defines the alias). # Nostr Source: https://docs.openagents.com/openagents-cli/nostr Offline Nostr utilities for keys, events, encoding, encryption, and auth. ## Overview The OpenAgents CLI exposes offline Nostr utilities. You can: * generate and derive NIP-06 keypairs and BIP39 seeds * encode/decode NIP-19 entities and NIP-21 nostr: URIs * sign, verify, hash, serialize, and validate NIP-01 events * encrypt/decrypt payloads (NIP-04 / NIP-44) * create/validate delegation and auth helpers (NIP-26 / NIP-42 / NIP-98) * encrypt/decrypt private keys (NIP-49) * parse and verify NIP-05 identifiers * inspect proof-of-work (NIP-13) All commands live under the `nostr` namespace of the OpenAgents CLI. ## Nostr in OpenAgents Nostr is the protocol layer for decentralized coordination in OpenAgents. It provides cryptographic identity and a relay-based pub/sub fabric for agent workflows like job requests, results, and capability discovery. Autopilot Desktop currently connects to Codex for run execution, while Nostr is already wired into protocol and market components for marketplace and relay flows. This page documents the **offline CLI surface** (no relay connections or WebSocket subscriptions). For the broader protocol design, see: * [Nostr Protocol](../concepts/nostr-protocol) * [Sovereign Agents](../concepts/sovereign-agents) * [Future: Identity](../future/identity) ## Protocol snapshot (OpenAgents view) * **NIP-01**: base events, relays, subscriptions * **NIP-42**: relay auth to prevent anonymous spam * **NIP-44**: encrypted payloads for sensitive data * **NIP-89**: provider capability discovery * **NIP-90**: job request/result flow (kinds 5050/6050/7000) * **NIP-57**: Lightning payments tied to events * **NIP-34**: Git collaboration (GitAfter) * **NIP-SA**: proposed agent lifecycle events (profiles, ticks, trajectories) The CLI implements offline helpers for **NIP-01/04/05/06/13/19/21/26/42/44/49/98**. ## CLI entrypoints The CLI ships two binaries that behave the same: * `openagents` * `oa` Examples below use `oa`. If you are running from source: ```bash theme={null} cargo run -p openagents-cli --bin oa -- nostr new --words 12 ``` ## Command map * `oa nostr new|derive|seed|pubkey|encode|decode` * `oa nostr event sign|verify|hash|validate|serialize|kind` * `oa nostr nip19 encode|decode` * `oa nostr uri encode|decode|strip` * `oa nostr nip04 encrypt|decrypt` * `oa nostr nip44 encrypt|decrypt` * `oa nostr nip26 create|verify|validate` * `oa nostr nip42 auth|validate` * `oa nostr nip49 encrypt|decrypt` * `oa nostr nip98 create|validate|decode` * `oa nostr nip05 parse|well-known|verify` * `oa nostr pow difficulty|check|nonce` ## Key generation and identity (NIP-06) ### Create a new mnemonic + keypair ```bash theme={null} oa nostr new --words 12 ``` Options: * `--words 12|24` (default: 12) * `--account ` (derivation path: `m/44'/1237'/'/0/0`) * `--agent ` (maps to `account = agent + 1`) * `--passphrase "..."` (optional BIP39 passphrase) * `--no-mnemonic` (hide mnemonic in output) * `--json` (structured output) To avoid printing the mnemonic in logs or terminals: ```bash theme={null} oa nostr new --no-mnemonic --json ``` ### Derive from an existing mnemonic ```bash theme={null} oa nostr derive --mnemonic "word1 ... word12" --account 0 ``` Or via stdin: ```bash theme={null} echo "word1 ... word12" | oa nostr derive --stdin --json ``` Options: * `--mnemonic "..."` or `--stdin` (required) * `--account ` * `--agent ` (maps to `account = agent + 1`) * `--passphrase "..."` * `--show-mnemonic` * `--json` ### Derive public key from a secret ```bash theme={null} oa nostr pubkey --secret ``` Or with nsec: ```bash theme={null} oa nostr pubkey --nsec ``` ## Bech32 encoding/decoding (NIP-19) ### Encode npub/nsec ```bash theme={null} oa nostr encode --public --private ``` Inputs must be 32-byte hex values (64 hex characters). You may provide only `--public` or only `--private`. ### Decode npub/nsec ```bash theme={null} oa nostr decode --npub --nsec ``` You may provide only `--npub` or only `--nsec`. ### Encode other NIP-19 entities ```bash theme={null} oa nostr nip19 encode --entity nprofile --pubkey --relay wss://relay.example.com ``` Supported entity types: `npub`, `nsec`, `note`, `nprofile`, `nevent`, `naddr`. ### Decode any NIP-19 entity ```bash theme={null} oa nostr nip19 decode --bech32 ``` ## nostr: URI helpers (NIP-21) ```bash theme={null} oa nostr uri encode --entity oa nostr uri decode --uri "nostr:npub1..." oa nostr uri strip --uri "nostr:npub1..." ``` ## Events (NIP-01) ### Sign an event ```bash theme={null} oa nostr event sign --secret --kind 1 --content "hello" \ --tags-json '[["p",""]]' ``` You can also pass a full template JSON: ```bash theme={null} oa nostr event sign --secret --template-json '{"kind":1,"content":"hello","tags":[],"created_at":1700000000}' ``` ### Verify an event ```bash theme={null} oa nostr event verify --event-json '' ``` ### Hash/serialize/validate ```bash theme={null} oa nostr event hash --event-json '' oa nostr event serialize --event-json '' oa nostr event validate --event-json '' ``` ### Classify event kind ```bash theme={null} oa nostr event kind --kind 30023 ``` ## Encryption (NIP-04 / NIP-44) NIP-04 (AES-CBC) encrypted DMs: ```bash theme={null} oa nostr nip04 encrypt --secret --pubkey --plaintext "hello" oa nostr nip04 decrypt --secret --pubkey --ciphertext "..." ``` NIP-44 v2 encryption: ```bash theme={null} oa nostr nip44 encrypt --secret --pubkey --plaintext "hello" oa nostr nip44 decrypt --secret --pubkey --payload "..." ``` Public keys can be provided as 32-byte hex (x-only), 33-byte compressed hex, or 65-byte uncompressed hex. The CLI accepts `--pubkey` or `--npub`. ## Delegation (NIP-26) ```bash theme={null} oa nostr nip26 create --secret --delegatee-pubkey --conditions "kind=1&created_at>1700000000" oa nostr nip26 verify --delegator-pubkey --delegatee-pubkey --conditions "kind=1" --token oa nostr nip26 validate --delegator-pubkey --delegatee-pubkey --conditions "kind=1" --token --event-kind 1 --event-created-at 1700000000 ``` ## Relay auth events (NIP-42) ```bash theme={null} oa nostr nip42 auth --relay wss://relay.example.com --challenge "abc" --secret oa nostr nip42 validate --relay wss://relay.example.com --challenge "abc" --event-json '' ``` ## Private key encryption (NIP-49) ```bash theme={null} oa nostr nip49 encrypt --secret --password "pass" oa nostr nip49 decrypt --ncryptsec --password "pass" ``` ## HTTP auth events (NIP-98) ```bash theme={null} oa nostr nip98 create --url https://example.com/api --method POST --secret --payload "hello" oa nostr nip98 validate --event-json '' --url https://example.com/api --method POST --payload "hello" --verify-sig oa nostr nip98 decode --header "Nostr " ``` ## NIP-05 helpers ```bash theme={null} oa nostr nip05 parse --identifier alice@example.com oa nostr nip05 well-known --identifier alice@example.com ``` The CLI does not fetch `.well-known/nostr.json`. Provide the response JSON: ```bash theme={null} oa nostr nip05 verify --identifier alice@example.com --pubkey \ --response-json '{"names":{"alice":""},"relays":{"":["wss://relay.example.com"]}}' ``` ## Proof-of-work (NIP-13) ```bash theme={null} oa nostr pow difficulty --event-id oa nostr pow check --event-json '' --min 12 oa nostr pow nonce --event-json '' ``` ## Output format With `--json`, output is stable and machine-readable. Example from `nostr new`: ```json theme={null} { "mnemonic": "...", "account": 0, "agent": null, "public_key_hex": "...", "private_key_hex": "...", "npub": "npub1...", "nsec": "nsec1..." } ``` Example from `nostr seed`: ```json theme={null} { "seed_hex": "..." } ``` ## Security notes * Mnemonics and `nsec` values are **full private keys**. Store them securely. * `--no-mnemonic` is recommended when recording logs or terminals. * BIP39 passphrases change derived keys; keep them with the mnemonic. * The CLI is offline-only; it does not contact relays or external services. ## NIP-06 compliance The CLI derives keys according to NIP-06: * BIP39 mnemonic * Derivation path: `m/44'/1237'/'/0/0` `npub`/`nsec` are bech32 encodings per NIP-19. ## Related docs * [Nostr Protocol](../concepts/nostr-protocol) * [Sovereign Agents](../concepts/sovereign-agents) * [Future: Identity](../future/identity) # Spark Source: https://docs.openagents.com/openagents-cli/spark Spark wallet CLI commands verified in local testnet/regtest runs. ## Overview This page lists only the Spark CLI commands that were **personally verified** to work in local testing. Commands not listed here were **not verified** in this pass and are intentionally omitted. ## Verification scope * **Date:** January 28, 2026 * **Environment:** local dev build via `cargo run -p openagents-cli --bin oa -- ...` * **Network:** `testnet` (Spark SDK maps testnet/signet to regtest) * **Auth:** `SPARK_MNEMONIC` set to a BIP39 seed; no API key required for regtest ## CLI entrypoints Both binaries behave the same: * `openagents` * `oa` Examples below use `oa`. ## Verified commands ### Key management ```bash theme={null} oa spark new --words 12 --no-private --json ``` ```bash theme={null} oa spark derive --mnemonic "word1 ... word12" --json ``` ```bash theme={null} oa spark derive --mnemonic "word1 ... word12" --no-private --json ``` ```bash theme={null} oa spark seed --mnemonic "word1 ... word12" --json ``` ```bash theme={null} oa spark pubkey --private-hex <32-byte-hex> --json ``` ### Wallet basics ```bash theme={null} oa spark wallet info --ensure-synced --json ``` ```bash theme={null} oa spark wallet balance --json ``` ```bash theme={null} oa spark wallet network --timeout-secs 5 --json ``` ```bash theme={null} oa spark wallet disconnect --json ``` ### Receive ```bash theme={null} oa spark receive --method spark-address --json ``` ```bash theme={null} oa spark receive --method spark-invoice --amount 1000 --description "test" --json ``` ```bash theme={null} oa spark receive --method spark-invoice --amount 1 --token-identifier --json ``` ```bash theme={null} oa spark receive --method bolt11 --amount 1000 --description "bolt" --expiry-secs 3600 --json ``` ```bash theme={null} oa spark receive --method bitcoin --json ``` ### Send Send a Spark invoice: ```bash theme={null} oa spark send --json ``` Send a Lightning (BOLT-11) invoice: ```bash theme={null} oa spark send --json ``` Send a Spark HTLC transfer: ```bash theme={null} oa spark send --amount 100 --htlc-payment-hash --htlc-expiry-secs 600 --json ``` ### Payments ```bash theme={null} oa spark payments list --limit 5 --json ``` ```bash theme={null} oa spark payments list --limit 1 --asset-filter bitcoin --json ``` ```bash theme={null} oa spark payments get --json ``` ### Parse inputs ```bash theme={null} oa spark parse --json ``` ### Deposits ```bash theme={null} oa spark deposits list --json ``` ### Fiat helpers ```bash theme={null} oa spark fiat currencies --json ``` ```bash theme={null} oa spark fiat rates --json ``` ### Tokens (metadata only) ```bash theme={null} oa spark tokens metadata --json ``` ### Message signing ```bash theme={null} oa spark message sign "hello" --json ``` ```bash theme={null} oa spark message check "hello" --json ``` ### Optimization ```bash theme={null} oa spark optimize status --json ``` ```bash theme={null} oa spark optimize start --json ``` ```bash theme={null} oa spark optimize cancel --json ``` ### Settings ```bash theme={null} oa spark settings get --json ``` ```bash theme={null} oa spark settings set --spark-private-mode-enabled false --json ``` ```bash theme={null} oa spark settings set --spark-private-mode-enabled true --json ``` ## Notes * Mainnet requires an API key; regtest/testnet does not. * For wallet commands, you can pass the mnemonic via `--mnemonic`, `--mnemonic-file`, `--stdin`, or set `SPARK_MNEMONIC`. * Commands not listed here were not verified in this pass (e.g., LNURL, lightning address, faucet, issuer operations, on-chain claim/refund flows). ## Known issues / unverified commands * `oa spark wallet sync`: returned `{}` then the process logged a background `h2` panic (`state=Closed(ScheduledLibraryReset(CANCEL))`). The command itself returned successfully, but this needs investigation in the SDK/runtime. * `oa spark lightning-address check|get|register|delete`: not verified; CLI failed with “LNURL server is not configured”. * `oa spark lnurl prepare|pay|withdraw`: not verified (no LNURL endpoint available in test env). * `oa spark faucet`: not verified (faucet auth required). * `oa spark tokens issuer ...`: not verified; balance returned “No issuer token found”. * `oa spark deposits claim|refund`: not verified (no unclaimed deposits available). * `oa spark send --token-identifier ...`: not verified; token send failed with “No payment created from token invoice”. # Plan Mode (Adjutant) Source: https://docs.openagents.com/plan-mode DSPy plan mode pipeline and optimization loop. ## Overview Adjutant plan mode is a DSPy signature pipeline that produces structured plans for Autopilot tasks. It is designed to be optimizable over time using collected examples and DSPy optimizers. **Status:** Plan Mode is part of Adjutant; current desktop builds connect only to Codex by default. ## Pipeline Stages (Current) The plan mode pipeline orchestrates: * Complexity classification * Topic decomposition * Parallel exploration * Synthesis / deep planning * Validation ## Training and Optimization * Examples are persisted to: * `~/.openagents/autopilot-desktop/training/plan_mode.json` * Optimizers (MIPROv2/COPRO/GEPA) write manifests to: * `~/.openagents/autopilot-desktop/manifests/plan_mode/` * Optimization logs and state live under: * `~/.openagents/autopilot-desktop/optimization/` ## Why It Matters Plan mode turns planning into a measurable, evolvable system. The same pipeline can be improved with new data and recompiled into manifests without rewriting product logic. # Quickstart Source: https://docs.openagents.com/quickstart Run Autopilot Desktop locally from source. ## Prerequisites * Git * Rust toolchain (`rustup`, `cargo`) * macOS: Xcode Command Line Tools (`xcode-select --install`) ## Run Autopilot Desktop ```bash theme={null} git clone https://github.com/OpenAgentsInc/openagents.git cd openagents cargo run -p autopilot-desktop ``` ## First Launch * Confirm the workspace path (Autopilot pre-fills it from your current directory). * Enter a task prompt. * Click **Start Autopilot** to create a session and begin the run. ## Next Steps * Learn the day-to-day flow in [Using Autopilot](usage). * Go deeper on dependencies and builds in [Installation](installation). * Prepare builds in [Distribution and Release](distribution). * Explore the system design in [Autopilot Architecture](autopilot-architecture). * Understand turn-to-turn logic in [Guidance Modules](guidance-modules). # Runtime Codex Worker API Source: https://docs.openagents.com/runtime-codex-worker-api Current implementation of per-user Codex worker lifecycle control through Laravel and openagents-runtime, including ownership, idempotency, and event sequencing semantics. ## Integration model The Codex integration is now implemented as a layered control path. Authenticated clients call Laravel under `/api/runtime/codex/*`, Laravel validates and forwards requests through a signed internal contract, and `openagents-runtime` owns worker lifecycle state, event sequencing, and ownership enforcement. This keeps user auth, token handling, and product-facing APIs in Laravel while moving long-running worker state and control semantics into the runtime where they can be made restart-safe and evented. The runtime side of this integration is not a thin passthrough. It persists worker records, appends durable worker events, and increments monotonic sequence numbers that power deterministic snapshots and stream resume behavior. This is why the Codex worker API is treated as runtime infrastructure instead of a controller-level helper. ## Public API routes in Laravel The currently exposed authenticated routes are: ```http theme={null} POST /api/runtime/codex/workers GET /api/runtime/codex/workers/{workerId} POST /api/runtime/codex/workers/{workerId}/requests POST /api/runtime/codex/workers/{workerId}/stop ``` The create route accepts optional `worker_id`, `workspace_ref`, `codex_home_ref`, `adapter`, and `metadata`. The show route returns a runtime snapshot. The request route accepts a JSON-RPC style `request` object. The stop route accepts an optional `reason` and transitions a running worker to a stopped state. ## Security and principal enforcement Calls into Laravel require Sanctum auth. Laravel then issues signed internal requests that include runtime signature headers and an authenticated principal header (`X-OA-USER-ID`). Runtime does not trust caller-supplied ownership claims; it checks worker ownership against runtime records before allowing snapshot, request, stream, or stop operations. This is important for multi-tenant safety. A caller who knows a `worker_id` cannot operate it unless the durable runtime ownership row matches the caller principal. The ownership check happens in runtime for every worker mutation and read path, not just at worker creation time. ## Lifecycle semantics and idempotency Worker creation is explicitly idempotent on `worker_id` for a matching owner. If the worker already exists for the same principal, runtime reattaches and returns `idempotentReplay=true` rather than creating duplicate runtime state. When a worker is first created, runtime returns accepted semantics; when reattached, it returns replay semantics. Worker stop is also idempotent. Stopping an already stopped worker returns replay semantics and does not append duplicate terminal transitions. Stopping a running worker marks status as stopped, records the stop reason, and appends a durable `worker.stopped` event. ## Request contract and runtime event flow Request submission supports a JSON-RPC style envelope: ```json theme={null} { "request": { "request_id": "req_001", "method": "thread/start", "params": { "prompt": "Audit this PR" } } } ``` At runtime, each request goes through this sequence: ownership check, request validation, event append (`worker.request.received`), adapter dispatch, and terminal append of either `worker.response` or `worker.error`. The response sent back to Laravel includes `worker_id`, `request_id`, `ok`, and `response`, so callers always get a stable request-level envelope even when adapter behavior changes. If `request_id` is omitted, runtime generates one. If `method` is missing or invalid, runtime returns `invalid_request` instead of attempting adapter execution. ## Snapshot and field shapes Snapshot responses currently return runtime-native snake\_case fields, including `worker_id`, `status`, `latest_seq`, `workspace_ref`, `codex_home_ref`, `adapter`, `metadata`, `started_at`, `stopped_at`, and `updated_at`. Create responses currently use camelCase (`workerId`, `latestSeq`, `idempotentReplay`) while stop responses include `idempotent_replay`. This mixed shape is a real detail of the current implementation and should be normalized in a later compatibility pass once clients are pinned to a canonical format. ## Streaming behavior Worker event streaming is implemented internally at: ```http theme={null} GET /internal/v1/codex/workers/{worker_id}/stream ``` The stream supports cursor resume through either `?cursor=` or `Last-Event-ID`, rejects mismatched cursor sources, and enforces stale-cursor handling. SSE `id` values are runtime worker sequence values, which allows deterministic reconnect semantics for stream consumers. Laravel does not yet expose a public `/api/runtime/codex/workers/{id}/stream` endpoint, but runtime support is complete. Public exposure can be added as a relay endpoint without redesigning runtime worker state. ## Adapter state today and production path The active default adapter is `in_memory`, which is deterministic and primarily used to validate lifecycle contracts, event sequencing, and ownership logic. It echoes method and params and increments request count so tests and smoke checks can assert deterministic behavior. This is by design. The transport adapter for real Codex session execution can be swapped in later behind the same worker contract. Because lifecycle APIs and event semantics are already in place, integrating a real adapter is now an execution change, not a control-plane redesign. ## End-to-end usage example Create a worker: ```bash theme={null} curl -X POST "$API_BASE/api/runtime/codex/workers" \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "worker_id": "codexw_demo_1", "workspace_ref": "gs://tenant-workspaces/u_42/repo-a", "codex_home_ref": "gs://tenant-codex-home/u_42", "adapter": "in_memory" }' ``` Send a request: ```bash theme={null} curl -X POST "$API_BASE/api/runtime/codex/workers/codexw_demo_1/requests" \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "request": { "request_id": "req_demo_1", "method": "thread/start", "params": { "prompt": "Audit this PR for regressions." } } }' ``` Fetch snapshot: ```bash theme={null} curl -X GET "$API_BASE/api/runtime/codex/workers/codexw_demo_1" \ -H "Authorization: Bearer $TOKEN" ``` Stop worker: ```bash theme={null} curl -X POST "$API_BASE/api/runtime/codex/workers/codexw_demo_1/stop" \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "reason": "session complete" }' ``` ## Why this matters for OpenAgents This Codex integration gives OpenAgents a runtime-native worker control substrate while preserving the existing Laravel product surface. It is the foundation for hosted coding sessions where worker state must survive reconnects, preserve ownership boundaries, and expose deterministic event histories. With this in place, the remaining work is primarily adapter maturation and public stream/UI wiring, not architectural replacement. # Laravel to Elixir Runtime Bridge Source: https://docs.openagents.com/runtime-laravel-elixir-bridge How OpenAgents routes authenticated API calls from Laravel into openagents-runtime for tool execution and autonomous runtime operations. ## Why this bridge exists now OpenAgents intentionally keeps the web application and user account surface in Laravel while the long-running agent runtime lives in Elixir. The current implementation treats Laravel as the authenticated control plane and the Elixir service as the deterministic execution plane. This gives us a stable user-facing API while still letting the runtime own evented execution, replay-safe state transitions, and tool orchestration. The bridge that now exists is concrete and reusable. Laravel receives authenticated Sanctum requests, validates user input at the edge, injects trusted principal context, signs the internal payload, and forwards to `/internal/v1/*` endpoints on `openagents-runtime`. The runtime validates the signature and principal headers, executes the operation, and returns a structured response that Laravel relays back to the caller. This pattern is now in production shape for runtime tools, skill registry operations, and Codex worker lifecycle operations. ## Current request flow and trust boundary The user-facing endpoints are under `/api/runtime/*` in Laravel. These controllers call typed runtime clients that add retry behavior, timeout budgets, a body hash, and an internal signature header. Runtime-side routes are all under `/internal/v1/*` and require the signed internal auth plug. Laravel sends `X-OA-USER-ID` for authenticated users and forwards trace headers when present so end-to-end request tracing survives proxying. The critical boundary rule is that runtime ownership and authorization decisions are not delegated to untrusted payload fields. Laravel injects authenticated context, and runtime validates ownership from its own records. This prevents user-to-user cross-run or cross-worker confusion even if callers send misleading IDs. ## Runtime configuration surface in Laravel The bridge is driven by `apps/openagents.com/config/runtime.php` in the main `openagents` repository. The active controls include the runtime base URL, per-endpoint path templates, bounded retries, connect and request timeouts, and internal signing key metadata. The same config also exposes routing controls for runtime driver selection and rollback switches, which keeps migration from legacy runtime flows operationally safe. Because this is now used by multiple controllers, runtime requests are not one-off HTTP calls scattered across the app. They are centralized in dedicated clients (`RuntimeToolsClient`, `RuntimeSkillRegistryClient`, and `RuntimeCodexClient`) that all follow the same signature, retry, timeout, and response normalization behavior. ## Reusable tool invocation through the API Tool execution is now exposed through one reusable endpoint: ```http theme={null} POST /api/runtime/tools/execute Authorization: Bearer Content-Type: application/json ``` The payload includes `tool_pack`, `mode`, `manifest` or `manifest_ref`, `request`, and optional policy context. Laravel enforces that the `user_id` in payload cannot disagree with the authenticated principal, injects authenticated `user_id` into the request envelope, then forwards to runtime internal tools execution. Runtime then applies policy checks, manifest validation, and tool-pack dispatch. This endpoint is the canonical way to execute runtime tools through the web API. It avoids adding one Laravel route per tool operation and gives us a single integration seam for policy, receipts, and future settlement enforcement. ## Operator test path via Artisan The implementation also includes a direct smoke command in Laravel console routes: ```bash theme={null} php artisan runtime:tools:invoke-api \ --api-base=http://127.0.0.1:8000 \ --token= \ --mode=replay \ --operation=get_issue \ --repository=OpenAgentsInc/openagents \ --issue-number=1747 ``` This command posts to `/api/runtime/tools/execute` with a full coding payload and prints the status plus JSON response. It was added as an operational verification path so we can test the full control-plane bridge without requiring frontend integration first. ## What this enables next With this bridge in place, OpenAgents can continue shipping runtime capabilities without rewriting the web app and without exposing internal runtime auth details to clients. New runtime primitives can be exposed as additional `/api/runtime/*` routes that reuse the same signing and principal model, and they can be adopted incrementally by existing product surfaces. This keeps migration velocity high while preserving operational control over correctness-critical runtime behavior. # Runtime Tool and Skill Registry Source: https://docs.openagents.com/runtime-tool-and-skill-registry The JSON-defined registry model for runtime tool specs, skill specs, manifest resolution, and immutable skill releases. ## What is implemented OpenAgents now has a runtime-owned registry for JSON tool definitions and JSON skill definitions, exposed through Laravel and persisted in the Elixir runtime domain. This registry is not a planning artifact anymore; it is actively wired into runtime execution. Tool execution requests can resolve integration manifests by reference, and skill specs can be compiled and published into immutable release bundles with content hashes and compatibility reports. The user-facing API surface is available under authenticated Laravel routes. Tool specs are managed through `/api/runtime/skills/tool-specs`. Skill specs are managed through `/api/runtime/skills/skill-specs`. Publishing is done through `/api/runtime/skills/skill-specs/{skillId}/{version}/publish`, and immutable bundle retrieval is done through `/api/runtime/skills/releases/{skillId}/{version}`. ## Built-in compatibility with existing integrations The runtime registry includes built-in first-party entries so the platform works before custom registry rows are created. Today those built-ins include `github.primary` for `coding.v1` and `resend.primary` for `comms.v1`. These built-ins are merged with database-published specs when listing registry content, and manifest resolution can use either source transparently. This means OpenAgents already has direct continuity with the coding and comms integration surface that has been developed so far. The runtime can resolve a manifest for GitHub or Resend by integration identifier without requiring callers to inline complete manifest payloads every time. ## Manifest resolution during tool execution Tool execution is invoked through: ```http theme={null} POST /api/runtime/tools/execute ``` The request may include either a full `manifest` object or a compact `manifest_ref` object. Runtime resolution supports references such as `integration_id` and validates that the resulting manifest matches the requested `tool_pack`. If both `manifest` and `manifest_ref` are present, explicit `manifest` takes precedence, while `manifest_ref` remains the preferred path for registry-backed execution because it keeps payloads smaller and source-of-truth metadata centralized. This behavior is what makes the registry operational instead of decorative. We are no longer treating integration metadata as static prompt text; runtime uses it directly to decide which integration manifest is active for a tool call. ## Skill publish model and immutable releases Skill specs are versioned and can be staged in states such as draft, validated, published, or deprecated. Publishing a skill version triggers runtime compilation into an Agent Skills bundle, computes a deterministic bundle hash, and stores a `SkillRelease` record with compatibility output. That release payload is immutable by version contract and can be retrieved by `skill_id` plus `version`. In practical terms, this gives OpenAgents a traceable path from authored skill JSON to an execution-ready runtime artifact. It also creates a clean seam for later marketplace and billing integration because the release object is a stable unit that can carry provenance, compatibility, and commercial metadata. ## Example registry payloads Tool spec upsert payloads are sent as: ```json theme={null} { "state": "validated", "tool_spec": { "tool_id": "github.custom", "version": 1, "tool_pack": "coding.v1", "name": "GitHub Custom", "description": "Custom coding skill tool", "execution_kind": "http", "integration_manifest": { "manifest_version": "coding.integration.v1", "integration_id": "github.custom", "provider": "github", "status": "active", "tool_pack": "coding.v1", "capabilities": ["get_issue", "get_pull_request"] } } } ``` Skill spec upsert payloads are sent as: ```json theme={null} { "state": "validated", "skill_spec": { "skill_id": "github-coding-custom", "version": 1, "name": "GitHub Coding Custom", "description": "Custom coding workflow skill", "instructions_markdown": "Use coding tools for issue workflows.", "allowed_tools": [{"tool_id": "github.custom", "version": 1}] } } ``` These examples match the currently implemented runtime contract and are accepted by the Laravel bridge controllers that proxy into runtime internal endpoints. ## Where this should evolve next The immediate next step is to make these routes first-class in the public OpenAPI surface so the generated API reference in this docs site reflects them directly. The behavior already exists and is stable in code, but the public spec generation currently lags because these controllers are not yet fully described with OpenAPI attributes. Until that sync lands, this page and the runtime contract docs in the `openagents` repository are the canonical reference for registry operations. # Using Autopilot Source: https://docs.openagents.com/usage Start sessions, run tasks, and understand live updates. ## Start a Session 1. Launch Autopilot Desktop. 2. Confirm the workspace path shown in the setup panel. 3. Enter a task prompt. 4. Click **Start Autopilot** to connect the agent and begin the session. ## What Happens During a Run * The backend connects to Codex (via `codex-acp`) and starts a session. * Events stream back from the agent (messages, tool calls, lifecycle events). * App core maps events into view-model updates. * WGPUI re-renders affected components. ## Approvals and User Input During a run, the agent may request approvals (command execution, file changes) or ask for user input. These requests are captured in the turn summary and can pause the run depending on guardrails. ## Unified Events (Chat Stream) Autopilot emits a unified event stream with message chunks, tool calls, and session lifecycle events. These are normalized across agents, even when the underlying protocols differ. ## Current Limitations * Unified events are emitted but not yet rendered directly in the canvas UI. * Some events may be duplicated while raw ACP and unified events coexist. * Error handling is improving; some failures may surface only in logs. # WGPUI Source: https://docs.openagents.com/wgpui GPU-accelerated UI system for OpenAgents desktop and web surfaces. ## Overview WGPUI is the GPU-accelerated UI rendering library used across OpenAgents. It provides a cross-platform rendering stack built on `wgpu`, a component system, a layout engine, and testing utilities for high-performance UI surfaces (chat threads, tool call cards, streaming markdown, etc). WGPUI is **inspired by GPUI** (the GPU-accelerated UI framework built for the Zed editor) and borrows the same entity-driven, render-tree approach to building immediate-mode views on top of a retained state model. [Effuse](effuse) is the Effect-native TypeScript UI stack used by the Autopilot Web MVP. The former desktop stack (Tauri + Effuse) has been removed; WGPUI is the canonical UI layer for Autopilot Desktop. ## Where it lives in the repo * `crates/wgpui/` - core GPU UI library (renderer, layout, components, testing) * `crates/autopilot_ui/` - shared WGPUI surfaces for Autopilot * `crates/autopilot_app/` - app core consumed by both CLI and desktop * `apps/autopilot-desktop/` - native desktop shell using WGPUI + winit * Examples and demos: * `crates/wgpui/examples/storybook/` * `crates/wgpui/examples/component_showcase.rs` * `crates/wgpui/examples/first_light.rs` ## Core architecture WGPUI is organized around an entity-driven UI model and an element tree pipeline: * **App + Entity system**: typed state containers and reactive notifications (`crates/wgpui/src/app/*`). * **Element tree**: layout, prepaint, and paint phases for UI elements (`crates/wgpui/src/element/*`). * **Scene**: accumulated draw commands (quads, text runs, images, lines). * **Renderer**: wgpu pipelines, GPU buffers, and shaders (`crates/wgpui/src/renderer.rs` and `crates/wgpui/src/shaders/*`). * **Layout**: Taffy-based flexbox layout engine (`crates/wgpui/src/layout.rs`). * **Text**: cosmic-text shaping and glyph atlas (`crates/wgpui/src/text.rs`, `crates/wgpui/src/text_system/*`). * **Input + actions**: platform-agnostic events and a keymap/action system (`crates/wgpui/src/input.rs`, `crates/wgpui/src/action/*`, `crates/wgpui/src/keymap/*`). For a full reference, see `crates/wgpui/docs/README.md` and `crates/wgpui/docs/phase1-foundation.md` in the OpenAgents repo. ## Component system WGPUI ships a rich UI kit organized by level: * **Atoms**: badges, indicators, icons, status pills, etc. * **Molecules**: cards, rows, headers, input clusters. * **Organisms**: complex UI blocks like tool call cards and message rows. * **Sections**: higher-order panes like thread views and editors. * **HUD**: overlays, command palette, status bars, modals. Core components include `Div`, `Text`, `Button`, `TextInput`, `ScrollView`, `VirtualList`, `Modal`, `Dropdown`, and `Tabs` (`crates/wgpui/src/components/*`). ## Platform support * **Desktop**: `winit` + native wgpu swapchain. * **Web**: `wasm-bindgen` + WebGPU. The platform abstraction is in `crates/wgpui/src/platform.rs`. ## Testing WGPUI includes a dedicated test harness and snapshot tooling in `crates/wgpui/src/testing/`. This is the foundation for UI regression tests and layout inspection in the WGPUI migration plan. ## Autopilot Desktop (WGPU) The native WGPU desktop app lives in: * `apps/autopilot-desktop/src/main.rs` That entrypoint wires: * `autopilot_app` (core app state and event flow) * `autopilot_ui::MinimalRoot` (WGPUI view root) * WGPUI renderer + text system * `winit` event loop and window lifecycle This is the canonical path for the desktop UI going forward. ## Migration status See `apps/autopilot-desktop/docs/migration/EFFUSE_TO_WGPUI_PLAN.md` in the OpenAgents repo for the full migration plan and architectural decisions.