When an agent’s answer can steer a patrol vessel, reliability stops being a benchmark score. Shippy, the maritime agent built by Ai2’s Skylight team, serves maritime domain awareness — a domain the team describes bluntly: the wrong answer has real impacts. Their retrospective in the technical post lands on a direct conclusion: in high-stakes operational domains, the hard problem is not picking a stronger model. It is building a system you can trust to be correct, to stay within its limits, and to stay stable across tasks — verified against live data that keeps updating.
That framing turns agent reliability from a model problem into an engineering problem, and the post’s value is how it decomposes the problem into four layers a builder can act on.
A three-part anatomy: soul, skills, config
Shippy defines the agent as three independently evolvable pieces:
| Layer | Contents | Responsibility |
|---|---|---|
| Soul | system prompt | persona and behavioral boundaries |
| Skills | markdown with structured frontmatter | workflows for specific request types |
| Config | remaining runtime settings | harness, model, runtime |
Skills follow the same agent-skills spec used by Claude Code and Codex, and cover Skylight API queries (events, vessel data), EEZ and MPA boundaries, vessel-track interpretation, and interactive map deep links. The track-interpretation skill builds on the activity classifications Skylight’s existing models — including Atlantes — already produce, rather than asking the agent to re-read raw signals from scratch.
Soul and skills ship inside a versioned Docker image, with secrets injected at runtime. Swapping the LLM or the agent harness is a config change, not a rebuild — today the harness is OpenClaw, an open-source agent framework, and the model is Claude Opus 4.6. The real payoff of the separation is governance: Shippy does not make legal determinations about whether a vessel is breaking the law, and it does not speculate beyond what the data supports. Those boundaries are explicit in the system prompt, not implicit in fine-tuning, which makes them auditable, versioned, and eligible for review and eval.
Deterministic tools: a purpose-built CLI over a complex API
Early prototypes let Shippy assemble API calls from scratch. The Skylight API exposes dozens of input types, nested filters, pagination cursors, and complex geometry, and the result was a steady stream of subtle bugs: malformed pagination silently dropping results, geometry encoding errors, and correct-looking queries returning wrong data because a filter type was misunderstood. The nasty part of these bugs is that they look right — the agent keeps reasoning on bad data, and a human only notices downstream.
The fix is interface convergence. Shippy issues no raw API calls; it goes through a purpose-built CLI: a single skylight events search with typed filter flags, while the CLI handles authentication, pagination, and structured output. The CLI carries full --help and clear error messages, so the agent can recover from a mistake instead of guessing. Output is always written to a local JSON file rather than piped through the shell — large result sets once hit pipe buffer limits and broke downstream tools like jq — and writing to disk also lets later steps reuse the results. Place-name resolution follows the same principle: asked about “fishing activity in Panama’s EEZ,” the skill resolves the name to a boundary polygon through Skylight’s regions API instead of guessing or hard-coding coordinates, and attributes results with deep links and data sources such as Global Fishing Watch and TMT.
Under the CLI sits a standardized API: events, vessels, regions, satellite imagery, and vessel tracks share two operations — search and aggregate — with inputs and outputs defined as typed schemas carrying field-level descriptions. The point of the layering is independent testability: the API has its own suite, the CLI can be exercised by humans or agents, and skills reference CLI commands instead of reinventing them. Each layer narrows what the next layer can get wrong.
Mothership: one sandbox per session
Skylight is a free platform serving, by the team’s own numbers, 300+ partners across 70+ countries — mostly government agencies and NGOs — which makes multi-tenant isolation one of the project’s biggest engineering challenges. The team built Mothership, an agent hosting platform that provisions a dedicated Kubernetes deployment for every user session. Pods package the agent runtime, skills, and CLI; the user’s Skylight JWT is injected at provision time, so API calls are automatically scoped to that user’s data; files created inside a session are never shared across users; the sandbox can write code and run multi-step analysis, but its network is limited to the services it needs.
What is worth noting is where the isolation happens: not by filtering in the application layer, but at runtime injection and the deployment boundary. Application-layer filtering depends on “remembering to check every time”; Mothership depends on “there is no path across the boundary at all.”
Evaluating the whole agent: live data and an LLM judge
Static benchmarks cannot grade a wired-in agent — how it selects tools, queries live data, and knows when to stop. Shippy’s eval scores the whole agent (model plus skills plus sandbox) against real data: subject-matter experts write scenarios and rubrics, set weights, and annotate ground truth; an LLM judge grades each criterion from 0 to 1 with written reasoning for why it passed or failed, and the weighted aggregate is checked against a fixed pass threshold.
Execution goes through a plugin for Harbor, an open evaluation framework: it spins up a real Shippy session on the version under test, runs against the same real data users see, in parallel, producing timestamped results and a report of score changes against the previous run. Any change to skills, model, or underlying data triggers a rerun. The key positioning: eval is the release gate — a version that regresses on the criteria does not reach end users.
Known wins and the failure modes that remain
In the latest eval round, guardrail tasks performed consistently: correctly refusing military intelligence requests, maintaining user data isolation, and attributing sources accurately. The failure modes were just as concrete: patrol-planning tasks where Shippy overstepped into tactical recommendations, boundary simplification that caused missed events, and one case where the agent invented a CLI command that does not exist. None of these are visible in a model benchmark, and each fed directly back into skill revisions.
Limits and builder guidance
The limits deserve equal airtime: there is no cross-thread memory yet, so analysts restate their jurisdiction each session; the map returns deep links only, with agent-driven UI control still ahead; easy and hard queries currently share one model, and the team plans model routing to send simple lookups to smaller ones; and Shippy is opening to early adopters on a rolling basis to stress-test it. Mothership itself was designed as general agent hosting — the lessons are slated to carry over to Ai2’s other projects, starting with EarthRanger.
Three judgments a builder can take away. First, put behavioral boundaries in a version-controlled system prompt, not in fine-tuning or team folklore — only auditable boundaries are testable boundaries. Second, rather than exposing a complex API to the agent directly, invest in a typed, self-documenting CLI that keeps the nondeterminism in the model and the determinism in the tools. Third, treat eval as a release gate, not an after-action report: whole-agent, live data, weighted rubrics, with regressions stopped before end users. Production agent reliability is not bought with a stronger model; it is accumulated by shrinking the error surface layer by layer.
Sources
AI-assisted summary compiled from the sources above, reviewed by a human before publishing.