100 agents, zero API keys
What breaks when 100 coding agents share a machine, and how we built solos-swarm in Rust to sandbox workers, fence stale writes, and keep the holdout test suite in another room.
The easiest way to run 100 coding agents in parallel is to spawn 100 terminal processes, pass your OPENAI_API_KEY or DEEPSEEK_API_KEY in the environment, point them at a shared directory, and hope they stay out of each other's way.
Anyone who has tried this on a real task knows how it ends. Two agents edit the same file at the same time. A third gets stuck in a retry loop and quietly burns through your monthly token budget while a connection drops and retries. If you are running an evaluation benchmark, at least one agent will eventually grep the disk for the answer key and report a perfect score.
I built solos-swarm because I wanted to run 100 or more @earendil-works/pi-coding-agent workers on a Linux machine without trusting any single worker process with a raw API key, an unmediated network socket, or the test suite used to score its work. Recent work like Dream-RSI shows how much leverage you get when you wrap an unmodified coding agent in a lightweight orchestration layer that treats accumulated discovery history as a replay simulator rather than changing the underlying agent itself. solos-swarm is the systems layer underneath that kind of loop.
Stripping the worker down to six tools
A standard coding agent comes bundled with bash, file editors, network fetchers, and direct model credentials. That works fine when one human is watching one cursor move across a laptop screen. It falls apart the moment 100 autonomous workers run unattended.
In solos-swarm, a worker process runs inside an unprivileged Linux sandbox built from bubblewrap, a read-only SquashFS root filesystem, Landlock v3 rules, a seccomp-BPF syscall filter, and a systemd cgroup scope. All built-in bash, filesystem, and direct model tools are stripped out before the Node.js process starts. The worker only sees a small TypeScript bridge at extensions/broker-tools.mjs that talks back to the Rust control plane over mutual TLS using six governed tools: claim_task, board_read, board_post, submit_finding, read_findings, and submit_artifact.
When a worker wants to call a model, it sends the prompt to the Rust broker. The broker counts the exact tokens offline using a SHA-256 verified tokenizer, reserves the worst-case cost in micro-USD before touching the network, attaches the upstream provider key on the control-plane side, and returns the completion. If you inspect the memory, environment variables, filesystem, or logs of all 100 worker sandboxes during a live run, the count of provider keys inside them is zero.
There is a real cost to locking a worker down this tightly. You lose the ability for an agent to improvise with an ad-hoc shell pipeline or install a missing package on the fly. Every deterministic tool has to run inside a separate wasmtime WASI sandbox with strict fuel and memory caps, or inside a dedicated microVM. Setup takes longer before the first prompt ever runs.
Fencing tokens and the dropped-socket trap
Coordinating 100 workers through a shared state store introduces two classic distributed-systems bugs: zombie workers overwriting fresh state, and dropped network calls silently double-spending your budget.
In solos-swarm, every workflow starts with a Goal that defines the objective, the maximum delegation depth, and the budget ceiling in integer micro-USD, where one dollar equals 1,000,000 micro-USD. When a worker claims a Task, the multi-group Raft store increments its revision counter and issues a monotonic fencing token bound to that specific task, worker, lease, and revision. Every subsequent tool call, board post, or model request must carry that exact fencing token. If worker-03 stalls past its lease timeout and another worker takes over the shard, any late message from worker-03 carries an older token and gets rejected at the gate.
Budget tracking uses the same pessimism. Before the broker sends an inference call to DeepSeek or a local llama.cpp server, it reserves the maximum possible cost across both the task envelope and the parent goal envelope:
If the upstream HTTP connection drops after the request bytes leave the socket, the broker does not release the hold and does not automatically retry the call. It marks the reservation Unknown and keeps the funds locked until an operator reconciles the receipt. That rule feels stubborn the first time a flaky connection freezes a slice of your budget, but it prevents a loop of timed-out retries from draining an account while the upstream API is still billing for every in-flight completion.
Keeping the answer key in another room
When we used a swarm to triage OpenAI's 719 math preprints and test regime models, the hardest part was making sure the proposer agents could iterate on candidate solutions without overfitting to the holdout set or reading the expected answers off the disk.
To prevent that, solos-swarm splits the runtime across three separate Unix user accounts and three separate daemons:
| Daemon | Command | Unix UID | What it holds |
|---|---|---|---|
| Control Plane | solos-swarm mvp serve | solos-swarm / 999 | Raft state, capability broker, cap ledger; cannot read holdout files or witness keys |
| Isolated Evaluator | solos-swarm mvp evaluator-serve | solos-eval / 995 | 0700 protected test bundles; scores search attempts and runs final_holdout once per trial |
| Audit Witness | solos-swarm mvp witness-serve | solos-witness / 994 | Verifies RFC 6962 Merkle consistency proofs and co-signs append-only tree heads |
Even the control plane cannot read the protected test bundle inside /var/lib/solos-swarm/evaluator/protected. During a run, workers submit candidate artifacts over a local socket and receive only an Ed25519-signed integer score on the search split. The final_holdout split is locked behind a one-shot gate that only fires once the trial finishes.
What happened when we ran the swarm live
On 12 October 2026, we ran the synthetic reconciliation benchmark live-20260412-132831 on a 128-core Debian 13 host using Node 24.18.0, @earendil-works/pi-coding-agent 1.1.0, and deepseek-chat.
In the initial ten-shard reconciliation cohort, workers agent-00 through agent-09 booted inside their bubblewrap and Landlock sandboxes with a one-dollar budget ceiling of 1,000,000 micro-USD. Each worker claimed one shard with zero duplicate claims, compared the records, posted one progress summary to the append-only Board, and submitted a cited Finding. Each worker took two model calls and finished its shard in 50.8 to 58.6 seconds.
Once all shard findings landed, agent-00 read the verified findings, ran two synthesis calls over 66.2 seconds, and submitted the final deduplicated artifact containing all 15 inconsistent records in sorted order. The run finished in 124.78 seconds across 22 model calls and 44,726 total tokens, split across 40,264 input tokens and 4,462 output tokens. Total settled spend came to 17,438 micro-USD, or $0.017438, with zero leftover holds and zero leaked keys.
We also built a matched 2x2 campaign runner via solos-swarm mvp campaign so we can test exploration policies against historical discovery trees the way Dream-RSI describes, without fooling ourselves about when a larger swarm actually beats a single worker. Every trial seed runs four cells under the exact same budget, call cap, and deadline: a single worker versus a parallel worker cohort, crossed with scored_history, which packs the top-K prior scored attempts into context, versus single_draft_refinement, which keeps only the single best draft. If a single worker with a clean context beats a parallel swarm on a task family under the same dollar cap, the campaign report shows it right away.
The repository is live at GuiBibeau/solos-swarm with the full Rust control plane, the WASI and Bubblewrap sandboxes, the offline Merkle verifier, and the four-part technical reference under docs/. Scaling from ten-worker cohorts to 100-agent and 500-agent runs under the same zero-key invariants is detailed in docs/BENCHMARKS.md.