Agent 02
Loomcell
The training agent
Reading time: 11 min
Loomcell is the interface for work that runs for days. You describe the run and the budget; the agent negotiates placement, holds the escrow, and decides when to checkpoint.
- of prior cost per trained token
38%
of prior cost per trained token
- nodes in a typical run
400
nodes in a typical run
- runs lost while learning
2
runs lost while learning
Checkpoint cadence turned out to be the entire problem.
The failure mode was not exotic. A synchronous data-parallel run across four hundred nodes is only as available as its least reliable participant, and early users set a checkpoint interval by habit — every thirty minutes, the number they had used on a single-tenant cluster where nodes did not simply leave.
On a network of independently operated hardware, thirty minutes was both too often for the reliable racks, which wasted throughput, and far too rare for the volatile ones, which lost half an hour of work every time an operator's power contract changed.
The fix was to stop exposing the number at all. The agent now derives checkpoint cadence per placement from that operator's own measured departure rate, and pins the run to a contiguous rack where it can. Nobody has set the interval by hand since.
“The honest version is that we lost two runs before we understood checkpoint cadence. Once the agent started setting it from the operator's own history instead of our guess, we stopped thinking about it entirely.”
Teodora Vance
Research Infrastructure Lead, Loomcell
Completed without intervention+41pts
Recovered from checkpoint-9pts
Lost-32pts
Source: Loomcell run logs, 60 runs before and after