The handoff is the part nobody writes down

Forward-deployed engineering turned into a genre this past year. Most of what got written covers the front of the job: how to hire these people, how they sit inside a customer’s problem, how they ship something real in a week.

The end isn’t covered. You embed. You build something real. At some point you leave, and the thing you built is supposed to keep working without you.

That transition has its own engineering. It stays invisible because it happens weeks after the demo everyone remembers, and because doing it well looks like nothing happening. This post is its mechanics on one kind of system: a fleet of LLM agents a client’s own team has to run, drill, and eventually defend to an auditor, with me gone.

The production readiness review

The problem isn’t new. It is worked out in detail somewhere outside the forward-deployed literature, because Google’s SRE organization faced a structurally identical question: under what conditions will one team take on operational responsibility for a system another team built?

Their answer is the Production Readiness Review. A PRR is a criteria-based assessment covering six areas: system architecture and its interservice dependencies; instrumentation, metrics, and monitoring; emergency response; capacity planning; change management; and performance across availability, latency, and efficiency. The part worth stealing is its status. Passing the PRR is a prerequisite for the SRE team to accept the service at all. It decides whether the transfer happens. Most review documents get filed after the transfer already has, which decides nothing.

Two properties carry over to a client engagement. The receiving team runs the review, the one to three engineers who will hold the pager. The people who built the system don’t. And the criteria are agreed in month one, while everyone is fresh and the departure is still hypothetical.

The rest of this post is the agent-platform version of those six areas.

No central registry

The choice that most determines whether a fleet of agents survives your departure is a dull one. There is no central registry. To add an agent you add a folder. To remove one you delete it. Everything the platform needs to know lives inside that agent’s directory, and CI finds agents by walking the tree.

agents/
  claim-summarizer/
    agent.yaml          # manifest: owner, model pin, rubric, paths
    prompt.md
    evals/
      cases.jsonl       # 40-case labeled acceptance set
    scorecard.yaml      # thresholds the client reads on their own
    runbook.md          # what to do when it pages
  label-checker/
    agent.yaml
    ...
platform/
  discover.py           # walks agents/, no central list
  promote_gate.py

The manifest is the whole contract for one agent: an owner on the client’s team, an exact model version, and pointers to the files sitting beside it.

# agents/claim-summarizer/agent.yaml
name: claim-summarizer
owner: platform-team              # the client's team owns this, not the vendor
entrypoint: app.agents.claim_summarizer:build
model:
  provider: internal-gateway
  name: gpt-4o
  version: "2024-11-20"           # pinned; a bump is a revalidation event
rubric_version: v4
prompt: prompt.md
eval_set: evals/cases.jsonl
scorecard: scorecard.yaml
runbook: runbook.md
oncall: "#claims-platform"

Discovery is a glob and validation is a schema. The one line that earns its place is extra="forbid".

# platform/discover.py
from __future__ import annotations
import hashlib
from pathlib import Path
import yaml
from pydantic import BaseModel, ConfigDict, Field, model_validator

AGENTS = Path(__file__).resolve().parent.parent / "agents"


class ModelPin(BaseModel):
    model_config = ConfigDict(extra="forbid")
    provider: str
    name: str
    version: str                      # exact, never a moving alias


class AgentManifest(BaseModel):
    # a key the manifest does not declare is an error, not a silent drop
    model_config = ConfigDict(extra="forbid")
    name: str
    owner: str
    entrypoint: str
    model: ModelPin
    rubric_version: str
    prompt: Path
    eval_set: Path
    scorecard: Path
    runbook: Path
    oncall: str
    dir: Path = Field(exclude=True)

    @model_validator(mode="after")
    def _anchor_paths(self) -> "AgentManifest":
        # paths are written relative to the agent's own folder; anchor them so
        # every caller reads the same file no matter what directory it runs from
        for field in ("prompt", "eval_set", "scorecard", "runbook"):
            p: Path = getattr(self, field)
            if not p.is_absolute():
                setattr(self, field, self.dir / p)
        return self

    @property
    def fingerprint(self) -> tuple[str, str, str]:
        # model, prompt and rubric are one versioned artifact
        prompt_sha = hashlib.sha256(self.prompt.read_bytes()).hexdigest()[:6]
        return (self.model.version, prompt_sha, self.rubric_version)


def discover() -> list[AgentManifest]:
    """Every directory under agents/ holding an agent.yaml is an agent. There is
    no central registry to keep in sync: to add one, add a folder."""
    found = []
    for manifest in sorted(AGENTS.glob("*/agent.yaml")):
        raw = yaml.safe_load(manifest.read_text())
        found.append(AgentManifest(**raw, dir=manifest.parent))
    return found

I overclaimed that line for a while, so it is worth being exact about what it buys. Every field above is required, so misspelling one fails on its own: write oncal and pydantic reports a missing oncall whether or not you forbid extras. What extra="forbid" catches is the opposite case, a key the schema never had.

Someone adds escalation: platform-tier-2 because it reads like something that ought to work, but the key is inert. Every required field is still present, so the manifest validates. The agent runs with default escalation while the file in version control says otherwise, and nobody sees the discrepancy until an incident goes to the wrong rota.

That is this whole post in miniature: a failure that is an absence rather than an error. With the line, discover() refuses the manifest and the pull request goes red.

The schema can’t see the other absence. owner and oncall are strings, and a string stays valid after the team it names is reorganised and after the channel it names is archived. Pydantic can tell you the field is there, but it can’t check that the value resolves to a live rota. So a second check resolves both against the client’s directory and chat workspace on every pull request, and a manifest naming an empty rota fails the way a missing field fails. Otherwise the first person to find out that the escalation target is empty is the escalation.

CI builds its job matrix from the same walk, and it keys on the same thing discovery keys on, the presence of an agent.yaml.

# .github/workflows/agents.yml
name: agent-evals
on: [pull_request]

jobs:
  discover:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.find.outputs.matrix }}
    steps:
      - uses: actions/checkout@v4
      - id: find
        # the matrix is every directory that holds an agent.yaml, the same key
        # discover.py uses; a folder without a manifest is not an agent to either
        run: |
          agents=$(find agents -maxdepth 2 -name agent.yaml -printf '%h\n' \
            | xargs -n1 basename | sort | jq -R . | jq -sc .)
          echo "matrix={\"agent\":$agents}" >> "$GITHUB_OUTPUT"

  evaluate:
    needs: discover
    strategy:
      fail-fast: false
      matrix: ${{ fromJson(needs.discover.outputs.matrix) }}
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      # owner and oncall have to resolve to people who exist this week
      - run: python platform/check_referents.py agents/${{ matrix.agent }}
      - run: python platform/run_scorecard.py agents/${{ matrix.agent }}

Keying both walks on the manifest matters more than it looks. An earlier version had discover.py require an agent.yaml while CI listed every directory under agents/ with ls -d agents/*/. A folder without a manifest became a failing matrix entry the Python side couldn’t see. A nested agent was visible to Python and invisible to CI. Two discovery paths have to agree on what an agent is, or you have rebuilt the central registry by accident through a side door.

A registry is a file somebody has to own. While I’m embedded I own it without noticing. Keeping it in sync is second nature to me and never surfaces as work. But the day I leave it turns into a trap: someone adds an agent, forgets the registry row, and the agent is never evaluated. Nobody notices for weeks.

I measured what the registry cost once, holding the agent’s content fixed so only the plumbing varied. Registering an already-written agent, prompt and 40-case acceptance set and runbook identical either way, meant editing five shared files: the registry, the CI matrix, a dashboard config, a scorecard index, and a secrets map. The keyboard work was about twenty minutes, but five shared files pulled in three reviewers from other teams, and the median elapsed time from branch to merge was eleven hours. Drop-a-directory changed nothing shared. The only reviewer was the team that owned the agent, and the median was twenty-five minutes.

The zero carries the section, more than the minutes. Zero shared files means zero maintainers to route through, which is what lets the client’s team grow the platform without reproducing the role I used to play.

Scorecards a stranger can read

A platform a client can’t see into is a platform they will call me about. So every agent publishes a scorecard written for someone who never met me. Each metric has a floor or a ceiling and a number, and “is this agent healthy” gets an answer instead of a discussion.

Written for a stranger and findable by one are separate problems. The client’s support lead calls this the claims bot. The alert calls it a groundedness floor breach. The repository calls it claim-summarizer, and so does every document I wrote about it.

Three vocabularies, none of them a search hit for the other two, and the person typing at three in the morning has the first two. So stop depending on search. The alert payload carries the runbook’s path, and its symptom line is written in the words of whoever complains rather than the schema’s.

# agents/claim-summarizer/scorecard.yaml
metrics:
  groundedness:
    measure: share of statements supported by retrieved context (LLM judge)
    floor: 0.90
    window: last 200 cases
  answer_accuracy:
    measure: exact-match on the 40-case acceptance set
    floor: 0.80
  p95_latency_ms:
    ceiling: 4500
  cost_per_1k_calls_usd:
    ceiling: 6.00
owner: platform-team
review: weekly

The rendered version is plain text on purpose, so the client’s read on an agent never depends on a dashboard login they will lose.

claim-summarizer   scorecard   (week of 2026-06-15)
------------------------------------------------------------
  groundedness          0.93    floor 0.90    PASS
  answer_accuracy       0.88    floor 0.80    PASS
  p95_latency_ms        4210    ceil  4500    PASS
  cost_per_1k_calls     5.10    ceil  6.00    PASS
  fingerprint           gpt-4o-2024-11-20 / prompt 0de9bf / rubric v4
------------------------------------------------------------
  status: HEALTHY         next review: 2026-06-22  (owner: platform-team)

Those floors need a note, and it isn’t the one I first wrote. My original argument was that 40 cases are too noisy for a floor at 0.85 while 200 cases are tight enough for one at 0.90. The second half doesn’t survive the first. At 0.93 over 200 cases the 95% interval runs 0.895 to 0.965, and 0.90 sits inside it. That is the objection I had just made to 0.85 on 40 cases.

No floor on a single reading escapes this. What makes a floor usable is the consecutive-window rule further down, and never the width of one interval. At a true rate of 0.93 the standard error over 200 cases is 0.018, so a single window falls below 0.90 about 5% of the time by chance; requiring two consecutive windows puts that at roughly one run in 430. On the 40-case set the same rule isn’t enough to rescue 0.85, which trips 28% of the time per window and 8% of the time twice running. That is why that floor is 0.80: 6% per window, and about one release in 280 for a pair. The acceptance set is a coarse tripwire for a large regression. The 200-case window is where a small one is detectable at all.

The acceptance set has its own expiry, and the labels go stale well before the cases do. A few of them are there because two people argued in month two and the client’s compliance lead ruled that a summary omitting a settled-claim date is wrong even though every sentence in it is true. The label records the ruling but loses the argument. When a model bump later flips that case, there is no way to tell from the set whether the model got worse or whether the case was always a judgement call. So the ruling now rides in the case record next to the label:

{"id": "clm-0031", "contested": true, "adjudicated_by": "compliance-lead",
 "why": "omitting the settled date reads as an open claim, though each sentence is true"}

They look like noise to whoever inherits the set, and a tidy-up deletes them first.

The fingerprint row does most of the work, but it’s the one people leave off. The judge behind groundedness is itself a model, with the same length, position, and self-preference habits as any other, so model, prompt, and rubric together are one versioned artifact. When any of the three changes the scorecard is measuring with a different ruler, and the client needs to see that without me in the room.

Runbooks broken on purpose

Runbooks live next to the agent, carry an owner and a last-drilled date, and get exercised by breaking the thing in staging and timing the fix.

# Runbook: claim-summarizer paging on a groundedness drop

**Owner:** platform-team   **Pager:** #claims-platform   **Last drilled:** 2026-06-02

## Symptom
Scorecard `groundedness` below 0.90 for two consecutive windows.

## First checks (5 minutes)
1. Is retrieval returning empty?      `make check-retrieval AGENT=claim-summarizer`
2. Did the index rebuild overnight?   `make index-age AGENT=claim-summarizer`
3. Did the model change under us?     compare the scorecard fingerprint to last week.

## Fix
- Stale index:  `make reindex AGENT=claim-summarizer`, then re-run the acceptance set.
- Model drift:  pin back to the last-validated version in `agent.yaml`, open a
  revalidation ticket, and do not silently accept the new checkpoint.

## Drill
Tested monthly by breaking retrieval in staging and timing a resolution, run by
whoever holds the pager that month on their own credentials. The bar is 60 minutes,
a clean reindex plus verification; the last four drills ran 52 to 58. A drill that
runs long means the runbook is wrong, not the runner.

The drill section is the one people skip, and most of what it tests isn’t knowledge. A runbook is a list of instructions plus an unwritten claim that whoever reads it is allowed to run them. make reindex writes to a bucket. Pinning a model version back edits a file covered by CODEOWNERS, and GitHub’s documented behaviour there is that a code owner who lacks write access is simply not assigned, with no error on the pull request, only a highlight on a file page that nobody passes through during review. None of that was visible to me. I was granted everything in week two and never lost it.

So the drill that counts is the one run from the pager holder’s laptop with their credentials, and the last one runs once my access is already revoked. Offboarding a contractor is the only change an engagement ships without a test, and it’s the one most likely to turn a runbook step into a step the remaining team lacks the permission to execute.

A dead permission produces no distinct signal either. The retrieval wrapper degrades to empty context on any error, so a revoked read looks exactly like the stale index the first check is written for, and the on-call engineer spends the night rebuilding an index that was fine.

Not only my experience: a study of high-severity incidents in a large cloud service found a class of them where mitigation stalled after the root cause was already known, and the largest share of the delay it blamed on documentation came from runbooks that were followed and still did not say which team to engage.

Two lines are worth tracking: how many incident types have a written runbook, and how many have one that has passed a live drill. The gap is the amount of false confidence in the system at any moment, and for a while I reported only the top line.

The drill line goes backwards in month six, which is the part of that chart I would have quietly smoothed a year ago. Two new agents shipped that month with runbooks written the same week and drilled the month after, so the denominator moved before the numerator did. A drill line that only ever rises is one measured against a frozen inventory.

The cost of getting this wrong is the clearest number I have. On an earlier engagement I built an index-rebuild process that was clever and entirely mine. It did an incremental swap with no downtime. It ran cleanly every time. I wrote a runbook and walked two of their engineers through it, and my own resolution time for a stale index sat around 55 minutes across four runs.

Six weeks after I rolled off, the upstream document store changed how it paginated, the incremental swap half-completed, and the index served a mix of two document versions. The runbook covered the failure I had imagined, a clean rebuild, but said nothing about the one that happened. The team followed it, it didn’t help, and the single incident ran three hours before someone called me.

That chart is worth reading precisely. Four failure classes held roughly steady when the client’s team took over: quota trips from 22 to 24 minutes over nine incidents then seven, token expiry from 18 to 21 over seven then six, agent regressions from 47 to 53 over five then five, pipeline stalls from 34 to 45 over eight then six. That is a spread from 9% to 32%, which is what medians over five to nine incidents look like; if these four had come out within a couple of points of each other I wouldn’t believe them either. Those runbooks had been drilled. The index rebuild broke the pattern, but it is a single incident and not a median, three hours against my 55-minute baseline. Their engineers weren’t worse than me. The one number that moved covers the procedure nobody had ever exercised against its real failure.

Keeping the client able to leave

Lock-in arrives by default, and each default was the fast choice at the time. I make it visible at that moment, usually as a one-paragraph decision record, and ask one question: on the day we leave, can they run this without us and without renewing anything.

# ADR-014: No proprietary workflow engine for agent orchestration

## Status: accepted

## Context
The vendor orchestrator saves about two weeks of build time. It also means the
client cannot run, inspect, or change a workflow without a license and our help.

## Decision
Orchestrate with plain Python and the queue the client already runs in production.

## Consequence
We ship about two weeks slower. In exchange, on the day we leave there is no
component only we can operate, and nothing to renew to keep the lights on.

I didn’t think about that ADR again until five months after leaving, when the client wanted to change how a workflow retried failed steps. It was a small edit to a plain Python file, made by one of their engineers on a Tuesday. With the vendor engine it would have been a support ticket to a company that wasn’t me, and a wait.

An ADR covers the decisions that had a meeting. But most decisions didn’t have one. They had two people arguing in a pull request about why the obvious simpler version does not work, and that argument is the reasoning the diff does not carry.

It is also the layer that does not travel with the repository. A clone carries commits, review threads live in the forge’s database, and a client who migrates hosts, or a vendor whose work happened on a fork, hands over code with its rationale left behind. So a reviewer who has just explained why something can’t be simpler doesn’t approve yet. The explanation goes into the file first.

At the start of an engagement the bus factor of everything I built is one, and that one is me. CI/CD and the on-call rota start at two, because those predate me and I was adding to someone else’s work. Every anti-lock-in choice moves those bars up, because each removes something only a licensed specialist or only I could touch.

That chart is computed rather than asserted, which matters because bus factor is otherwise a thing people claim about their own projects. The literature calls it truck factor, and the original formulation from Avelino and colleagues scores each developer’s degree of authorship over each file from commit history, calls someone a file’s author above a threshold, then removes whoever authors the most files, one after another, until more than half the files have no author left; the number of removals is the truck factor. Bus Factor Explorer produced the bars above, and the formula it implements is not Avelino’s but the later one from Jabrayilzade and colleagues, in which authorship decays exponentially and halves about every five months, so someone who has not touched a file in a year barely counts. The tool adopted it because that paper reports better estimates than the original. Computing per subdirectory is what turns one number for a repository into per-subsystem bars.

One caveat is that truck factor counts authors, but an author isn’t automatically an operator. Someone can have written half a subsystem and still not be able to run it at three in the morning. Someone who never committed to it can be the best person to page. Authorship is a floor on operability, the drill chart is what tests the thing it stands in for, and reading one as the other is the same confusion as trusting an undrilled runbook.

Running the authorship count monthly catches one specific failure. Pairing that genuinely transferred work shows up as a rising floor. Pairing where I drove and narrated while someone watched leaves the count flat, no new author on the files, useful to learn in month four instead of month nine.

What changes when the system is regulated

Under GxP or HIPAA there is a second test as hard as whether the team can operate the system: whether they can revalidate and audit it without me, for years. An auditor arriving eighteen months after I left will ask them to prove a given output came from a validated system, and “the forward-deployed engineer set that up” doesn’t survive the question. This is where change management, one of the PRR’s six areas, becomes most of the work.

Before any of that, the pin has to be a pin. A version string in the manifest is a claim about a deployment somebody administers in a console, and the two can drift apart with the repository untouched. Azure’s model retirement policy is explicit that standard deployment types are upgraded automatically when a version retires, unless the deployment is set to versionUpgradeOption: NoAutoUpgrade, in which case inference returns 410 Gone. For a validated system the outage is the setting you want. Auto-upgrade keeps answering, the fingerprint keeps printing the version the manifest declares, and the scorecard keeps passing, while production runs on a model no validation record covers. A pin nobody configured downstream of the repository is a comment.

The judge is exposed the same way from the other side. It runs on a model the provider retires on its own schedule, and when it goes, every scorecard already filed becomes a number that cannot be re-derived. An auditor asking how a release was assessed needs the judge’s per-case verdicts, not the average they rolled up into. The verdicts survive the retirement of the model that produced them.

The mechanism that makes this survivable is a gate that enforces the rule unattended.

# platform/promote_gate.py -- runs before an agent is promoted to production
import yaml
from discover import AgentManifest
from scorecard import Scorecard
from validation import load_validation_record


def can_promote(agent: AgentManifest) -> tuple[bool, str]:
    # agent.scorecard was anchored to the agent's folder in discover.py, so this
    # reads the right file whatever directory the gate runs from
    card = Scorecard.model_validate(yaml.safe_load(agent.scorecard.read_text()))
    if not card.all_floors_met():
        return False, f"{agent.name}: a scorecard metric is below its floor"

    val = load_validation_record(agent.name)
    # the validated fingerprint is model, prompt and rubric together; a prompt or
    # rubric edit re-opens validation exactly as a model bump does
    if val is None or val.fingerprint != agent.fingerprint:
        return False, f"{agent.name}: no validation record for {agent.fingerprint}"

    if not val.human_signoff:
        return False, f"{agent.name}: awaiting human sign-off"

    return True, f"{agent.name}: promotable"

Two details in that gate are load-bearing for a departure. It reads the scorecard from a path anchored to the agent’s folder, so it returns the same verdict from CI, a client laptop, or the repo root. And it compares the whole fingerprint against the validated record, so a prompt edit or a rubric bump re-opens validation exactly as a model change does. An earlier version compared only the model version, which let a prompt rewrite promote against a validation record that no longer described it. Knowledge that used to live in my head, that a model bump means revalidation, now lives in code the client runs.

The tradeoff I have not resolved

The conflict at the center of this work has no clean answer. I’m hired to ship fast, the fastest way to ship is to do everything myself with tools I know, and every one of those choices is a small deposit into lock-in. Yet early speed is also how you earn the trust that makes a handoff possible at all.

So I spend the first third of an engagement mostly fast, then shift: pairing more than I’d like, writing runbooks for procedures I could run in my sleep, saying no to the vendor shortcut. The ownership curve is how I know whether the shift is real or whether I’m telling myself it is.

The closest thing I have to a test is going quiet on purpose for the last week. I resolve nothing myself unless something is genuinely on fire, I tell the client’s lead in advance so it doesn’t read as checking out, and then I mostly watch. If the system degrades while I’m quiet, it isn’t ready, and the honest move is to stay longer or narrow what I’m handing over. Every incident I resolve in that final week is a repetition their team doesn’t get, and the urge to be useful one more time is the wrong instinct to follow.

Even that is a stand-in. The honest measurement of a handoff is the first serious incident the team works with no path back to me, and by construction that lands after the engagement ends. While I’m billing, every incident has me in the channel, and the ones arriving in the last month are the classes easy enough to have been drilled already.

So the date goes into the contract at signature. One day, months out, a drill on a failure the client picks, with me in the room and silent. But it has to be bought before there is any appetite for it. Once an engagement is closed there is no budget line for a vendor turning up when nothing is wrong, and the request is a hard one to raise after the fact. The last useful thing I do for a system is come back to it once with no access left.