Nobody owns the join between two agent platforms

A discovery organisation collects agent platforms the way it collects data stores. The enterprise surface arrives attached to the productivity suite. The managed agent runtime arrives with the first-party agents the platform team wrote. The warehouse team’s assay data gets an agent because the warehouse shipped one, and the lakehouse team’s omics tables get another for the same reason. Then a question needs two of them at once, and the answer is to put them on a wire.

Read on its own, every one of those platforms is sound. Each has a governance model, a deployment story, an eval harness and a support contract, and each is documented by people who thought hard about it. What none of them has is an opinion about the other three.

Over a year on the mesh those four produced, almost every defect I ended up explaining to an architecture review board lived in a join: two contracts that meet, agree completely on the message, and disagree about everything the message implies.

This is seven of those joins, with the measurement that found each and the artifact that outlived me. The stack is Gemini Enterprise in front of ADK agents on Vertex AI Agent Engine, Snowflake Cortex Agents over a semantic view, a Databricks agent over Unity Catalog, and six MCP servers, three of them wrapping bioinformatics pipelines, all speaking A2A. Two adjacent problems are covered elsewhere and I will not re-cover them: getting a per-user token to the far side of a hop, and getting agent tools to work inside an egress-locked perimeter. This is what breaks once those are solved.

What each platform brought an opinion about

The reference architecture I drew first was wrong in an instructive way. It was a box diagram: four platforms, arrows between them, protocols on the arrows. Everyone approved it, because a box diagram cannot be disagreed with.

flowchart TB
  U[Scientist] --> GE[Enterprise agent surface]
  GE -->|A2A| ORCH[Orchestrator on the managed runtime]
  ORCH -->|A2A| SF[Warehouse data agent]
  ORCH -->|A2A| DBX[Lakehouse data agent]
  ORCH -->|A2A| MK[Maker agent] -->|A2A| CK[Checker agent]
  ORCH -->|MCP| BIO[Variant annotation, sequence search]
  SF --> SFP[(Row access and masking policies)]
  DBX --> UCP[(Row filters and column masks)]
  BIO --> NOP[(No catalog)]
  style SFP fill:#0f766e22,stroke:#0f766e,stroke-width:2px
  style UCP fill:#0f766e22,stroke:#0f766e,stroke-width:2px
  style NOP fill:#dc262622,stroke:#dc2626,stroke-width:2px

The version that turned out to be useful was a table of what each surface has an opinion about and where that opinion stops. It is boring to look at and it is the thing I would hand over first.

Two columns in that table are the whole article. Identity is covered in the identity post and I will lean on its conclusion rather than repeat it: per-user pass-through is what makes the far side’s own policy engine the authority, and it works.

The policy-objects column is what nobody had looked at. Sixty policy objects live in two engines that cannot read each other, and a third surface has none. Adding surfaces adds grants, so the mesh answers whatever any one of them will answer. It withholds only what every surface able to reach the fact withholds. One surface’s restriction binds the mesh only if the other two happen to carry the same restriction. The loosest surface decides, and the question of which surface that is on any given fact is the thing no engine computes and no team owns.

A review board is well equipped to approve a topology and structurally unable to approve that, because it is exactly what no single team brought to it. The question that eventually made ours useful is not “is this design sound”. It is “for each edge on this diagram, name the person who finds out first when it breaks”. Half the edges came back with no name. Two came back with a team name where I had asked for a person, which in practice was the same answer.

Three catalogs, and the loosest one decides

Start with the warehouse, because its governance is the strongest of the three and its failure is the most interesting. Assay results carry a programme code, and access to a programme is a grant in a mapping table, so the row access policy is a lookup.

create or replace row access policy governance.assay_by_programme
  as (programme varchar) returns boolean ->
    exists (
      select 1 from governance.programme_grants g
      where g.programme_code = programme
        and is_role_in_session(g.role_name)
    );

alter table discovery.assays.results
  add row access policy governance.assay_by_programme on (programme_code);

create or replace masking policy governance.mask_subject as (val varchar) returns varchar ->
  case when is_role_in_session('CLINICAL_UNBLINDED') then val else sha2(val, 256) end;

alter table discovery.assays.results
  modify column subject_id set masking policy governance.mask_subject;

IS_ROLE_IN_SESSION rather than CURRENT_ROLE is the detail that decides whether the policy works under an agent at all. CURRENT_ROLE returns the one role active in the session, and Cortex Agents resolve session permissions from the querying user’s default role, so a scientist whose grant sits on a secondary role is denied by a policy written to allow them. The role-hierarchy form is inclusive of every role in the session, which is what you want when the session was established by a platform instead of by a person typing USE ROLE.

The lakehouse expresses the same intent in a different dialect, from a different source of truth.

CREATE OR REPLACE FUNCTION governance.programme_filter(programme STRING)
RETURN EXISTS (
  SELECT 1 FROM governance.programme_grants g
  WHERE g.programme_code = programme
    AND IS_ACCOUNT_GROUP_MEMBER(g.group_name)
);

ALTER TABLE omics.variants.calls
  SET ROW FILTER governance.programme_filter ON (programme_code);

CREATE OR REPLACE FUNCTION governance.mask_subject(v STRING)
RETURN CASE WHEN IS_ACCOUNT_GROUP_MEMBER('clinical-unblinded') THEN v ELSE sha2(v, 256) END;

ALTER TABLE omics.variants.calls
  ALTER COLUMN subject_id SET MASK governance.mask_subject;

Those two blocks look like translations of each other and are not. The warehouse resolves a Snowflake role hierarchy. The lakehouse resolves account group membership, and Unity Catalog notes in passing, inside an admonition on a page about mapping tables, that all filters run with definer’s rights except the functions that check user context, SESSION_USER and IS_ACCOUNT_GROUP_MEMBER, which run as the invoker. That is the whole security model of the block above, and it is one sentence in a note.

governance.programme_grants exists twice, once per platform, loaded by two pipelines from one upstream entitlement system on two schedules. The day one of those pipelines fell behind, the two catalogs disagreed about a programme for eleven hours. Nothing anywhere reported a problem, because each catalog was correctly enforcing the state it had.

There is a worse thing in those two blocks, and it survived two review boards before anyone saw it. Both masks are sha2 at 256 bits over the same subject identifier with no salt on either side, so a subject present in both catalogs gets the identical digest in the warehouse and in the lakehouse. A role entitled to neither clear column can therefore join masked assay rows to masked variant rows on the mask value and assemble a per-subject record spanning both platforms.

Each mask is defensible alone. A deterministic mask is what you want when a masked column still has to join to itself, and every published example of both features looks exactly like this. But the mesh turns it into a leak, because the mask has become a stable cross-platform subject key. The repair is a per-platform salt, or one tokenisation service issuing a different token per subject per platform. Either way it needs the two catalogs to agree on something. Which is the same agreement nobody owns, arriving inside the two blocks I had been using as the illustration of the problem.

I left before that one landed, and I do not know how it ended. The salt policy was written, costed and agreed in a room. But re-masking a live column means a migration on two platforms whose change windows are set by two different teams, and the last version I saw had the warehouse going first with the lakehouse following in a quarter nobody had committed to. A half-applied salt is worse than none, because it looks like the fix is in. The one artifact in the list at the end of this post that I cannot say survived me is that one.

Then there are the retrieval tools attached to each side, where the two vendors made opposite calls on the same hazard. Databricks refuses: you cannot create an AI Search index from a table that has row filters or column masks applied. Snowflake permits, and documents the consequence plainly. A Cortex Search service runs with owner’s rights, so any role able to query the service may query any data the service has indexed regardless of its privileges on the underlying objects, and for a service over a table carrying row-level masking policies, querying users see results from rows the owner’s role can read even where their own role cannot. Neither vendor is wrong. One removed the foot-gun and one documented it, but a federation gets both answers at once because it holds both products.

The sharper version of that split is inside one product, and finding it changed how I read a limitations section. Unity Catalog now has a second, attribute-based way to write the same restrictions, and the two ways resolve the search hazard oppositely. The table-level form fails loudly: the index will not build. The attribute-based form does not fail at all.

Databricks documents that ABAC policies on a source table do not apply to AI Search indexes created from that table, that the index syncs every row and enforces neither row filter nor column mask when serving queries, and that keeping a masked column out of the index is a setting someone has to remember. Building the index is not free, since the principal doing it has to be exempt from the policy to read the rows at all, and that is a real control. But it is a control at build time over an index that enforces nothing at query time. The two identity models differ underneath as well, the older filters running as definer and ABAC evaluating as the session user. So one platform, one restriction, two syntaxes, and whether a masked column reaches a retrieval index depends on which syntax the person writing the policy happened to prefer.

The third surface has no policy objects at all. The bioinformatics pipelines run under one service principal with read access to everything they annotate, which is the correct design for a pipeline but the wrong one for a tool an agent can call.

Nothing in MCP repairs this. Its schema defines four behavioural annotations on a tool, readOnlyHint, destructiveHint, idempotentHint and openWorldHint, and says in the same place that every property in that structure is a hint with no guarantee of describing the tool faithfully. The tools page then puts the obligation on the caller, and a client must treat them as untrusted unless the server is trusted. There is no field for whose data this is.

So I stopped reasoning about it and measured. Sixty questions, written to be answerable from more than one surface where possible, asked as three personas with deliberately different entitlements: a bench chemist, a clinical pharmacologist, and an external CRO collaborator. Thirty-eight of the sixty had a data path on at least two surfaces, giving 114 comparable trials per run. The probe ran three times over six weeks.

Fourteen divergences in the middle run, twelve and fifteen in the others. Six were denied in the warehouse and answered by the lakehouse agent. Three had a column mask in the lakehouse and came back in clear from the warehouse. Five were governed on both the warehouse and the lakehouse and answered from a pipeline with no catalog. Another thirty-one trials were denied everywhere, so forty-five of the 114 met a policy on at least one surface and fourteen of those forty-five got their answer anyway.

The gradient across personas is what I would put in front of a review board. The persona with the fewest entitlements produced twice the divergences of the persona with the most, because divergence needs a policy to disagree about and the widely-granted questions have none.

That probe is a scheduled job now and it is the most valuable thing I left behind. It holds one question, one persona, one expected disposition per surface, and it fails when two surfaces disagree. The expected dispositions are maintained by the governance team and not by me, which is the part that made it survive my leaving.

A denial is not a fault, and the orchestrator could not tell

The probe measures what the catalogs would do if asked once. Production asks again, which changes the number.

A2A is precise about how a task ends. The specification defines four terminal states, TASK_STATE_COMPLETED, TASK_STATE_FAILED, TASK_STATE_CANCELED and TASK_STATE_REJECTED, and two it calls interrupted rather than terminal, TASK_STATE_INPUT_REQUIRED and TASK_STATE_AUTH_REQUIRED. Two more say the work is still running, TASK_STATE_SUBMITTED and TASK_STATE_WORKING, and TASK_STATE_UNSPECIFIED is the zero value that an unset state field deserialises to. Nine in total, and the distinctions between them are careful. What the protocol has no opinion about, because it cannot, is which of those a policy denial is.

In our mesh a denial arrived as TASK_STATE_FAILED from one platform and as a completed task whose artifact was a polite sentence about insufficient privileges from another. The orchestrator had a retry policy, which every orchestrator has, and it did what a retry policy does. It read a failed delegation as a transient fault and re-planned onto the other data surface.

Over twenty weekdays, 34,380 of 37,120 delegations completed, 1,940 ended failed, 610 were cancelled and 190 rejected. Six hundred and four of the failures carried a policy denial. The orchestrator re-planned 412 of those and got a full answer 233 times, about twelve a weekday.

A retry policy written by the agent team converts a decision made by the governance team into a load balancing decision, but nobody chose that. The probe had put that rate at 14 of the 45 trials denied on at least one surface, 31%. Production, where the retry runs for real, put it at 233 of 604, 39%. Same phenomenon, two denominators, and the production one is the honest one.

There is a quieter version of the same problem on the lakehouse side. Databricks Runtime versions below 12.2 LTS do not support row filters or column masks and fail securely, which the documentation defines precisely: no data is returned. An empty result is not a denial to anything downstream, so an agent pointed at a stale compute configuration is an agent whose governed tables have become empty ones.

The repair has to live in the client, because the protocol cannot carry it. Classify the terminal state, and treat a denial as terminal for the question rather than for the task.

import httpx

A2A_VERSION = "1.0"
TERMINAL = {"TASK_STATE_COMPLETED", "TASK_STATE_FAILED",
            "TASK_STATE_CANCELED", "TASK_STATE_REJECTED"}
INTERRUPTED = {"TASK_STATE_INPUT_REQUIRED", "TASK_STATE_AUTH_REQUIRED"}
IN_PROGRESS = {"TASK_STATE_SUBMITTED", "TASK_STATE_WORKING"}
UNSPECIFIED = "TASK_STATE_UNSPECIFIED"

# The whole state space, asserted rather than assumed. `classify` handles every
# member, and the test below drives one case per member. A tenth state means
# the spec moved and this file has to move with it.
ALL_STATES = TERMINAL | INTERRUPTED | IN_PROGRESS | {UNSPECIFIED}
assert len(ALL_STATES) == 9

# Keys we have seen a platform put its own denial code under. A2A names none of
# these; every one is a bilateral agreement with one vendor.
CODE_KEYS = ("errorCode", "sqlState", "reason", "code")


class Denied(Exception):
    """A governance decision. Never retried, never re-planned elsewhere."""


class NeedsHuman(Exception):
    """The far side asked something the orchestrator is not allowed to answer."""


class Abandoned(Exception):
    """Terminal and empty. The work stopped, so no answer exists to return."""


async def send(client: httpx.AsyncClient, url: str, request: dict) -> dict:
    # The version goes in a header, not the body. An agent receiving an empty
    # A2A-Version must assume 0.3, so omitting it does not mean unversioned, it
    # silently selects the protocol before the rename. And `messageId` is a
    # field of `request["message"]`, not of the request: the JSON-RPC `id` is a
    # transport correlator that happens to be convenient to key off it.
    resp = await client.post(
        url,
        headers={"A2A-Version": A2A_VERSION},
        json={"jsonrpc": "2.0", "id": request["message"]["messageId"],
              "method": "SendMessage", "params": request},
        timeout=120.0,
    )
    resp.raise_for_status()
    body = resp.json()
    if "error" in body:
        raise RuntimeError(f"{url}: {body['error'].get('message')}")
    return body["result"]


def platform_code(task: dict) -> str:
    """Dig a platform error code out of the slots A2A actually has.

    `TaskStatus.message` is a `Message`: role, parts, messageId, contextId,
    taskId, metadata, extensions, referenceTaskIds. There is no code on it and
    none on `Task`. A code therefore arrives in a `metadata` map or inside a
    data part, on the status message or on an artifact, and which of those it
    is depends on the platform.
    """
    carriers = [(task.get("status") or {}).get("message") or {}]
    carriers += task.get("artifacts") or []
    scopes = []
    for carrier in carriers:
        scopes.append(carrier.get("metadata") or {})
        for part in carrier.get("parts") or []:
            scopes.append(part.get("metadata") or {})
            if isinstance(part.get("data"), dict):
                scopes.append(part["data"])
    for scope in scopes:
        for key in CODE_KEYS:
            value = scope.get(key)
            if isinstance(value, (str, int)) and str(value):
                return str(value)
    return ""


def classify(task: dict, denial_codes: set[str]) -> None:
    """One disposition per state, with no state left to fall through."""
    state = task["status"]["state"]
    if state in INTERRUPTED:
        raise NeedsHuman(state)
    if state == "TASK_STATE_REJECTED":
        raise Denied(state)
    if state == "TASK_STATE_CANCELED":
        # Terminal, and carrying neither a result nor a decision. Returning
        # normally here reports an answer that was never produced.
        raise Abandoned(platform_code(task) or state)
    if state == "TASK_STATE_FAILED":
        # A denial and a timeout are the same state here, and only the far
        # side's own code separates them.
        code = platform_code(task)
        raise (Denied if code in denial_codes else RuntimeError)(code or state)
    if state == "TASK_STATE_COMPLETED":
        # One of our platforms returns a denial as a completed task carrying a
        # polite artifact. The state does not say so; the metadata does.
        code = platform_code(task)
        if code in denial_codes:
            raise Denied(code)
        return
    if state in IN_PROGRESS:
        raise RuntimeError(f"send returned a task still running: {state}")
    # Everything else is the zero value or a string this build does not know.
    raise RuntimeError(f"unhandled task state: {state or UNSPECIFIED}")

The TASK_STATE_CANCELED branch was missing for a quarter. The first version handled the interrupted states, rejection, failure and completion, then checked that whatever was left was terminal. The set of terminal states was right, but cancelled is in it. So a cancelled task passed the check, the function returned normally, and the caller read a normal return as an answer. Six hundred and ten delegations in the window ended there, every one arriving back at the orchestrator as a successful empty result.

Cancelled means the work stopped before it produced anything: our own deadline, or the far side’s admission control, or an operator draining a runtime. The honest disposition is that no answer exists and the question goes back to the person who asked it. That is a third outcome next to a denial and a fault, and it needs an exception of its own or it borrows the meaning of whichever branch it happens to land in. The test now drives one case per state, because the defect here was a branch that was missing, and a missing branch is invisible to every test written against the branches that exist.

import pytest

from mesh.a2a import ALL_STATES, Abandoned, Denied, NeedsHuman, classify

DENIAL_CODES = {"002003", "PERMISSION_DENIED"}

# state -> (platform code on the task, what classify must do with it)
CASES = {
    "TASK_STATE_COMPLETED": (None, None),
    "TASK_STATE_FAILED": ("57014", RuntimeError),
    "TASK_STATE_CANCELED": (None, Abandoned),
    "TASK_STATE_REJECTED": (None, Denied),
    "TASK_STATE_INPUT_REQUIRED": (None, NeedsHuman),
    "TASK_STATE_AUTH_REQUIRED": (None, NeedsHuman),
    "TASK_STATE_SUBMITTED": (None, RuntimeError),
    "TASK_STATE_WORKING": (None, RuntimeError),
    "TASK_STATE_UNSPECIFIED": (None, RuntimeError),
}


def task(state: str, code: str | None = None) -> dict:
    status: dict = {"state": state}
    if code:
        status["message"] = {"metadata": {"errorCode": code}}
    return {"status": status}


def test_the_cases_cover_the_protocol():
    assert set(CASES) == ALL_STATES


@pytest.mark.parametrize("state", sorted(CASES))
def test_one_disposition_per_state(state):
    code, expected = CASES[state]
    if expected is None:
        assert classify(task(state, code), DENIAL_CODES) is None
    else:
        with pytest.raises(expected):
            classify(task(state, code), DENIAL_CODES)


@pytest.mark.parametrize("state", ["TASK_STATE_COMPLETED", "TASK_STATE_FAILED"])
def test_a_denial_code_outranks_the_state(state):
    with pytest.raises(Denied):
        classify(task(state, "002003"), DENIAL_CODES)

Both halves of platform_code are the article in miniature. The warehouse returns 002003 for an object the caller may not see, which is the same code it returns when the object is absent, because it refuses to confirm existence to someone without privileges. The lakehouse returns a permission reason in a data part. Both are reasonable, although neither is standard.

A2A does however have a good structured slot for errors, an array on the error object whose entries carry a @type and should use well-known types from the google.rpc error model where they apply, ErrorInfo and BadRequest among them. The fields that would carry the meaning, reason and domain, are ErrorInfo’s own. The a2a-protocol.org domain you will see in that section sits inside its worked example, so it is an illustration of the shape and tells you nothing about what any vendor will put there.

A policy denial travels a different road anyway, because the call succeeded and it was the task that failed. On the task side a code fits in exactly one place, metadata, which the spec defines as a key-value map holding any JSON value and then deliberately leaves the keys open. So CODE_KEYS is a record of bilateral agreements, one per platform, in a field every vendor is free to fill its own way. Every federation I have seen accumulates a table like it inside an exception handler. Ours is a file next to the client, one row per platform, with a test per row. The tests are fixtures captured from real denials, because the shape is only learnable from one.

The clarifying question has no route home

TASK_STATE_INPUT_REQUIRED cost the most user trust, and it cost it silently.

A2A models human-in-the-loop as a task state. A remote agent needing a clarification puts the task into an interrupted state and waits for the client to send another message on the same task, which is a sound design for a client with a human attached. In a federation the human is attached to exactly one platform’s front end, three hops away. The client is an orchestrator inside a managed runtime with no display and no inbox. The protocol is asking a question of a component that has no mouth.

The spec is candid about why nothing here is fixable at the protocol layer. Identity information is handled at the protocol layer and not within A2A semantics, and its enterprise guidance says payloads do not carry user or client identity directly. The stated design principle is opacity, in the spec’s own words that agents “collaborate based on declared capabilities and exchanged information, without needing to share their internal thoughts, plans, or tool implementations”. Opacity is what makes federation possible, but it is also why the far side cannot know that the entity it is talking to isn’t the entity that can answer.

Of 37,120 delegations over twenty weekdays, 4,412 entered an interrupted state at least once. The orchestrator filled in the missing parameter 3,180 times and gave up 1,232 times. There was no third branch.

We adjudicated 200 of the 3,180 with the scientist who had asked the original question. Forty-one had the parameter wrong, 20.5% with a 95% interval of plus or minus 5.6 points, extrapolating to 652 tasks over the window answered on an assumption nobody made deliberately. Twenty-nine of the forty-one returned an answer that looked complete: a number, a cohort size, a sentence.

The most common wrong fill was a date range. A remote agent asked which study window to use, the orchestrator supplied the trailing twelve months because that is what the question sounded like, and the scientist had meant the window of a specific protocol amendment.

Two changes followed. Interrupted states stopped being auto-answered and started returning to the enterprise surface as a rendered question carrying the task id, which is a product change and not a protocol one. And the agents we owned stopped asking: one that cannot resolve a parameter now fails with the question in its message instead of suspending, because a failure appears in a scorecard and an interrupted task does not. That second change converted a recoverable state into an unrecoverable one to make it observable. Right for an agent whose questions come from other agents, and vandalism for an agent a human talks to.

Two agents, one prior

The verification design was a maker-checker pair, the control a regulated organisation reaches for first because it already runs it on documents. A maker drafts, a checker reviews, and the pair emits only what the checker passes. Agreement between them was 94% in the first month and everyone was pleased.

High agreement is what a duplicate control looks like. Both agents ran the same base model with different instructions, and both read the same retrieved context, so their errors were independent in neither the weights nor the evidence.

There is literature on the first half of that, and it is easier to over-read than to read. Models do recognise and prefer their own generations, and that paper reports a linear relationship between how well a model recognises its own output and how strongly it favours it. But the relationship was obtained by fine-tuning models to move their self-recognition rather than by measuring the two as they arrive, so it is evidence for a mechanism and not a number to carry into your own system. I have argued that at length about importing a self-preference coefficient into an eval, and it applies to me here. The second result does transfer: models cannot reliably self-correct reasoning without external feedback, and sometimes degrade after the attempt. Neither one tells you what a second vendor’s weights are worth against your defects.

So we seeded defects and counted: 240 outputs from four weeks of production traffic, 96 seeded with a known defect across four classes at 24 each, 144 left clean, one adjudication pass per configuration over the same items in the same order.

Read the second row as the paired comparison it is, which took me two goes. The same 240 items went through all three configurations in the same order, so the question is not whether two independent proportions differ. It is whether the same items flipped, and the test for that is McNemar’s, which looks only at the items the two checkers disagreed about. If they disagreed on thirteen, the two-sided p is 0.001. If they disagreed on thirty-nine, more than a third of the seeded set, it is 0.055. Every split in between sits under 0.06.

The thirteen points are real, and I had them filed as noise for a quarter by testing them twice wrongly. First against a 95% interval around a single proportion, which adjudicates no difference at all. Then against the unpaired difference interval, which does adjudicate a difference and is plus or minus 14 points. Both answers were computed correctly on a comparison nobody ran, because every item went through both checkers.

Being real isn’t the same as being defence in depth, and this is where the row stops helping. The gain arrived with the false-flag rate more than doubled: thirteen more catches and eleven more false flags on the same 240 items, so the reviewers went on reading roughly the same volume of flagged output with a larger share of it clean.

Per class it is lumpier still. On unit and scale errors the different-vendor checker caught seven where the same-model checker caught nine, and the paired test governs that fall exactly as it governs the gain. Both configurations saw the same 24 seeded items, so the evidence sits in the discordant pairs, and a net of two cannot reach significance under any split of them. The most favourable split has the two checkers disagreeing on exactly two items with both going the same way, which McNemar’s exact test puts at p = 0.50, and every wider disagreement is worse than that. The fall is noise, and it is the same test that says so.

This is where this post and my write-up on judge panels have to agree out loud, because at a glance they do not. There, a mixed-family panel bought about 0.05 of kappa over a single swapped strong judge, and most of what it bought was self-preference cancelling across families. Here, a second vendor’s weights buy thirteen points at twice the false flags. Both hold, because they are measuring different failures. A panel judges a preference, where the bias is that a judge likes its own phrasing and a second family cancels that directly. A checker judges a claim, where the failure is that it read the maker’s numbers. Decorrelating the weights does nothing about the numbers.

The third row is not in doubt. The independent evidence path means the checker never receives the maker’s context. It re-queries the warehouse, re-runs the annotation and re-resolves every identifier under the same user’s grants. Thirty-seven points, and the false-flag rate came back below the different-vendor configuration, which surprised me until it did not: most of the second configuration’s false flags were the checker objecting to a claim the shared context underdetermined, and a checker with its own evidence resolves those.

Varying the weights decorrelates the reasoning and leaves the evidence shared. Varying the evidence decorrelates the input to the reasoning, which is where most of our defects were born: a stale snapshot, a cohort filter applied at the wrong level, a unit column read as micromolar when the source was nanomolar. Two models reading one wrong number agree about it.

It costs roughly double in tokens but far less than double in latency, because the checker issues its own query alongside the maker’s instead of after it, so what it adds is its own reasoning and not a second round trip: the p50 of a verified answer went from 11.5 to 14.2 seconds. We run the independent path on outputs that carry a number or feed a decision, and the shared-context checker everywhere else.

A missing confidence signal is not a low one

Underneath the checker sat a confidence gate: take the mean token log-probability of an answer, threshold it, and route anything below to a deterministic fallback or a human. Whether that is a sound instrument in the first place is a question I have already worked through in public on a different pipeline, with the adjudicated set to go with it, and the answer was that it measures how likely those tokens were and not how likely the claim is. On this mesh it separated 480 adjudicated answers at AUC 0.61, a little above the interval that pipeline reported but still close enough to chance that neither number supports a gate. Arguing about which weak separator is weaker is not worth the space. The part that is specific to a federation is different, and it is worse.

Log-probabilities are available unevenly across a mesh, but the unevenness does not follow vendor lines.

The right column is the seam. A managed platform that does not support the parameter does not reject it; it accepts the field, ignores it, and returns a response with the log-probability structure absent or empty. A gate written against one vendor’s schema does not fail when pointed at another. It reads a missing value, and whatever your code does with a missing value becomes your policy for that hop.

Ours passed. That was the default branch of an if and nobody chose it.

Coverage was worse than the table suggests. A token distribution has no standard place in A2A, which is a different statement from having nowhere to go. Message, Artifact and the send request each carry a metadata map that the spec defines as holding any JSON value, and an extension can strongly type what goes into it, so a distribution can physically cross the wire. What it cannot do is arrive unarranged. No field in the protocol names it, so a gate reads one only from an agent that was told to send it, but no vendor’s managed agent has been told. Fifty-eight per cent of our questions reach a federated data agent, and on all of them the gate saw nothing from the hop that produced the numbers, then passed.

from dataclasses import dataclass

from mesh.a2a import Denied  # the governance exception from the client above


@dataclass(frozen=True)
class Verdict:
    disposition: str          # pass | blocked | escalate
    reason: str


def reverify(claim) -> bool:
    """Re-derive one machine-checkable assertion from its source and compare.

    A claim carries the query, accession or pipeline call that produced it, so
    the check runs under the asking user's grants and returns a fact, not a
    probability. Those grants are why it raises `Denied`: the re-check reaches
    the same catalogs the answer did, and can be refused by them.
    """
    return claim.recompute() == claim.value


def gate(claims, logprob_mean: float | None, threshold: float) -> Verdict:
    """Absence of a confidence signal is not a pass. It is no verdict."""
    if claims:
        # A checkable claim never consults the model's confidence.
        failed, unchecked = [], []
        for c in claims:
            try:
                if not reverify(c):
                    failed.append(c)
            except Denied:
                # The first seam, arriving one layer down. A claim we are not
                # allowed to re-derive is not a claim that survived the check.
                unchecked.append(c)
        if failed:
            return Verdict("blocked", f"{len(failed)} of {len(claims)} claims failed re-check")
        if unchecked:
            return Verdict("escalate",
                           f"{len(unchecked)} of {len(claims)} claims not re-checkable: policy denial")
        return Verdict("pass", f"{len(claims)} claims re-derived")
    if logprob_mean is None:
        # Federated hop, or a model that ignores the parameter. Either way the
        # gate has no reading, and a missing reading is not a low-risk reading.
        return Verdict("escalate", "no confidence signal on this hop")
    if logprob_mean < threshold:
        return Verdict("escalate", f"mean logprob {logprob_mean:.3f} < {threshold}")
    return Verdict("pass", "no checkable claim, confidence above threshold")

The first rule is that a checkable claim is never gated on confidence, because confidence is a worse instrument than the check. Two hundred and ninety-three of the 480 answers carried at least one machine-checkable assertion, and 118 of the 172 wrong answers fell in that subset, so the error rate among checkable answers was 40% against 29% among the rest. The answers with hard numbers are the wronger ones. On that subset the deterministic re-check caught 111 of 118, and every one of the seven misses was a case where the re-check was itself refused by a policy, which is the first section’s problem arriving in the middle of this one. That is what the third arm of gate is for. A claim nobody is allowed to re-derive has not passed a check, and reading the denial as a pass would put the same failure one layer further down than the place we had just finished fixing it.

The second rule is that a missing signal escalates. That was expensive for about a week, but it is what made the gate a control instead of a decoration. It is also why wrapping the bioinformatics pipelines in MCP servers pays for itself. A variant annotation is a function, so re-running it and comparing is cheap and exact, and the comparison lands on the annotation itself and not on anything the model said about it.

Fourteen release trains, and no system to test

Everything above assumes there is a thing called the system that you can evaluate. There was not, and a failed regression run is what showed me why.

Agent deployment was continuous and zero-registration by design, which I still think is right. No console click anywhere: the agent card is generated from the repository, published at /.well-known/agent-card.json, and the deployment is a bundle. On the lakehouse side that is Declarative Automation Bundles, the feature you will find under Databricks Asset Bundles in anything written before the rename, which touches neither the CLI command nor a line of existing configuration. The target carries the pin.

bundle:
  name: discovery-mesh

variables:
  agent_version:
    description: Unity Catalog model version promoted by this train

targets:
  prod:
    mode: production
    # No run_as here. Databricks rejects a bundle that sets run_as alongside
    # model_serving_endpoints, so the deployment identity for this resource is
    # whoever holds the CI credential, and it is governed outside the bundle.
    resources:
      model_serving_endpoints:
        omics_agent_endpoint:
          name: omics-agent
          config:
            served_entities:
              - entity_name: omics.agents.lakehouse_agent
                entity_version: ${var.agent_version}
                scale_to_zero_enabled: false

mode: production does more than it looks like. It validates that every related pipeline is marked development: false, checks the current git branch against the target’s, and where a service principal is not used it requires run_as and permissions and refuses paths overridden to a specific user. The mode you are not in is the interesting one: development mode disables the deployment lock, so two engineers deploying one bundle at once is a supported operation there and a race in a shared workspace.

The trouble was never a single train. It is that there are fourteen. Over one quarter the deployment log shows 65 releases across 14 agents, from 13 for the busiest down to a single release each for the three quietest, which is 66 distinct mesh configurations in 91 days and a mean configuration lifetime of 1.4 days. A full cross-agent regression run, including adjudication of everything it flagged, took a median of 2.4 days.

An aside I have never found a use for. Nobody on that programme could agree on what to call the thing. Mesh was mine and it stuck, but the platform team said fabric, the data team said estate, the architects wrote federation in every document, and one of the vendors kept saying network of agents in workshops. For about two months the four words were used interchangeably in the same meetings by people who each thought the other three were describing something else, and the version-negotiation bug in the next paragraph was open for most of it. I would love to claim the vocabulary caused the bug. It did not, we found the bug by diffing two JSON documents, and I still think about it.

Modelling changes as Poisson at 65 per 91 days, the probability that the configuration under test is still in production when the verdict lands is 18%. Eighty-two per cent of our evaluation results described a mesh that no longer existed. Nothing was failing. Every train was green and every scorecard was current, but the composite the review board thought it had approved had never been tested as a composite.

The protocol has a sharper version of the same problem. A2A carries its version in an A2A-Version header, but an agent receiving no header must assume 0.3, which is the version before the rename that moved every method name and every task-state string. The specification tells SDK authors that a client needing current features should request a specific version and avoid automatic fallback, to prevent silently losing functionality. A federation without explicit version negotiation doesn’t break. It downgrades and keeps answering.

The same gap sits inside the SDK. ADK’s remote-agent client still defaults to its legacy integration, and which shape of card it emits depends on which major version of the A2A SDK the build resolved, so two agents from one repository can advertise differently because their images were built a fortnight apart.

Publishing a card is also not the same as being registered against it, and that is where zero-registration leaks. One registry imports a card from its URL and stores what it read. The enterprise surface takes the card inline, as a JSON-escaped string inside the registration call, under a size limit. Either way the registered card is a copy taken at a moment, yet the agent goes on serving a card that moves underneath it. We shipped a capability that the surface in front of it had never heard of, and it took three weeks to diagnose, because nobody had thought to compare two documents that were never supposed to differ.

So the mesh got a manifest: one document, every agent, pinned version, agent-card digest, negotiated protocol version, and the digest of the eval run that cleared it. Promotion is one policy evaluation over that document, and the rules that matter are the completeness rules rather than the quality thresholds.

package mesh.promote

import rego.v1

default allow := false

allow if {
	complete
	count(deny) == 0
}

# Every key the deny rules read has to be required here: the seven agent fields
# and the three top-level inputs. Presence is checked with object.keys rather
# than truthiness, because two of the fields are booleans and a bare reference
# to a false value fails a rule body the same way a missing key does. This is
# the version that took a third pass: see below.
required_agent_keys := {
	"name", "version", "card_digest", "a2a_version",
	"eval_digest", "reads_governed_data", "policy_probe_passed",
}

complete if {
	is_number(input.expected_agent_count)
	is_string(input.evidence.mesh_eval_digest)
	is_string(input.manifest.negotiated_a2a_version)

	count(input.manifest.agents) == input.expected_agent_count
	every a in input.manifest.agents {
		required_agent_keys == required_agent_keys & object.keys(a)
		a.name != ""
		a.version != ""
		a.card_digest != ""
		a.a2a_version != ""
		a.eval_digest != ""
	}
}

deny contains "manifest incomplete: an agent is missing a name, version, digest or eval" if {
	not complete
}

deny contains msg if {
	some a in input.manifest.agents
	a.a2a_version != input.manifest.negotiated_a2a_version
	msg := sprintf("agent %v speaks A2A %v, mesh negotiated %v",
		[label(a), a.a2a_version, input.manifest.negotiated_a2a_version])
}

deny contains msg if {
	some a in input.manifest.agents
	a.eval_digest != input.evidence.mesh_eval_digest
	msg := sprintf("agent %v was cleared against a different mesh configuration", [label(a)])
}

deny contains msg if {
	some a in input.manifest.agents
	a.reads_governed_data
	not a.policy_probe_passed
	msg := sprintf("agent %v has no passing cross-catalog policy probe", [label(a)])
}

label(a) := object.get(a, "name", "<unnamed>")

complete is doing something narrower than the usual fail-closed argument. The failure worth catching here is not a malformed request. It is a manifest that is honestly missing an agent because a train did not run, which reads as a clean mesh unless the expected agent count arrives as a separate input. That count comes from the registry, so the document cannot vouch for its own completeness.

The first version of that policy had the same bug one level down, and I only found it because a reviewer ran it against a record I wouldn’t have thought to build. complete required the version, the digests and the protocol version. It did not require name, because name is only used to write the message. But every deny rule interpolates it, and in Rego a sprintf over an undefined key makes the whole rule body undefined, so the rule emits nothing.

Feed it an agent that is on the wrong protocol version, cleared against a stale evaluation, and has no passing probe, then delete its name, and the policy returns allow: true with an empty deny set. Restore the name and the same record produces three denials. A gate that fails open on the sloppiest record in the manifest is worse than no gate, because it also produces a green check.

I fixed that by requiring name, and a second reviewer showed me I had fixed the instance and left the class. The same undefined-key mechanism opens three more doors. Drop reads_governed_data from an agent and the probe rule cannot fire. Drop input.evidence and the stale-evaluation rule cannot fire. Drop negotiated_a2a_version and the protocol rule cannot fire. That last one is the downgrade this section is about, and the policy waved it through with an empty deny set.

Two of the seven fields are booleans, so presence has to be tested with object.keys and not by reading them, because a bare reference to false fails a rule body exactly the way a missing key does. The general form is the only one worth writing down: a deny rule that reads a key it did not require is not a control, it is a control-shaped comment, and the way to keep that true over time is to derive the required set from the rules rather than maintain it beside them. The test feeds the policy one malformed manifest per omitted key and asserts allow == false on each.

The last rule made the federation legible to the review board. An agent that reads governed data cannot be promoted without a passing cross-catalog probe, which turns the probe from the second section into a release blocker with a named owner instead of a monitoring job someone can silence.

Weekly pinned trains cut 66 configurations a quarter to 13 scheduled plus 6 hotfixes. The scheduled change falls outside the 2.4-day evaluation window by construction, so only a hotfix can supersede a verdict, and the superseded rate fell from 82% to 15%. The mesh still absorbs 65 agent releases a quarter. They now land at a scheduled moment the evaluation window is built around.

Terraform holds what is not an agent, and this is where the multi-vendor reality is least romantic:

terraform {
  required_providers {
    google     = { source = "hashicorp/google" }
    snowflake  = { source = "snowflakedb/snowflake" }
    databricks = { source = "databricks/databricks" }
  }
}

Three providers, three release cadences, three notions of what a preview feature is. Registration is where it thins out. google_agent_registry_service will put an agent into Google’s agent registry. But nothing will put one into the enterprise surface, which stays an API call in a pipeline step, and no provider from any vendor spans all four platforms.

The inverse is worse. An agent that fails its gate stops being deployed and stays discoverable until something withdraws its card, and withdrawal is a different operation on each platform. Deregistration is not the inverse of registration anywhere in this stack, and the day I learned that, a retired agent had been answering questions for nine days.

Green scorecards, measured on questions nobody asks

The last join hid all the others.

Every agent had a scorecard, a condition of the review board’s approval and a good condition. Each team maintained a 120-question eval set for its own agent, ran it in CI and published accuracy. The board saw a wall of green and so did I.

The distribution is wrong. An agent in a mesh does not receive user questions. It receives sub-questions an orchestrator rewrote: shorter, more literal, stripped of the context the orchestrator already resolved, phrased in the orchestrator’s idiom instead of a scientist’s. So we re-scored each agent on 120 orchestrator-generated sub-questions covering the same capability, drawn from production traces.

Two sets of 120 are two different question sets, so the interval that matters is the one on the difference, and it is wider than the interval on either column. The half-widths run 9.4, 10.0, 7.5 and 10.5 points down the table. That puts the two 17-point drops well clear, the 8-point drop marginally clear and worth one more run rather than a decision, and the 2-point drop nowhere near. The ordering is the finding anyway. The agent with the worst standalone score barely shifted, because that team had built its eval set from production traffic instead of writing questions, but the agent with the best standalone score was furthest from what it actually receives.

The lakehouse agent was easiest to repair because its harness already takes a dataset as an argument. Databricks agent evaluation runs through mlflow.genai.evaluate() with an explicit list of scorers, so changing the distribution changes the data argument and leaves the judges alone, which is the right separation.

One migration detail is worth carrying. MLflow 3 requires you to name every scorer you want, where MLflow 2 assembled a default metric set from the model_type you passed. An upgrade that narrows what you measure without announcing it is a thing you catch by reading the diff or not at all.

What took the argument was the authorship rule that followed. An agent’s eval set may no longer be written by the team that owns the agent. It is sampled from the traffic the agent receives, and the owning team may add to it but may not remove from it. Removals go through the same review as a policy change.

What survives you

The specific stack will not generalise. The protocol versions moved twice while I was on it, one vendor’s tool-type names were renamed under us, and the paragraph about a search service running with owner’s rights may describe a fixed bug by the time anyone reads this.

But the shape generalises. Every failure here is a place where two correct components meet and neither one’s documentation is wrong. Three catalogs enforcing exactly what they were told. A retry policy doing what retry policies do. Two agents agreeing because they were built to agree. A confidence number that is a real number about a real distribution, read as though it were about something else. Fourteen deployment pipelines, each one exemplary.

So the artifacts I would fight to leave behind are the ones that turn a join into an object with an owner. The surface table from the first section, saying who enforces what and where the enforcement stops. The cross-catalog probe, wired into the promotion gate so no one can switch it off alone. The per-platform denial-code table, with a test. The one-page salt policy that stops two catalogs masking the same identifier to the same digest, which is the one I could not get over the line. The mesh manifest schema and the policy that refuses an incomplete one. And the rule that eval sets are sampled from traffic rather than written from imagination.

All six are cheap, and none of them required a vendor to change anything. Each one takes something that was previously nobody’s and gives it a name, a file and a person.