Faithful to the wrong version

Faithfulness metrics ask whether an answer follows from the passage the system retrieved, which on a static corpus is close enough to the question you care about. On a corpus where the same rule has three revisions on file it comes apart, because the retrieved passage can be a revision that stopped being in force months ago, so the answer is faithful, correctly cited, internally consistent, and wrong, and every check in a standard pipeline passes it.

The reason those old revisions are on file at all is worth stating first, because the obvious fix is to delete them and that is the wrong fix. A regulated corpus is bitemporal: every revision has an effective date, the day it took force, and the index has a transaction time, the day it learned about the revision. You keep the superseded revisions on purpose: to answer what the SOP required last March when a deviation was opened, to reconstruct which label was current when a batch shipped, to satisfy a retention schedule. So “in force” is not a flag on a document. It is a function of the date you are asking about. That single fact reorders the whole problem, because the dominant distractor here is a near-duplicate: revision C of the same section sitting one cosine step from revision D. So the reflex that helps on a messy corpus, raise k and let the model sort it out, does the opposite, since every extra passage is disproportionately likely to be another revision of the rule you already found, which is exactly the material that produces a faithful-but-wrong-version answer. The fix is to collapse a section to its in-force revision before the model sees anything, and to keep a gate at the end as the backstop.

What follows is the retrieval eval, the chunker, the citation check, the as-of gate, and the index as a second copy of the corpus that ages on its own clock. Think drug labels, loan-servicing policies, manufacturing SOPs: structured, versioned, access-controlled, consequential.

Two failures answer-level evals do not see

The failures below fall through a gap between two stages a generic pipeline does not have. The chunker decides whether a dosage table survives as a unit and whether the model is ever handed a section number to cite; a dedup step between retrieval and generation decides whether the model sees one revision of a rule or five. Both examples are redrawn so no real document shows through.

Q: “What is the maximum single dose for adult patients?” A: “The maximum single dose for adults is 400 mg, per Section 7.4.2.”

The 400 mg is correct and verbatim in the retrieved passage. There is no Section 7.4.2; the dosing text lives in Section 2.1. The chunk the model saw carried no locator, the citation format demanded one, and the model produced something plausible, which a faithfulness eval passes because the claim is supported and nothing in a faithfulness check looks at the citation.

Q: “How long must this record be retained?” A: “Records must be retained for 5 years from the date of creation.”

Faithful word for word to the retrieved SOP passage, which is revision C. Revision D, effective two months before the question was asked, changed the period to 7 years. Faithfulness, groundedness, and citation-format checks all pass, because each takes the retrieved passage as ground truth. The only wrong thing about the answer is the age of its source, and no answer-level metric measures that.

A third failure has no dialogue at all. A naive splitter cut a dosing table mid-row, so the retrieved chunk held “30 to 40 kg … 250” while “mg, twice daily” landed in the next chunk, and the answer came back as “250” with no unit. That one is upstream of anything the model or the eval can repair, so it is where the build starts.

Measuring retrieval before you touch the model

A strong model papers over mediocre retrieval by leaning on what it already knows, which is the one thing you do not want where the whole point is to ground on the controlled document. So retrieval gets its own eval: 240 questions, each with a handful of passages a domain reviewer graded 0 to 3 for relevance, scored with nDCG@k on exponential gain, 2^rel - 1, discounted by the log of the rank. The tempting way to compute that is sklearn.metrics.ndcg_score, and it is wrong in the flattering direction for two reasons that compound. It builds the ideal ranking from the documents you pass in, so if you pass only the retrieved docs, a relevant passage the retriever missed never enters the IDCG and the recall failure goes unpenalised. It also uses linear gain, rel, with no switch for the exponential form. A run that returns one grade-1 passage while missing two grade-3s scores a perfect 1.000 under ndcg_score where it should score about 0.08. Reaching for a general-purpose ML metric here is itself the tell; anyone who has run a TREC-style loop reaches for pytrec_eval or computes it directly against the full judged set:

import math

def dcg(ranked, judged, k=10):
    # exponential gain, matching the definition above; log2(rank+1) discount
    return sum((2 ** judged.get(d, 0) - 1) / math.log2(i + 1)
               for i, d in enumerate(ranked[:k], start=1))

def evaluate(run, qrels, k=10):
    """run: qid -> ranked doc_ids. qrels: qid -> {doc_id: graded relevance}."""
    ndcgs, recalls = [], []
    for qid, ranked in run.items():
        judged = qrels[qid]
        # IDCG over every judged doc, not just the retrieved ones, so a missed
        # grade-3 still enters the ideal and the recall miss is charged for
        ideal = sorted(judged.values(), reverse=True)[:k]
        idcg = sum((2 ** rel - 1) / math.log2(i + 1)
                   for i, rel in enumerate(ideal, start=1))
        ndcgs.append(dcg(ranked, judged, k) / idcg if idcg else 0.0)
        relevant = {d for d, g in judged.items() if g > 0}
        recalls.append(len(set(ranked[:k]) & relevant) / max(1, len(relevant)))
    return {"ndcg@k": sum(ndcgs) / len(ndcgs),
            "recall@k": sum(recalls) / len(recalls)}

There is a subtler trap in the qrels themselves. Grade a document, then change your chunker, and the thing you graded no longer exists as a unit, so judged.get(d, 0) scores every new chunk 0 and the eval compares one real system against noise. The way out is to judge char-span offsets in the source text layer rather than chunk ids, and let a chunk take the maximum grade of any judged span it overlaps, after which one qrels file scores any chunking. The judgments are pooled to depth 20 over the union of both chunkers’ output and adjudicated once, so the same gold set scores both on equal terms, and on that set the structure-aware chunker takes nDCG@10 from 0.52 to 0.74, most of the lift being the chunker rather than the reranker, which is the argument for measuring retrieval on its own.

The retriever underneath is hybrid, and the reason is the identifiers. A drug name, a policy number, a CFR reference are exactly what a dense embedding blurs together and what BM25 is precise about. A cross-encoder then reranks the top 50 down to 8 and writes its score onto each chunk, which the dedup step reads later.

A chunker that carries the locator

The naive splitter causes two of the three failures above. It counts tokens and cuts, through a table, between a cross-reference and the section it points at, across a numbered list, and it discards the section id the passage came from. The structure-aware version splits a table only on row boundaries with the header repeated, so a number is never separated from its unit, and every chunk carries revision and effective_date for the gate at the end plus a real section_id copied out of the document, so at citation time the model copies an id it was handed instead of inventing one.

The locator is the hard part, and the first version of this system hid it. unstructured hands you page_number and parent_id, not “s2.1”. Deriving a real locator means parsing the numbering scheme out of the Title elements and carrying a section stack across page breaks, appendices that restart the numbering as letters, and the many elements that carry no number of their own:

import re
from dataclasses import dataclass
from unstructured.partition.pdf import partition_pdf
from unstructured.chunking.title import chunk_by_title

SECTION_RE = re.compile(r"^\s*((?:\d+|[A-Z])(?:\.\d+)*)\s+\S")

@dataclass
class Chunk:
    text: str
    doc_id: str
    revision: str            # e.g. "SOP-114 rev D", for the gate
    section_id: str          # the real locator, lifted from the document
    effective_date: str      # the day this revision took force (valid time)
    is_table: bool
    score: float = 0.0       # written by the reranker; read by dedup

def section_locators(elements) -> dict[str, str]:
    """Map element id -> locator by carrying a section stack. Depth comes from the
    numbering itself, because hi_res rarely populates category_depth. A numbered
    Title sets the locator, an unnumbered one nests under its numbered parent, and
    a page break resets nothing."""
    stack: list[tuple[int, str]] = []
    loc: dict[str, str] = {}
    current = None
    for el in elements:
        if el.category == "Title":
            m = SECTION_RE.match(el.text)
            if m:
                label, depth = m.group(1), m.group(1).count(".")   # 2->0, 2.1->1, A.1->1
            else:
                depth = (stack[-1][0] + 1) if stack else 0
                parent = stack[-1][1] if stack else ""
                label = f"{parent}>{el.text.strip()}" if parent else el.text.strip()
            while stack and stack[-1][0] >= depth:
                stack.pop()
            stack.append((depth, label))
            current = label
        loc[el.id] = current
    return loc

def structure_aware_chunks(path, meta, max_chars=2000):
    # infer_table_structure keeps a table as one element with its HTML intact,
    # which is what stops a row being cut in half by a token counter
    elements = partition_pdf(filename=path, infer_table_structure=True, strategy="hi_res")
    loc = section_locators(elements)
    for chunk in chunk_by_title(elements, max_characters=max_chars,
                                combine_text_under_n_chars=500):
        # chunk_by_title never merges across a section boundary, so every original
        # element in a chunk shares one locator; take it from the first
        origin = chunk.metadata.orig_elements[0]
        is_table = chunk.category in ("Table", "TableChunk")
        yield Chunk(
            text=(chunk.metadata.text_as_html if is_table else chunk.text),
            doc_id=meta.doc_id,
            revision=meta.revision,
            section_id=loc.get(origin.id) or "?",
            effective_date=meta.effective_date,
            is_table=is_table,
        )

The uncomfortable part is that the fix fails in the same direction as the problem it solves. hi_res is a layout model doing table detection on a rendered page, and it does not always get the table: merged cells confuse the structure inference, rotated and multi-page tables are unreliable, and the last few rows of a table on a page can be dropped with no error at all. Picture that on a dosing table sorted by weight band, where the rows most likely to be lost are the last ones, the highest bands and the largest doses. A chunker that shreds a table gives you obviously broken chunks you notice, whereas one that drops the bottom three rows gives you clean, well-formed chunks that are wrong precisely where a wrong answer is most dangerous. So extraction gets validated against something the layout model did not produce: count the <tr> elements in the extracted table, compare that to a row count parsed from the document’s own text layer, and route any disagreement to a human at ingest instead of into the index.

A better boundary has a ceiling, and the document sets it. chunk_by_title never merges across a section boundary, which is what makes the locator assignment sound and also what guarantees that some obligations arrive in pieces, because the unit a regulation governs is a clause and the unit the chunker can see is a section. A rule in 4.2 qualified by an exception in 9.1 retrieves as a complete, well-formed, correctly cited passage that reads as unconditional. No max_chars fixes it, because the exception is elsewhere in the document, and a chunker big enough to span the two has stopped chunking. What is left moves downstream, to a citation check that scores a claim against the rule span and the exception span together, which only helps when the model cited both. Seven of the wrong answers in the audit below are this, and none of them are chunking failures.

Checking citations one claim at a time

An eval that scores a whole answer is too coarse for any of this. Split the answer into atomic claims and check each citation on its own terms, and follow the standard exactly, because the obvious implementation gets it wrong on this corpus. ALCE (Gao et al., 2023) defines citation recall as whether the concatenation of a claim’s cited spans entails it, then measures precision by necessity: a citation is redundant if the claim still holds after you remove it. Scoring each span against the claim in isolation, which is what the first cut of this system did, undercounts exactly the split-obligation claims above, where neither the rule span nor the exception span entails the claim alone.

import re
from sentence_transformers import CrossEncoder

# label order for this checkpoint is (contradiction, entailment, neutral)
nli = CrossEncoder("cross-encoder/nli-deberta-v3-base")
ENTAIL = 1

QTY = re.compile(r"\d+(?:\.\d+)?\s*(?:%|(?:mg|ml|kg|g|hours?|days?|months?|years?)\b)",
                 re.IGNORECASE)

def quantities(text: str) -> set[str]:
    return {m.group(0).lower().replace(" ", "") for m in QTY.finditer(text)}

def premise(span: Chunk) -> str:
    # an MNLI cross-encoder never saw HTML at training time, so a table premise
    # goes in linearised to "header: value" rows, not as raw text_as_html
    return linearize_table(span.text) if span.is_table else span.text

def entails(spans, claim, threshold=0.7) -> bool:
    if not spans:
        return False
    concat = "\n".join(premise(s) for s in spans)      # ALCE: does the whole span set entail?
    probs = nli.predict([(concat, claim)], apply_softmax=True)[0]
    return probs[ENTAIL] >= threshold

def check_claim(claim, cited, retrieved, threshold=0.7):
    """cited: [span_id]. retrieved: span_id -> Chunk."""
    missing = [s for s in cited if s not in retrieved]
    if missing:
        # the cheapest check in the pipeline: a locator that was never retrieved
        # cannot support anything, and it costs zero model calls
        return "fabricated_locator", missing
    spans = [retrieved[s] for s in cited]
    unbacked = quantities(claim) - set().union(*(quantities(s.text) for s in spans))
    if unbacked:
        # the entailment model is the wrong instrument for a number, so the number
        # is matched by string before the model gets an opinion on the sentence
        return "unsupported_quantity", sorted(unbacked)
    if not entails(spans, claim, threshold):
        return "unsupported_citation", cited
    # precision by ablation: a citation is redundant if the rest still entail
    redundant = [cited[i] for i in range(len(cited))
                 if entails(spans[:i] + spans[i + 1:], claim, threshold)]
    return "ok", redundant

A pair of details in there are the kind you only learn by running it. The premise for a table has to be linearised, because the cross-encoder was trained on sentence pairs and has never seen an HTML <table>; feed it text_as_html and you are scoring entailment out of distribution and the number means nothing. And apply_softmax=True is not optional on this three-label checkpoint, since the model emits raw logits and the threshold of 0.7 only means a probability once you have put the logits through softmax.

The quantity check in front of the entailment pass is there because of a hard limit. Entailment models are weak at exactly the token this corpus turns on: EQUATE, the benchmark built for that question, found that NLI systems do not beat a majority-class baseline on quantitative pairs and do not learn to reason with quantities implicitly (Ravichander et al., 2019). A premise reading 250 mg twice daily and a claim reading 500 mg twice daily share every content word and differ in one character. So the number never reaches the model at all. It comes out of the claim by rule and has to appear in the cited spans first, a shallower check that fails the other way, accepting a number present in the passage but attached to the wrong weight band, which is what table validation at ingest is for.

The cheapest check closed the biggest hole. Confirming a cited section id was actually retrieved is a set lookup with no model call, and it is the entire defence against the fabricated locator. Before the chunker carried real ids, about 12% of cited answers contained at least one section id that appears nowhere in the source document, and after, under 1%. The entailment pass is where care is needed with the number you report. The checker filtered at p[ENTAIL] >= 0.7, so the survivors pass entailment by construction and quoting their pass rate as precision would be circular. The honest number is human-labelled. On a 200-answer sample, one domain reviewer scored raw citation precision at 0.71, and on delivered answers it reaches 0.96.

Recall moves too, from 0.86 to 0.92, and the reason is easy to state backwards. Pure filtering could not do it, because dropping a citation never invents one. What moves recall is that a blocked claim is sent back for regeneration, and that check_claim flags an uncited claim as well as an unsupported one, so a claim that should have carried a citation and did not gets another attempt on the same pass. Precision and recall are both measured on delivered answers, a different population from the raw ones, and reporting one on delivered output and the other on raw output gives you an incoherent pair.

The 0.96 is also the number the checker’s own accuracy predicts, which is the consistency check to run. On the same human-labelled sample the checker calls the entail class correctly about 88% of the time. Raw precision of 0.71 leaves 29% bad citations, of which the checker misses roughly an eighth, so the residual should land near 3.5% and delivered precision near 0.965. It does. Had the delivered number come out well above what the checker’s error rate allows, the measurement would be the thing to distrust, not the pipeline.

The gate is an as-of join

Faithfulness takes the retrieved passage as truth. On a bitemporal corpus a passage is truth only if it comes from the revision in force for its section as of the date the question is about. That word “section” is load-bearing. A document-level gate blocks any answer citing a document with a newer revision on file, so a rev-D edit to section 5 blocks a correct answer citing section 2.1 that has not changed since rev B. The gate diffs at the section level and joins on the effective date, which is why every chunk carried one:

from datetime import date
from enum import Enum

class Decision(Enum):
    PASS, BLOCK = "pass", "block"

def asof_revision(doc_id, section_id, valid_at, known_at=None):
    """Revision in force for this one section, on two clocks.

    valid_at  (valid time)       : which revision was in force on that date
    known_at  (transaction time) : restricted to revisions the registry had
                                   recorded by then; None means everything we
                                   know now"""
    edits = [r for r in REVISIONS[doc_id]
             if section_id in r.changed_sections
             and r.effective_date <= valid_at
             and (known_at is None or r.recorded_at <= known_at)]
    if not edits:
        return None
    # same-day effective dates are ordinary in document control, so break ties on
    # the registry sequence, never on dict order
    return max(edits, key=lambda r: (r.effective_date, r.seq)).revision

def freshness_gate(cited_spans, valid_at=None, known_at=None):
    valid_at = valid_at or date.today().isoformat()
    stale = [s for s in cited_spans
             if s.revision != asof_revision(s.doc_id, s.section_id, valid_at, known_at)]
    return (Decision.BLOCK, stale) if stale else (Decision.PASS, [])

For a live question valid_at is today and known_at is unset, while for an auditor asking what the SOP required when a deviation was opened last March both are last March, and the function returns what the system would have said then, which is the answer the auditor needs. Freezing only the valid axis gives you today’s beliefs about March, which reads correct and is not what happened. The gate is the backstop, though; the structural fix sits earlier, in collapsing each section to its in-force revision before the model sees anything, so the five copies of a rule that retrieval dragged in never crowd the context.

def dedup_to_current(spans, T):
    best = {}
    for s in spans:
        if s.revision != asof_revision(s.doc_id, s.section_id, T):
            continue                            # a superseded copy never reaches the model
        key = (s.doc_id, s.section_id)
        if key not in best or s.score > best[key].score:
            best[key] = s
    return list(best.values())

Raising k is the opposite move. Longer context degrades accuracy once the window fills with plausible but irrelevant passages (Liu et al., 2023), and here those passages are the near-duplicate revisions. The gate then catches the residual, where a span slipped through with a revision the registry has since moved past.

Faithfulness and correctness mostly move together on a static corpus and diverge on a versioned one, and the divergence is a single cell of a 2x2. On a 500-answer audit set, deliberately oversampled toward multi-revision documents to characterise a rare failure:

A faithfulness gate keeps both faithful cells and rejects both unfaithful ones. It passes 440 answers, 38 of them wrong, so 8.6% of what faithfulness approves on this set is incorrect, and that subset looks most trustworthy of all because it is internally consistent with a real document. Attributing all 38 to versioning would be the thesis grading its own homework, so the audit broke them down: 26 were faithful to a superseded revision, 7 cited the general rule and missed a qualifying exception in another section, and 5 misread a table. Only the first 26 are what the freshness machinery removes; the middle 7 are what the concatenation-and-ablation citation check is for, and the last 5 are what table validation catches at ingest.

The wrong-version rate has to be read carefully, because the number that is easy to compute is not the number that matters. Counting blocked answers gives a detection rate, and a gate converts wrong answers into blocked ones without lowering how often retrieval serves stale, so a gate-only system blocks about 7% of answers on the revised subset forever. That is the number the shadow audit tracks. The delivered wrong-version rate, what reaches a user, is what dedup drives down, to about 0.6%.

The alarm has to watch the right clock, and for a while I didn’t have it on the right one. The obvious alarm counts stale cited spans, but a lagging index cannot be caught that way. Dedup and the gate both read the registry the index is lagging behind, so they agree with each other by construction. If document control publishes rev E and the index has not ingested it, the registry lookup also returns rev D, the cited rev-D span matches, and the gate passes. What exposes the lag is the gap between two clocks the corpus already has, in that the registry records a revision at recorded_at, the index stamps a chunk with indexed_at, and a publish the pipeline has not caught up with is a revision whose recorded_at has no corresponding chunk.

def index_lag(as_of=None):
    """Revisions the registry knows about that the index has not ingested. A
    property of the two clocks, not of query traffic, so it fires whether or not
    anyone has asked about the affected section yet."""
    as_of = as_of or date.today().isoformat()
    have = {(c.doc_id, c.revision) for c in INDEX_MANIFEST}
    # non-empty is the alarm; age of the oldest pending revision is the severity
    return [(doc_id, r.revision, r.recorded_at)
            for doc_id, revs in REVISIONS.items() for r in revs
            if r.recorded_at <= as_of and (doc_id, r.revision) not in have]

That is the signal that survives dedup shipping. The stale-cite rate still measures blast radius, alarmed on a jump above its running baseline rather than on an absolute level that sits at the corpus’s natural churn and would trip every run. But it is a lagging indicator. It goes quiet once dedup starts filtering the same spans the gate would have caught, which is precisely when you most want to know the index is behind. The gate ran in measure-only mode from week one, which is how it could see the week-four spike: document control published a batch of new revisions, the index took a few days to catch up, and for that window retrieval served last month’s revision of documents that had already changed while faithfulness stayed high throughout. The team read the spike, shipped dedup, and flipped the gate to blocking after week four, and a gate that only started blocking in week five could not have detected the week-four spike at all.

The index is a copy, and it ages on its own clock

Everything above treats the corpus as the thing that moves. The index moves too, on a different schedule, and where they disagree the index answers anyway. Access control is where that disagreement has a name. Chunks carry the entitlements of the document they came from, and a question is answered over the subset the asker may see. Apply that subset after the vector search and it is a post-filter, which fails by coming up short: pgvector documents the arithmetic plainly, that with an approximate index the filter runs after the scan, so a selective predicate leaves a fraction of the candidates the scan was sized for (pgvector). Nothing downstream knows it happened. Faithfulness, the citation check and the freshness gate are each scoped to the passages that came back, so a restricted user gets a confident answer built from whatever fraction of the evidence their clearance let through, and is never told that the material answering the question is not theirs to see.

Push the predicate into the search and the problem moves. It works. A store that builds filter-aware edges into its graph from a payload index returns k results that all satisfy the predicate (Qdrant). But the predicate now has to be a field the index holds, and an entitlement is not a property of the document. It is a property of a person’s membership in a group, and that membership changes in the identity system, where nothing publishes a revision and nothing triggers a re-index. Elasticsearch draws the boundary explicitly: a document-level security role query may not make remote calls, which rules out a terms lookup (Elastic). The filter cannot ask what the entitlements service says now. It repeats what the index was told.

So the split runs the other way from the obvious one. Group ids go on the chunk, because those change at the speed of document control; the person is resolved to their groups per query, because that changes at the speed of a leaver processed on a Friday afternoon. A change to a document’s own ACL is then a re-index, and it belongs on the same lag alarm as a revision publish. Neither arrangement reaches the ranking statistics.

-- both predicates go inside the search, not after it. pgvector filters after the
-- index scan, so without iterative scans a narrow entitlement returns short
SET hnsw.iterative_scan = relaxed_order;

SELECT chunk_id, doc_id, section_id, revision
FROM chunks
WHERE acl_groups && $1        -- groups resolved per query from the identity system
  AND effective_date <= $2    -- valid time; the section-level as-of join runs after
ORDER BY embedding <=> $3
LIMIT 50;

Elastic says so directly about document-level security, that scores are computed without taking the role query into account (Elastic), and any BM25 index not partitioned per tenant takes its idf from documents the asker cannot see. No passage leaks, but the ranking a restricted user gets is still a function of documents that are not theirs, which is worth knowing before you tell an auditor an answer came only from what that user could access.

The second disagreement is a redaction. A policy changes, the document is reissued with an identifier removed, and the text in the index updates when the document is re-ingested, while the vector updates only if something re-embeds it, so a pipeline that skips unchanged doc ids, or an embedding cache keyed on doc id and revision, hands the old vector back. The index then holds a vector computed from a string that exists nowhere in the corpus, and that is not only a stale ranking signal. Embeddings are close enough to invertible to be treated as the text they came from: a multi-step inversion recovers 92% of 32-token inputs exactly, including names out of clinical notes (Morris et al., 2023). Whatever the redaction policy says about the string it says about the vector and about every backup the vector sits in. Key the embedding cache on a hash of the chunk text and a redaction produces a new key by construction.

Erasure is the same shape with a deadline. The request has to reach the source document, the chunks, the vectors, the cache, and any stored answer that quoted the passage, and only the first is in the document-control system’s inventory. In Lucene a delete only flags the document inside its segment, and the bytes go when a merge rewrites that segment (Elastic), so the day a record stops being returned and the day it leaves the disk are different days, and an attestation is about the second. The retention schedule can also forbid the deletion outright, since GDPR Article 17(3)(b) does not require erasure where processing is necessary for compliance with a legal obligation. For a record you must keep, the resolution is redaction of the personal data inside it, which puts you back at the paragraph above, holding a vector.

What each stage costs

The two cheap rows carry the version story, and they are cheap because they are metadata joins over data document control already maintains. Dedup is prevention and the gate is detection, and reading the ledger as if the 9-millisecond gate bought the rate drop credits the wrong one. The entitlement and as-of predicates get no row at all, because they are filters inside the retrieval the first row pays for. The expensive row is the citation check, a second model pass over every answer, which makes it a decision by risk tier: on internal drafting or a search box, run the free locator-existence check and the gate and skip entailment; where an answer reaches a regulator, a patient, or a lending decision, pay the 580 ms. There are also corpora where most of this is overkill. If the documents never version, run the citation check and drop the freshness machinery, and if they are flowing prose with no tables and no cross-references, a recursive splitter gets you most of what the structure-aware chunker would. The full apparatus is the price of a corpus that is structured, bitemporal and compartmented at once.

Where this stays open

The gate certifies that a cited span is from the in-force revision. It does not, however, certify that the span answers the question, and a model that grounds a confident answer in the correct revision of the wrong section clears every check here, so the honest scope of this post is the version and citation failures rather than correctness in general. The changed_sections set the whole as-of join depends on is only as good as the diff that produced it. If document control ships a revision marked as touching section 5 when it also reworded section 2.1, the gate treats a superseded 2.1 span as fresh and nothing downstream catches it. The section diff is a control now, which means it needs the same review discipline as the code, and a revision that lands without a trustworthy changeset is a compliance event rather than a data-entry lag.

The index has a version history of its own, on neither of the corpus clocks. The as-of join reconstructs which revision was in force in March. It does not reconstruct the answer given in March, because that answer came out of a particular index: an embedding model, a chunker version, a reranker checkpoint. Re-embed with a better model, which you will, and the same question at the same valid_at and known_at returns a different set of passages and can cite a different section, with nothing in the bitemporal machinery reporting a change, because the corpus did not move. So the answer record carries the index build alongside the two dates, and the locators in it have to outlive the build that minted them, which is the constraint the qrels ran into and has the same answer: span offsets in the source text layer, never chunk ids.

The last open edge is the checker’s own drift. The 0.88 entail accuracy was measured once, on one reviewer’s labels, on today’s document mix. Add a document type with tables the cross-encoder linearises badly, or a reviewer who grades exceptions more strictly, and that number moves without anything in the pipeline reporting that it did. The checker is a model, its labelled set ages like any other, and the version machinery it certifies is only as current as the last time someone checked the checker.