Reliable Extraction, Batch, and Independent Reviewers

Valid JSON proves the shape survived. It does not prove the facts did.

Type: Reference Languages: Python Prerequisites: Validate the Claim, Not the Confidence, Structured Output Is an Untrusted Contract; Phase 14, Lesson 39 Time: ~135 minutes

Learning Objectives

• Define extraction criteria that reduce false positives and ambiguous labels
• Use schemas, examples, nullable fields, enums, and evidence spans deliberately
• Separate syntax, schema, semantic, and provenance validation
• Design bounded retry and independent reviewer passes
• Choose real-time or batch processing from workflow requirements

The Problem

A pipeline extracts contract obligations into valid JSON. Every record matches the schema. Legal reviewers still reject 18 percent.

The model fills missing dates with plausible values, labels background statements as obligations, and maps unfamiliar categories to the nearest enum. A retry loop feeds the same prompt back until validation passes. Since validation checks only types, the invented values become more confidently formatted.

The team solved serialization and mistook it for correctness.

The Concept

Define the Judgment Before the Schema

A schema says what fields exist. Criteria say what qualifies.

For an obligation extractor, define:

• obligated party must be explicit or unambiguously linked
• required action must be stated, not merely discussed
• trigger and deadline are extracted only when supported
• evidence span must contain the claim
• unknown values remain null
• unsupported category uses other with a note or triggers review
• exceptions and negations change the result

Without these rules, annotators, model, and evaluator apply different tasks.

Use Few-Shot Examples for Boundaries

Examples are most useful where reasonable people make different judgments.

Include:

• a clear positive
• a near miss
• a negated obligation
• missing date represented as null
• a category outside the enum
• two obligations in one paragraph
• conflicting clauses

Each example should demonstrate the reason, not only the answer. Do not flood context with redundant easy cases.

Make Absence Representable

If a field may be unknown, the schema needs an explicit state. Forcing a string encourages invention.

{
  "type": "object",
  "properties": {
    "party": {"type": ["string", "null"]},
    "action": {"type": "string"},
    "deadline": {"type": ["string", "null"]},
    "category": {
      "type": "string",
      "enum": ["payment", "delivery", "reporting", "other"]
    },
    "evidence_span": {"type": "string"},
    "needs_review": {"type": "boolean"}
  },
  "required": ["party", "action", "deadline", "category", "evidence_span", "needs_review"],
  "additionalProperties": false
}

Required plus nullable forces an explicit decision: supported value or known absence. It prevents silent field omission.

Use Tool Use for Typed Output

A no-side-effect extraction tool can carry the schema. Tool choice can require that typed record when the application needs it. Strict schema features can guarantee valid structure where current APIs support them.

Do not call a real action tool just to obtain structured output. Extraction and execution have different authority.

Validate in Four Layers

flowchart LR
    O["Model output"] --> J{"Syntax valid?"}
    J --> S{"Schema valid?"}
    S --> M{"Semantics valid?"}
    M --> P{"Evidence supports claim?"}
    P --> A["Accept"]
    J -->|"no"| R["Targeted repair"]
    S -->|"no"| R
    M -->|"no"| H["Retry or review"]
    P -->|"no"| H

Syntax

Can the payload be parsed?

Schema

Are fields, types, enums, and bounds valid?

Semantics

Do cross-field relationships hold? A deadline cannot precede an effective date when the domain forbids it. A needs_review false result cannot accompany an unsupported category.

Provenance

Does the evidence span actually support the extracted claim, and does it come from the correct source version?

Only the last two detect many confident hallucinations.

Feed Back the Smallest Useful Error

On repair, return structured validation feedback:

{
  "category": "semantic_validation",
  "field": "deadline",
  "message": "The extracted date does not appear in the evidence span.",
  "allowed_action": "Set deadline to null or select a supported span."
}

Do not say only "try again." Keep the original source and prior result. Limit retries. Repeated semantic failure should escalate instead of converting uncertainty into latency and cost.

Separate Generator and Reviewer

The generator extracts. The reviewer receives source, candidate record, and a rubric. It checks:

• required evidence exists
• span supports every non-null claim
• negation and exceptions were handled
• category fits the definition
• unknowns were not invented
• conflicts and ambiguity are flagged

Use a fresh context for stronger independence. The reviewer returns finding IDs, fields, evidence, and disposition. It does not silently rewrite the record.

Measure reviewer precision and recall against human labels. A model judge is an instrument, not ground truth.

Choose Batch for the Workflow

The July 2026 CCAR-F public guide specifies a 50 percent Message Batches cost reduction, an up-to-24-hour processing window with no guaranteed latency SLA, and no multi-turn tool calling inside one batch request. Those are dated exam reference facts, not a promise that pricing or service limits will remain unchanged. Confirm current pricing, limits, retention, and feature compatibility in the Message Batches documentation before deployment.

Batch fits:

• large offline extraction
• evaluation datasets
• nightly classification
• backfills and reprocessing
• independent review after generation

Real-time fits:

• interactive user response
• tasks with a strict short latency bound
• adaptive tool use during the same request
• workflows requiring immediate approval or feedback

Do not use batch when the next step depends on an external action the model must observe mid-request. Precompute inputs or split the workflow into jobs.

Make Batch Jobs Reconciliable

Give every item a stable custom_id. Persist source version, schema version, prompt version, and expected output location. Results may return out of order.

Handle:

• success
• validation failure
• provider failure
• expiration
• duplicate submission
• partial job completion
• retry after source change

Never join results to inputs by array position.

Evaluate the Error You Care About

For extraction:

• field precision and recall
• exact or normalized match where appropriate
• evidence-support rate
• false-positive rate for high-risk fields
• null calibration
• category confusion matrix
• reviewer disagreement
• cost and latency per accepted record

Averages can hide a dangerous false-positive class. Stratify by document type, language, length, and risk.

Build It

Interactive Lab

20-batch-review-confidence

Use the confidence and review simulator to move records through syntax, schema, semantic, and provenance gates. Adjust false-positive cost and reviewer coverage to see why valid JSON and model confidence are insufficient release criteria.

Practice Lab

Change one supported date to an invented value, run the four validation layers, and route the failed record to adjudication rather than another blind retry.

Shipped Artifact

The filled outputs/extraction-review-report.md contains a batch job with stable custom_id values, nullable unknowns, shuffled results, review findings, and an adjudication state.

Verify It

Run its deterministic verifier:

cd certifications/claude/lessons/20-reliable-extraction-batch-and-reviewers
python3 code/main.py
python3 -m unittest discover -s code/tests -v

The quiz checks repair, batch, and reviewer decisions.

Capstone Connection

Carry the verified report into the Architect Foundations extraction scenario as evidence for all four validation layers.

Create an extraction pipeline for support-policy changes.

Output Contract

Extract policy ID, effective date, affected region, action type, threshold, evidence span, source version, and review state. Every uncertain field is nullable or has an explicit other state.

Dataset

Build at least 40 examples:

• 15 clear changes
• 10 background statements with no change
• 5 negations or exceptions
• 5 missing dates or thresholds
• 5 conflicting versions

Passes

1. generator with strict schema
2. deterministic syntax and schema validation
3. semantic relationship validation
4. independent evidence reviewer
5. human adjudication for disagreements

Experiment

Compare zero-shot criteria, few-shot boundary examples, and generator plus reviewer. Report false positives, evidence support, cost, and latency.

Batch Design

Submit records with stable IDs. Randomize result order in a test. Inject partial failure and prove reconciliation keeps completed records and retries only safe items.

Use It

In production, store the raw source separately from normalized extraction. Keep the source version and evidence offsets. When criteria or schema change, create a new output version rather than overwriting historical decisions.

If a human corrects a record, store a reason code. Use disagreements to improve criteria and the evaluation set before modifying the prompt.

For high-risk extraction, review can be stratified: every high-impact field, low-evidence record, or new document type plus a random sample of ordinary cases.

Exam Decision Patterns

When JSON is valid but content is wrong, add semantic and evidence validation. When consistency is weak at a judgment boundary, use explicit criteria and few-shot examples.

Prefer answers that:

• use null or other rather than invention
• force a typed output without triggering a real action
• feed specific validation errors back with a retry limit
• separate generator and reviewer
• use batch for asynchronous, tool-independent workloads
• reconcile results with stable IDs

Common Traps

Schema Equals Truth

Types cannot prove that a value appears in or follows from the source.

Required Non-Nullable Fields

The model invents a plausible value because the contract has no representation for absence.

Infinite Repair

The same ambiguous source produces repeated guesses. Escalate after a bounded attempt.

Reviewer Rewrites Silently

The system loses which claim failed and why. Return structured findings before any controlled correction.

Exercises

1. Add a semantic rule linking threshold and currency.
2. Design negative examples that reduce false obligations.
3. Calibrate a reviewer against human labels and report disagreement.
4. Build stable-ID reconciliation for shuffled batch results.
5. Compare cost per accepted record for one-pass and reviewer pipelines.

Key Terms

Further Reading

• Claude structured outputs documentation
• Claude Message Batches documentation
• Phase 11, Lesson 03 for structured outputs from first principles
• Phase 14, Lesson 39 for reviewer agents
• Phase 17, Lesson 15 for batch architecture