Skip to main content
Hashnode
AI Decoded

When AI Hallucinations Are Really Data Contract Failures

Updated 3 min readMarkdown
A
Abivarma R S

Short version of what I took away from this one: when a RAG assistant quotes a confidently wrong number, pull the retrieved records before you blame the model.

Often the model was handed the wrong number and repeated it faithfully. A new prompt can't fix that. Neither can a new vendor.

Here's the mechanism, with an invented example. An insurer's claims copilot retrieves from its policy database. A migration for a new mobile app renames max_coverage_limit_usd to per_incident_deductible_usd. Unit tests pass, the ETL job exits cleanly, no alert fires. The type didn't change, so a type check sees nothing wrong.

Days later an adjuster asks for a policy's coverage cap. The retrieved chunk has no coverage limit field, only dollar figures under deductible labels. The copilot answers $12,500 instead of the real $1,250,000. It didn't invent that number. It retrieved it.

Classic BI had a buffer: a bad record shows up as an odd chart and someone notices. In RAG, the pipeline's output is all the assistant sees. A green job that writes reinterpreted fields into the index ships the problem straight to the user.

Kleppmann's framing in DDIA is the useful lens: schema evolution is compatibility in two directions, new code reading old data and old code reading new data. The format matters for renames too. Avro matches fields by name, so a safe rename keeps the old name as an alias. Protobuf matches by tag number, so names can change but tags can't. A raw JSON dump checks nothing.

What I'd put between the source and the model:

  • Compatibility checks in CI. Register every schema an AI system consumes. Confluent's default mode is BACKWARD; for AI consumers I'd set FULL or FULL_TRANSITIVE explicitly. Renames or removals of fields an AI consumer reads go through a deprecation window.
  • A validator before embedding. Pydantic mode="after" validators for invariants like coverage limit >= deductible. Use raise ValueError, not assert, because -O strips asserts. Rejects go to a dead-letter queue with a named owner.
  • Provenance on every chunk. Schema version, source, as-of timestamp, so you can filter stale chunks and trace an incident to the exact record.

Plus one team rule: when a wrong answer traces back to bad source data, the fix goes into the contract or the pipeline, not the system prompt.

To be clear about the limits. Contracts don't make answers correct. If the contract maps "coverage cap" to the wrong field, you get a wrong answer with a clean audit trail. They also cost something: pushback from backend teams, a quarantine backlog someone has to work through, real effort on legacy systems. Start where a wrong answer costs money or creates compliance risk.

The full article has the diagrams, the sources and the complete four-step audit: https://medium.com/towards-data-engineering/the-contaminated-fuel-trap-when-ai-hallucinations-are-data-contract-failures-55cca4d41377

About the author

  • Abivarma R S
    I write about AI agents in production: retries and idempotency, tool permissions, evaluation, cost and the data foundations underneath. 13+ years from mainframes to cloud data engineering to enterprise AI. Full deep dives on Medium (abivarma.medium.com), Sunday digest on Substack (abivarma.substack.com). Views my own.

Comments(1)

Join the discussion
OOneLifeVibe23h ago

The insurance example makes the failure mode concrete in a way a general warning about hallucinations never would. A field renamed from max_coverage_limit_usd to per_incident_deductible_usd, with the type unchanged, passes unit tests and a clean ETL exit, and then the copilot faithfully reports $12,500 instead of $1,250,000. That is the key reframe: the model did not invent the number, it retrieved it, so tuning the prompt or swapping the vendor cannot fix it.

I also liked the contrast with classic BI, where a bad record shows up as an odd chart and someone notices, while in RAG the pipeline output is all the assistant sees. The three defences you put between source and model fit that diagnosis: compatibility checks in CI, a Pydantic validator with mode="after" for invariants like coverage limit being at least the deductible, and provenance on every chunk so stale data can be filtered and an incident traced. The detail about using raise ValueError instead of assert, since -O strips asserts, is exactly the kind of small, practical point that saves someone a second incident.

More from this blog