The benchmarks point the same way, with caveats. In a 2023 data.world benchmark (43 business questions on an insurance schema, zero-shot), GPT-4 writing SQL got 16.7% right. Through a knowledge graph of the same data it got 54.2%. Same model, same questions. On the questions that needed more tables, SQL accuracy was 0%. Models have improved since: in dbt Labs' 2026 rerun, current models writing raw SQL scored 64.5% on 11 questions. That's a vendor test, and still not a number I'd put in a board report.

SQL skill isn't the gap. Meaning is. Three failures keep showing up:

• Polymorphic columns. amount means gross, net or a credit memo depending on another column. The model writes SUM(amount). Valid SQL, wrong meaning.
• Join fan-out. Guess a join path at the wrong grain and rows multiply. Sums inflate. Nothing errors out.
• Definition schism. Finance, Marketing and Support each mean something different by "active customer". The agent gets whichever one its tables happen to encode.

The pattern I'd use has three tiers. The raw lakehouse stays the store, with no direct agent access. Metric definitions live in versioned YAML (dbt Semantic Layer, Cube), reviewed in git. The agent asks for metrics by name, for example through the dbt MCP server's query_metrics, and the layer compiles the SQL. Turn off execute_sql and text_to_sql for that agent. dbt's own tool list warns that execute_sql doesn't block write statements.

In dbt's 2026 rerun, the semantic layer scored 100% on the 8 questions it could answer and declined the 3 multi-hop ones instead of guessing. A refusal beats a wrong answer.

It's not a cure, and I'm skeptical of parts of my own playbook. Consistent isn't correct: the agent can still pick the wrong metric, and a definition can be wrong or stale. Getting Finance, Sales and Product to agree on "churned" is political. One-off exploration doesn't fit a fixed metric list.

Before agents touch the warehouse, I'd start here:

1. Ask three developers to compute your main metric from raw tables. If they differ by even a couple of percent, an agent writing raw SQL won't do better.
2. Take broad raw table access away from agent service accounts. Curated views or semantic-layer APIs only.
3. Write your first five canonical metrics in YAML, in git, with required review.

Data volume isn't data readiness.

The full article has the diagrams, the benchmark breakdown, all sources and the complete four-step readiness audit: https://abivarma.medium.com/the-semantic-debt-trap-why-ai-cannot-query-your-raw-data-lakehouse-f3cdd1c48fa3

Often the model was handed the wrong number and repeated it faithfully. A new prompt can't fix that. Neither can a new vendor.

Here's the mechanism, with an invented example. An insurer's claims copilot retrieves from its policy database. A migration for a new mobile app renames max_coverage_limit_usd to per_incident_deductible_usd. Unit tests pass, the ETL job exits cleanly, no alert fires. The type didn't change, so a type check sees nothing wrong.

Days later an adjuster asks for a policy's coverage cap. The retrieved chunk has no coverage limit field, only dollar figures under deductible labels. The copilot answers $12,500 instead of the real $1,250,000. It didn't invent that number. It retrieved it.

Classic BI had a buffer: a bad record shows up as an odd chart and someone notices. In RAG, the pipeline's output is all the assistant sees. A green job that writes reinterpreted fields into the index ships the problem straight to the user.

Kleppmann's framing in DDIA is the useful lens: schema evolution is compatibility in two directions, new code reading old data and old code reading new data. The format matters for renames too. Avro matches fields by name, so a safe rename keeps the old name as an alias. Protobuf matches by tag number, so names can change but tags can't. A raw JSON dump checks nothing.

What I'd put between the source and the model:

• Compatibility checks in CI. Register every schema an AI system consumes. Confluent's default mode is BACKWARD; for AI consumers I'd set FULL or FULL_TRANSITIVE explicitly. Renames or removals of fields an AI consumer reads go through a deprecation window.
• A validator before embedding. Pydantic mode="after" validators for invariants like coverage limit >= deductible. Use raise ValueError, not assert, because -O strips asserts. Rejects go to a dead-letter queue with a named owner.
• Provenance on every chunk. Schema version, source, as-of timestamp, so you can filter stale chunks and trace an incident to the exact record.

Plus one team rule: when a wrong answer traces back to bad source data, the fix goes into the contract or the pipeline, not the system prompt.

To be clear about the limits. Contracts don't make answers correct. If the contract maps "coverage cap" to the wrong field, you get a wrong answer with a clean audit trail. They also cost something: pushback from backend teams, a quarantine backlog someone has to work through, real effort on legacy systems. Start where a wrong answer costs money or creates compliance risk.

The full article has the diagrams, the sources and the complete four-step audit: https://medium.com/towards-data-engineering/the-contaminated-fuel-trap-when-ai-hallucinations-are-data-contract-failures-55cca4d41377

That's my argument, not a survey finding, but the numbers leave room for it. In McKinsey's 2026 State of AI survey, nearly nine in ten respondents report regular AI use in at least one business function. Only 37% attribute any EBIT impact to it, and about 6% clear the high performer bar. These are self-reported shares of respondents, and not reporting impact doesn't mean a program is stuck. Still, that figure barely moved. It was 39% in the 2025 edition.

Leadership tends to buy AI as an application. Engineering inherits it as an infrastructure tax: integration, data work and review that the application's price doesn't show. EY estimates tokens are roughly a third of what an agent costs to run. That's a consultancy estimate and your ratio will differ, but I wouldn't budget from a per-token price.

Accuracy hides cost too. Take a hypothetical: 95% per clause, eight clauses, independent errors. 0.95^8 is 66.3%, so roughly a one-in-three chance of at least one extraction error per contract. Someone has to pay for the checking.

What I'd propose is a layer between the model and the backends, which I call the Substrate Adapter Pattern. It's my composition of existing patterns, not an established one, and no source I've found measures its results.

• Normalization gate. A deterministic proxy turns messy legacy data into validated, typed records before the model sees a token.
• Execution sandbox. Every tool call is a non-binding proposal. A deterministic gateway checks permissions, business invariants and an idempotency key, then stages the change in an outbox. Above a risk threshold, a person signs off.
• Scoped steps. Narrow, single-purpose steps instead of one general agent, each with a check you can test, so you can see which one failed.

This doesn't make answers correct. A typed schema proves shape, not meaning. If the assistant proposes a valid credit on the wrong account, idempotency just guarantees it's applied exactly once. That still takes agreed definitions, data quality work and business review.

Before a pilot gets more budget, I'd start here:

1. Price the verification. You can't check only the wrong outputs, because you don't know which ones they are.
2. Make write tools proposal-only. Any direct POST, PUT or DELETE without an idempotency gateway gets converted.
3. Track cost per verified transaction against your pre-AI baseline, with model fees, infrastructure and review time included.

Towards AI published the full article, and thanks to them for taking it. It has the diagrams, the sources and the complete checklist: https://pub.towardsai.net/the-substrate-trap-why-enterprise-ai-use-outruns-its-ebit-impact-6de1f29e6606

I'd start with the first and make the second earn its place with a test.

Where vectors break

Cosine similarity scores topical closeness. That's why vector search is fine for a single-document lookup like a return policy. It can't follow a link from one fragment to the next, the way a foreign key ties two rows together.

Take three facts in separate chunks: CHIP-402 is assembled into Engine-B, Engine-B is installed in Titan-EV, Titan-EV is committed under contract LON-FLEET-26. Ask about the chip's impact on contract penalties and top-k retrieval may return the first and third and miss the middle one. That's a failure mode to test for, not something I've measured. Without that link the model has to bridge the gap itself, and it can invent relationships between entities that are all real.

What the graph changes

The hybrid design I propose keeps vector search. Ingestion also extracts entities and relationships into a property graph. A dependency question finds the root entity, runs a bounded traversal, and the path goes into the prompt with the relevant clauses. The model explains a path it was given. It doesn't infer links from fragments.

It isn't free. Extraction runs over the whole corpus. "Bavaria AutoPart GmbH" and "BAP Logistics" have to resolve to one entity, without gluing together things that are separate. Dense graphs need hard hop limits, 2 or 3, say. And extraction errors or stale records end up in the path.

Before you graph anything

• Build a test set of about 20 questions where the answer needs facts from two or more documents, and score it next to your single-fact questions. Single-fact fine and multi-hop low points at retrieval structure. That's a heuristic, not proof.
• Try query decomposition or iterative retrieval: ask for the chip's vehicles, then those vehicles' contracts. Some multi-hop cases get fixed with no graph at all.
• If that's not enough, pilot a bounded graph in one high-risk domain with dense dependencies. Don't attempt an enterprise-wide graph in one step.

The full article has the diagrams, the sources and the complete checklist: https://abivarma.medium.com/the-polaroid-trap-why-vector-search-fails-at-multi-hop-enterprise-reasoning-507dfeb19664

· · ·

Seven agents. Each one running at 90% accuracy. I know exactly how the design review went — someone pulled up a table of benchmarks, everything green, the PR got approved. Clean architecture. Beautiful on paper. Deployed to production.

End-to-end success rate on that system? 48%. Half the requests failing. Every individual agent still showing green on every monitoring dashboard.

That's 0.9 multiplied by itself seven times. I've seen this pattern across fintech underwriting teams in Mumbai, SaaS engineering floors in Hyderabad, document processing pipelines in Pune. The system looks solid at every individual layer and collapses as a whole. Nobody calculated the product. They only ever measured the parts.

This is the agent coordination trap. It's not a model problem. It's not a prompt problem. It's a topology problem that gets baked in before the first line of orchestration code is written—and most teams discover it at the worst possible moment.

Originally published on Medium — read it there for the full member experience.

The Multi-Agent Math Nobody Runs Before Shipping

If an AI agent achieves 85% accuracy per action, a 10-step workflow succeeds about 20% of the time. This isn't a theoretical concern—it's compound probability, and it applies to every multi-agent pipeline regardless of how good the underlying models are.

The research backs this up hard. LangChain's State of Agent Engineering report from late 2025 found that quality remains the #1 blocker for teams trying to move agentic systems from pilot to production. Not infrastructure. Not cost. Not integration complexity. Quality—specifically, the quality degradation that emerges not from any single agent but from the system behavior of agents chained together.

A January 2026 analysis in Towards Data Science documented what researchers called the 17x Error Trap: independent multi-agent systems—agents running in parallel without coordination—amplified errors by 17.2 times. Compared to 4.4x in systems with a proper orchestrator. That's not a small difference. That's the difference between a demo and a disaster.

The research from Google DeepMind published in late 2025, "Towards a Science of Scaling Agent Systems," ran controlled experiments across multiple coordination topologies and found something architects keep rediscovering the hard way: accuracy gains begin to saturate, and then decline, as agent count increases beyond four. They called it the coordination tax. Every handoff costs you a model call. Every model call introduces variance. Past four agents, you're paying compounding interest on decisions you already made.

This isn't a new problem. It's the distributed systems CAP theorem equivalent for AI: you can have accuracy, speed, or scalability—but chaining independent agents together means you're actively trading the first one away without realizing it.

Why "Bag of Agents" Became 2025's Most Expensive Architecture Mistake

Arjun's team at a Hyderabad SaaS company spent eight months building what they called their "AI workforce": twelve agents, each fine-tuned for a narrow task. Customer intent classifier. Ticket prioritizer. Knowledge base retriever. Response drafter. Sentiment analyzer. Escalation decision maker. And six more beyond that.

The reasoning was logical. They'd seen the microservices playbook work. This felt like the AI equivalent—composable, replaceable, modular. Each agent small enough to understand, test, and swap out. What's not to like?

Turns out: the space between agents.

What they hadn't read was the MAST study—Multi-Agent Systems Failure Taxonomy, published March 2025, which analyzed 1,642 execution traces across seven open-source frameworks. The finding that should have stopped them: coordination failures, not model failures, cause the majority of multi-agent system breakdowns. The agents aren't the problem. The space between agents is.

The "bag of agents" pattern—independent specialists, loosely coupled, each doing their job—sounds elegant until you realize errors propagate without any mechanism to catch them. Agent two receives bad output from agent one and doesn't know it's bad. Agent three inherits that corruption and compounds it. By agent seven, you're not debugging an AI system. You're debugging an error amplification machine.

Arjun's system hit a coordination overhead problem that no one on the team had mapped. Every agent handoff required a serialization step—packaging context, re-injecting into the next prompt, managing memory state. In LangChain, their framework of choice, those abstractions added over a second of latency per API call. Twelve agents. Potentially twelve seconds of framework overhead per request, before a single real computation happened.

💡 The problem isn't the agents. The problem is the topology. A bag of agents is not an architecture—it's a liability.

Deloitte's Tech Trends 2026 report found only 11% of organizations have agentic AI in production. The rest are stuck in pilot. Gartner predicts 40% of agentic AI projects will be scrapped by 2027—not because the models failed, but because organizations can't operationalize them. What sounds like an adoption problem is actually an architecture problem that nobody wanted to name.

What the Architects Who Got It Right Actually Did

Priya, a senior ML architect at a Bengaluru fintech, hit this exact wall with her loan underwriting pipeline. She didn't rebuild from scratch—she kept the specialized agents—but she restructured the topology entirely over two weeks. Here's what actually changed:

She added a critic agent. After each specialist completed its task, a lightweight validation agent checked the output before passing it downstream. Not perfect—the critic itself could fail—but it caught the class of errors that were cascading: malformed JSON, out-of-range values, missing required fields. Her end-to-end accuracy went from 48% to 74% in one week.

She moved to a centralized orchestrator pattern. Instead of a linear chain where agent N feeds directly into agent N+1, all outputs routed through a central controller that maintained global state and could invoke retry logic. Centralized topologies limit error amplification to 4.4x versus 17.2x in decentralized systems. The orchestrator cost in compute. It was worth every GPU cent.

She cut three agents. The regulatory checker, the summary generator, and the approval router were merged into a single agent with a carefully structured prompt and a constrained output schema. Reducing from seven to four agents—below that four-agent coordination tax threshold—brought her latency down by 40% and her reliability up by another 15 percentage points.

She implemented Langfuse for tracing. Not LangSmith, not custom logging. Priya needed to see exactly where in the chain failures were occurring, at what rate, with what context. Without tracing infrastructure, you're not debugging. You're guessing.

Real teams building reliable agentic systems in 2026 are making similar calls. At Amazon, the lessons from running agentic systems at scale involve building evaluation frameworks that catch errors at the agent boundary—not just at the system output. Every boundary is a failure surface.

None of this is glamorous. None of it looks good in a demo. But demos don't run at 3am.

The Mental Model Shift Nobody Teaches in Courses

Here's what Chip Huyen gets right in AI Engineering (O'Reilly, 2025) that most agentic AI tutorials miss: evaluation is the core engineering discipline, not agent design. If you haven't built evaluation into every handoff point—if you don't know what good looks like at each agent boundary—you don't have a system. You have a blackbox that occasionally produces outputs you hope are correct.

The mental model shift that the successful 11% seem to have made is this: design for failure, not for capability. The question isn't "what can this agent do?" The question is "when this agent fails—and it will—how does the system contain it?"

This isn't defensive programming. It's engineering reality. A 90% accurate agent failing 10% of the time, compounded across seven steps, gives you a coin flip. The same 90% accurate agent inside a system with checkpointing, retry logic, and a critic step gives you something you can actually deploy.

Arjun's team eventually did the same—after six months of trying to make twelve agents work, they rebuilt with four, added mandatory output validation between each step, and put in a circuit breaker pattern straight from distributed systems: if an agent fails three times on the same input, route to human review. Boring architecture. Runs in production without incidents.

Priya's loan pipeline now processes at 94% end-to-end accuracy. Not because she added more agents. Because she stopped treating agent count as a proxy for system capability.

Only 11% of organizations have multi-agent systems in production right now. That gap isn't about model quality—GPT-4o, Gemini 2.0, Claude 3.7 are more than capable. It's about whether the team that designed the system ever sat down and multiplied the error rates together before they shipped it.

Most didn't. I know because I've been in those architecture reviews.

The teams actually shipping aren't building more agents. They're building fewer, with harder boundaries between them. They're treating every agent handoff as a failure surface to be defended, not a connection to be celebrated. They're asking "how does this fail?" before "what can this do?"

The math is unforgiving. Your system's end-to-end reliability is not the average of your agents—it's the product.

Pull up your current multi-agent architecture. Count the agents in your critical path. Raise your per-agent accuracy to that power. What does the number tell you?