Why Enterprise AI Coding Agents Fail Without a Data Graph

Providing a complete enterprise code and data graph is the single most critical factor determining whether AI coding agents succeed in corporate environments. Smarter models and refined harnesses improve local execution, but they cannot navigate distributed enterprise architectures without visibility. Without a structured, cross-system map of how code and data actually interact, AI agents produce hallucinatory dependencies or fail to deliver usable outputs. Access to that graph, not model quality, is the primary bottleneck to scaling enterprise AI.

We measured this effect by running AI agents across real-world enterprise engineering tasks, holding the underlying LLM, harness, prompt structure, and execution budget strictly constant. We benchmarked three separate setups independently: plain local repository access, Language Server Protocol (LSP) symbol indexing, and Foundational Context supplying the unified enterprise code and data graph via MCP. The results revealed a drastic divide. Access to deep code and data graphs, built through direct source code analysis rather than warehouse metadata alone, was the sole variable that transformed agents from unreliable text matchers into accurate, enterprise-ready automation tools.

Throughout this piece, data graph (sometimes called an application data graph) means a map of how code and data actually connect across an enterprise: which service reads a table, which job writes it, and which downstream report depends on it, traced back to the source that created each connection. That is what this benchmark tested.

Benchmark Results: Measuring Agent Accuracy Across Enterprise Workloads

Benchmark Case 1: Mainframe Dataset Deletion and Certification Safety

The Task: A platform team plans to clean up eleven mainframe datasets. Before deleting any asset, the team must prove that no downstream service, batch process, program, or configuration binding touches it anywhere across the company. The agent must process the list item by item and issue a definitive cleanup certification.

Each configuration was evaluated across three independent runs at maximum model reasoning depth (claude-opus-5@high):

Both the standard AI agent and the LSP-equipped agent failed to deliver a single usable output. Rather than hallucinating risky approvals, both setups repeatedly declined the task. This silence highlights a core limitation of local file search: an agent cannot verify if a missing reference is truly unused or simply hidden inside a repo, job scheduler, or database schema it cannot access.

Even when we provided access to the company's entire codebase on local disk, ungrounded agents still declined every trial. In enterprise environments, dataset identifiers are often constructed dynamically across execution boundaries (see Appendix A), rendering static text search completely ineffective.

When equipped with Foundational Context over MCP, the agent delivered actionable answers in 100% of trials. It correctly identified dead assets while maintaining a 100% precision rate (zero false positives). In enterprise data infrastructure, a single false positive means authorizing the deletion of live data. That mistake tends to surface days later as a critical production incident.

Benchmark Case 2: Cross-System Impact Analysis

The Task: Ahead of a database schema change, build a complete inventory of every code asset, across COBOL mainframe jobs, Spring Boot services, and raw JDBC queries, that interacts with the ACCOUNT table. The challenge goes beyond tracking passive data flows. It requires verifying active code execution paths and schema access across language boundaries.

Each configuration was evaluated across three runs using claude-opus-5@high:

(Note: The Language Server Protocol (LSP) test was omitted here because the target codebase consisted of mainframe COBOL, leaving the attached Java language server with nothing to index. This underscores that single-language symbol tools cannot cross polyglot execution boundaries.)

While the standalone AI agent located most target references, it generated three false positive dependencies per response. In enterprise change management, false positives undermine trust. When an AI agent falsely claims a service will break, engineers waste hours reviewing assets that are not impacted.

Ungrounded agents rely on string matching, which confuses generic file names with active database queries. Foundational Context grounds the agent with the full enterprise code and data graph, tracing cross-platform dependencies over MCP to ensure only verified, execution-path dependencies are returned.

Why Smarter Reasoning Cannot Replace a Data Graph

Enterprise AI failure is rarely a story about model quality. Across our benchmark evaluations, the exact same model, operating in an identical agent harness with the same prompt and turn budget, delivered authoritative answers in one configuration and complete failure in the others. The variable was not reasoning power. It was whether the agent could see across systems at all.

Large language models excel at processing text placed directly in front of them. However, they are far weaker at the core operation enterprise work constantly demands: tracing a symbol, table, or critical data element through multi-hop dependencies across disparate languages, repositories, and unlinked established systems. This is not a reading comprehension task. It is graph traversal over a multi-system graph that standard tools have never built, attempted instead via text search, a primitive approach that lacks any concept of identity.

Six Enterprise Use Cases That Depend on the Same Code and Data Graph

Navigating dependencies across multi-repo architectures is an enterprise-wide challenge that goes well beyond code editing. Lacking a complete enterprise code and data graph, AI agents encounter the exact same structural limitations across core operational disciplines:

1. Coding and Change Planning: Modifying an API or a schema requires identifying every downstream service, batch job, report, and test suite across systems the single developer or agent is not even aware of.
2. Incident Response: When a nightly job fails, engineers must rapidly map its multi-hop downstream dependencies across code and data pipelines. Without cross-system visibility, determining which operational services, reports, and executive dashboards are serving stale data, and alerting their owners before business hours begin, requires hours of manual auditing.
3. Compliance and Audit: Demonstrating regulatory compliance, such as BCBS 239 for risk data aggregation or GDPR for PII handling, requires generating verifiable, end-to-end evidence of every pipeline, service, and database process authorized to mutate or derive from governed fields.
4. Security and Vulnerability Review: Mitigating security risks requires tracing execution paths backward from critical database queries or API handlers to their original sources, ensuring untrusted inputs, such as unsanitized user form submissions or external third-party API payloads, are properly validated before the application acts upon them.
5. Migration and Modernization: Successfully modernizing established systems and migrating core workloads requires verifying that no untracked callers or hidden dependencies remain. Without a complete enterprise code and data graph across all systems, turning off or migrating established platforms invites silent, costly production outages.
6. Documentation and Architecture: Static design diagrams and outdated documentation fail to capture how complex systems interact in production. By unifying code and data dependencies into a single data graph, technology teams eliminate the bottleneck of chasing identifiers across disconnected systems, ensuring an accurate, real-time picture of operational architecture.

While these six enterprise workloads serve different stakeholders with distinct requirements, they all depend on the exact same graph traversal. Unifying code and data into a single enterprise data graph resolves this bottleneck across the entire technology organization.

Why Enterprise Complexity Breaks AI Agents Without a Data Graph

At smaller scales, this blindness does not exist, which explains why enterprise AI pilots look convincing in sandbox environments before failing in production. A small company's codebase easily fits inside a single context window or an engineer's head, allowing an agent to read the entire workspace without needing an architectural map.

In contrast, large enterprises operate across thousands of fragmented code repositories and heterogeneous technology stacks where established mainframes, cloud microservices, and third-party platforms execute along the exact same critical transaction path. AI agents fail at enterprise scale not because queries demand harder reasoning, but because multi-repository fragmentation prevents any single session from seeing across system boundaries, leaving standard text search completely blind to the dependencies it cannot see.

The Limits of Symbol Indexing: Why LSPs Cannot Resolve Cross-System Dependencies

The standard engineering approach to improving code visibility is attaching a symbol server powered by the Language Server Protocol (LSP), the core technology IDEs use to resolve definitions and references. While LSPs replace text-pattern matching with true semantic parsing, testing shows this approach fails to close the enterprise knowledge gap for two structural reasons:

• Non-Symbol Infrastructure Bindings: Enterprise connections rarely exist as code-level symbol references. When a Java microservice and an established COBOL program execute against the same database table, no import, shared interface, or direct function call connects them. They align solely on a string table name within a database catalog, a boundary a standard symbol query cannot cross.
• Repository Isolation: A Language Server Protocol (LSP) instance indexes symbols exclusively within a single workspace boundary. Determining which external service modifies a database table requires tracing calls across repositories owned by separate teams and written in different languages. Because no code-level import exists across these boundaries, an LSP has no reference to follow.

Furthermore, equipping an agent with an LSP can actively degrade performance. By indexing localized candidate symbols without cross-system knowledge, the language server surfaces disconnected references that increase false positives, yielding higher error rates during compliance and audit reviews.

Why Smarter Models Cannot Infer Missing Knowledge

The common assumption is that this limitation is temporary, and that waiting a few model releases or upgrading agent harnesses will naturally solve the problem. Benchmark data explicitly refutes this hope. Across multiple model tiers, from standard configurations to the strongest reasoning setups, raw capability failed to close the visibility gap. In fact, advanced reasoning models operating without a data graph to ground them produced zero usable answers. Sharper reasoning simply made the model better at recognizing it lacked the evidence required to certify a decision.

A smarter model reasons better about the information placed directly in front of it, but it cannot infer cross-system evidence that was never provided. Swapping agent harnesses yields the exact same bottleneck: whether evaluating abandoned dashboards, data pipelines, or cross-platform permissions, an AI model cannot reason over dependencies that were never collected.

Benchmark Methodology: Environment, Tasks, and Controls

Benchmarking non-deterministic AI systems is notoriously easy to skew and misinterpret. To ensure our results reflect genuine capability rather than artificial advantage, this section details the environment design, task suite, and isolation controls engineered for data integrity. Appendix A covers the complex code patterns tested, while Appendix B contains the granular, per-task metrics.

Test Environment: Simulating Enterprise Architecture

To simulate realistic enterprise architecture, we needed an environment characterized by multiple languages, repositories, and ownership boundaries, alongside a classic mainframe-to-distributed split connected solely by shared data. We adapted IBM's public CICS Banking Sample Application into four distinct repositories:

The Java and COBOL applications share DB2 tables, but lack direct cross-imports or shared symbol references. The database itself is the sole data bridge. This structural dynamic reflects a core enterprise challenge. By keeping the codebase deliberately compact, four repositories across two platform families and a few hundred files, we established an optimal baseline, giving ungrounded tools their best possible chance to succeed.

Test Tasks: Simulating Everyday Engineering Work

Rather than focusing on isolated code generation, our benchmark evaluates tasks that enterprise engineers tackle daily:

• Cleanup and Cost Reduction: Identify which of 11 datasets can be safely decommissioned, where a single remaining consumer anywhere in the stack blocks deletion.
• Incident Handling: Map the blast radius of table changes by listing every program, class, or module enterprise-wide that reads or writes to the ACCOUNT table.
• Compliance Review: Generate audit trails for regulated records by identifying every pipeline and code path that mutates (INSERT, UPDATE, DELETE) data read by a targeted Java class.
• Migration Planning: Locate every program writing to a specific output file whose path is dynamically generated at runtime.

To prevent self-consistency bias, ground truth was manually authored with full source citations by software engineers and cross-reviewed by mainframe and Java domain experts, never extracted from the lineage graph under test.

All runs shared an identical agent harness, underlying LLM, prompt structure, and turn budget. The benchmark variants differed strictly in what graph of code and data the agent could see:

Securing the Benchmark: Mitigating Pre-Training Recall and Runtime Escapes

Evaluating AI agents on public codebases risks measuring training data recall rather than real-time code analysis. To ensure our metrics reflect live graph traversal rather than parametric memory or environment exploits, we engineered three strict safeguards:

• Identifier Obfuscation: We systematically renamed Java namespaces and mainframe dataset identifiers while preserving the underlying codebase structure. An identifier recalled from pre-training data no longer maps to the source on disk.
• Zero-Repository Control: To quantify memory bias, we executed the entire task suite with all repositories removed. Models scored zero across every task, proving that answers require active code analysis rather than memorization.
• Containerized Sandboxing: Early runs showed unconstrained agents attempting to read host files and locate answer keys. Every test now executes within an isolated container restricted exclusively to its assigned workspace. Any run that fails to prove strict containment is excluded from our published figures.

From Benchmark to Production: Why the Graph Advantage Multiplies at Scale

While our benchmark demonstrates a clear advantage, the real-world gap in production enterprise environments is significantly larger. A controlled testbed of four small repositories cannot fully capture the architectural friction of live enterprise stacks. Two primary mechanisms account for this divergence, both heavily penalizing ungrounded agents:

• Heterogeneous Platform Boundaries: Real enterprise environments span dozens of platform types beyond standard git repositories, including mainframes, cloud workloads, job schedulers, messaging queues, data warehouses, stored procedures, and BI dashboards. As data moves across these boundaries, identifiers morph and lineage trails break. Foundational's enterprise code and data graph maps across these disparate systems directly, whereas traditional tools halt at the repository edge.
• Search Degradation at Scale: Unindexed text and symbol search degrade rapidly as codebase volume expands, flooding agents with superficial, false-positive matches. An agent backed by an enterprise code and data graph traverses relationships directly across system boundaries without relying on keyword matching.

Because small benchmarks lack this degree of platform sprawl, they represent a conservative baseline. Reliable impact analysis, cross-system navigation, and code generation require Foundational's enterprise code and data graph, and within our enterprise customers, we see even bigger performance improvements than what this benchmark reflects.

What the Benchmark Proves

An enterprise AI agent without system knowledge is simply a reasoner operating in the dark. Our benchmark proves that model capability cannot offset a missing data graph. Without one, even top-tier models produce hallucinated dependencies or outright failures.

Scaling AI beyond single-repository pilots requires recognizing three core realities:

• Scale Breaks Ungrounded AI: Pilots succeed because single codebases fit in working memory. Production environments span cross-platform dependencies that require automated mapping.
• The Graph Is Infrastructure: Agent frameworks and LLM models are ephemeral. A vendor-neutral data graph mapping your code, data, and cross-system dependencies serves as durable, long-term infrastructure.
• Traceability Drives Reliability: End-to-end provenance across established and cloud stacks prevents accuracy collapse on mission-critical workflows.

Foundational, a data and AI governance platform built on source code analysis, delivers this persistent enterprise data graph. Connected via MCP, Foundational provides the unified code and data graphs necessary to power every AI deployment across your entire technology stack. See how Foundational's cross-platform lineage builds this graph from source, and read the companion piece on why AI agents cannot reason accurately without source-code-derived context for the argument behind these results.

Frequently Asked Questions

What is Foundational Context, and how is it different from a metadata catalog or an LSP?
Foundational Context is the enterprise code and data graph this benchmark tested, delivered to an AI agent over MCP. Unlike a metadata catalog, which records where data lives after the fact, or an LSP, which resolves symbols only within one workspace, it traces how code and data actually connect across repositories, languages, and platforms, built through direct source code analysis rather than reconstructed metadata.

Which AI models were used in this benchmark, and does the result hold across models?
Each task ran three times at two model configurations, claude-opus-5 at high reasoning effort and the cheaper claude-sonnet-5 at default effort. Both showed the same pattern: without a data graph to ground them, usable answers were rare or absent regardless of model strength, and adding Foundational Context closed the gap for both. We have since run the same comparison with other state-of-the-art agents from other vendors, with similar results.

How was the benchmark protected from AI models simply memorizing the answer?
We renamed Java namespaces and mainframe dataset identifiers so pre-training recall could not map to the test corpus, ran the full task suite with all repositories removed to confirm every model scored zero without code access, and sandboxed every run in an isolated container so an agent could not read host files to find the answer key directly.

What does this mean for a team already using an AI coding agent today?
An agent with only local repository access or an LSP can already help inside a single file or service. This benchmark shows that the same agent, asked a cross-system enterprise question such as what depends on this table or is this dataset safe to delete, needs a code and data graph to answer reliably. Without one, agents tend to decline the task rather than guess, which is safer than a wrong answer but still blocks the work.

Does this depend on which AI coding harness or agent framework a team uses?
No. The benchmark held the harness, prompt structure, and model constant across every run and changed only what graph of code and data the agent could see. The gap between ungrounded and grounded runs held across both model configurations we tested, so the result is about what the agent can see, not which specific coding harness wraps it. We would expect the same gap with any harness that lacks a code and data graph.

Appendix A: Why AI Agents Need Graphs, Not Grep

Enterprise dependencies are rarely written down in the place you would search for them. They sit behind indirection: a name assembled at run time, a value injected from configuration, a binding that exists only inside a database. That is why text search (grep) and language servers (LSP) fall short when the job is building an enterprise data graph of code and data, and the three patterns below are the shapes that defeat them most often. All three are drawn from this benchmark's corpus, so the examples are COBOL and Java, but none of the patterns is specific to either language. We see similar shapes in every enterprise environment we work in, across current and established stacks alike.

A fourth pattern follows them, and it was deliberately left out of the benchmark. Its evidence is operational rather than written in code, so the corpus contains nothing to score against, and an agent that can only read code has no route to the answer however much code it is given. We include it because it is a large share of what enterprises actually ask.

1. Runtime-assembled identifiers

Mainframe JCL constructs physical dataset names dynamically using symbolic parameters. A shared procedure establishes default templates, and at execution time, a production job overrides those parameters at the call site. Program SETLEXT1 actually writes to BANK.PROD.SETTLE.CLEARING, a string that literally exists in zero files on disk. Text search will never find it, because there is nothing to find. Worse, a near-identical sibling job overrides FILE=CLRWORK, so one of five look-alike jobs does not touch the target dataset at all. An agent that pattern-matches on the job names gets a false positive to go with its missing dependency.

Resolving the real dependency requires performing the symbol substitution across job boundaries, exactly as the system does at run time. The same shape appears in a milder form throughout application code, wherever a call target or a resource name is held in a variable rather than written literally, and each layer of indirection is another hop an agent has to chase correctly before it has an answer.

2. Names that live in configuration, not code

The teller service writes an audit row on every transaction, but the table being written is never named in the code. It is named once, in a deployment configuration file. Now ask the question an auditor asks: what writes AUDIT_TRAIL? Searching the whole company codebase for that string returns exactly one hit, the configuration line, and zero Java files. The class that actually writes it, TellerAudit, does not contain the string at all. An agent has to connect a configuration key to an injected field to a concatenated SQL statement before it can say anything, and that is one deployment among many: the same compiled class writes whichever table the environment happens to configure.

3. Cross-system bindings that exist only in the database

Two codebases share a DB2 table. A COBOL program updates it through embedded SQL. In a separate repository owned by another team, a Java service inserts into the same table through inline JDBC. These statements modify the same physical rows while sharing zero imports, build artifacts, or repository links. Text search does surface both, and that is the trap: it surfaces them alongside every other file that mentions the word, with no way to tell which hits are the same physical table, which are a similarly named table in another schema, and which are reads versus writes. In this small corpus the token appears across dozens of files. In a real environment it is thousands, and the agent has no basis for ranking them.

No language server closes this gap either, because there is no reference to resolve. The binding exists only at the database catalog level, and recovering it requires parsing the SQL out of both languages and mapping the results onto a single schema graph.

4. Out-of-band operational telemetry

This pattern was not part of the benchmark, for the reason given above: the corpus is source code and this evidence is not in code at all, so there was nothing to score and no way for an ungrounded agent to succeed. It belongs here anyway, because these are among the most common questions enterprises ask:

• Which dashboards are genuinely unused and safe to clean up?
• Which tables are accessed by external partners through a direct data share?
• Which batch pipelines have not executed since the last quarterly close?

Answering these requires joining static code graphs with operational telemetry: warehouse query logs, job scheduler execution histories, and BI usage metrics. Something must collect and pre-compute this state before an agent can consult it mid-task. No amount of repository access substitutes for runtime telemetry.

Appendix B: Example Benchmark Results

This appendix shows the per-task results for the two examples above, as a sample of a broader suite of enterprise tasks run the same way. The figures are from the Claude Code benchmark. We have run the same comparison with other state-of-the-art agents from other vendors, and the results are similar.

Each benchmark test ran each task three times, at two models: claude-opus-5@high, the strongest configuration available to us, and the cheaper claude-sonnet-5@default. The suffix is the reasoning effort each model ran at, so the two differ in effort setting as well as in model.

Example 1: Which datasets are safe to clean up

"Usable answers" is the share of runs that produced a decision at all. "Required names found" is the share of the genuinely safe datasets the test cleared. "Names that were correct" is the share of what it named that was right, and reads N/A where a test named nothing, since there are no claims to score.

At claude-opus-5@high:

At claude-sonnet-5@default:

Example 2: What breaks if we change this table

The consumer enumeration for the ACCOUNT table. Every run produced an answer here, so the columns are coverage and accuracy rather than whether an answer arrived.

At claude-opus-5@high:

At claude-sonnet-5@default:

Note that the ungrounded agent scores better here on the cheaper model than on the stronger one, which looks backwards until you read the last column. Both models found the same required names. The stronger model searched harder and returned far more candidates, and the extra candidates it could not actually connect to the table became wrong answers. Without a way to verify a connection, more capability spends itself on producing more plausible material rather than more correct material. This is a single task and we would not generalize from it, but it is the same effect the language server produced when it surfaced additional candidates it could not link to the question.

Glossary

• Required names found (recall): the share of the known-correct answers the agent named.
• Names that were correct (precision): the share of what the agent named that was right. Undefined, shown as N/A, when the agent named nothing at all.
• Critical data element: a field the business tracks under governance, whose consumers must be known before it changes.
• MCP (Model Context Protocol): the open standard by which an agent calls external tools; here, the query surface for Foundational Context.
• LSP (Language Server Protocol): the technology an IDE uses for per-language symbol resolution and references.

Ready to see what your own agents are missing? Talk to Foundational about giving your AI agents the enterprise code and data graph this benchmark shows they need.