01 — The Signal 02 — Executive Summary 03 — Gaps in Traditional DE 04 — Competencies & Tooling 05 — Project Archetype Map 06 — From the Field 07 — Where AI Attaches 08 — Target Architecture 09 — Concept-by-Concept Redesign 10 — Migration Roadmap References & Reading

Point of View · July 2026

Data Engineering,
rebuilt by AI

RR Ravi Rali · Founder, R2 Digital
Business Transformation - AI Strategy - Solution Delivery

For twenty years data engineering optimised one thing: moving rows reliably from systems of record to dashboards. That job is now the smaller half. AI has changed both what we build pipelines with and what we build them for — and the discipline is being rebuilt from the connector up.

Read the thesis

53%of data-team capacity spent maintaining and troubleshooting existing pipelinesFivetran / dbt Labs 2026, secondary analysis

7%of companies qualify as data "reinventors" — genuinely ready to scale advanced AIAccenture, AI-Ready Data

72 / 24prioritise AI-assisted coding vs. AI-assisted pipeline managementdbt Labs, State of AE 2026

30%more market share for vendors that re-architect for agentic, machine-to-machine data access by 2027Gartner

The AI-native data platformlive

01 — The Signal

What the last six months actually changed

Not opinion — the published record. Between February and July 2026 the analyst houses and the platform vendors converged on the same conclusion from opposite directions: the data platform is no longer being designed for human consumption, and the pipeline is no longer being designed to be written by hand.

SIGNAL 01

Agents became the primary data consumer

Gartner now frames "agent-ready data" as a market-share event: providers that re-architect data management systems for agentic, machine-to-machine workflows can capture up to 30% more market share by 2027. At the 2026 D&A Summit the firm separated a context layer — ontology, schema, metrics, policy-as-code — from an intelligence layer above it. That is an architecture instruction, not a trend note.

Implication The consumer of your pipeline no longer reads a dashboard. It reads metadata, and it acts.

SIGNAL 02

The platforms shipped agentic data engineering

Databricks positioned Lakeflow (June 2026) as "agentic data engineering" — agents that not only build but operate pipelines, with a background ZeroOps agent doing failure detection, root-cause analysis and sandbox-validated fixes. Snowflake's June 2026 release put smart pipelines, natively-hosted dbt projects, column-level lineage in Horizon and a specialised coding agent into the same workflow. Microsoft anchored Fabric on a reframed context layer with agent-authorable Data Factory skills.

Implication Agentic DE is now a product category with GA features, not a lab demo.

SIGNAL 03

Acceleration outpaced trust

dbt Labs' State of Analytics Engineering 2026 (363 respondents) found 72% prioritise AI-assisted coding but only 24% prioritise AI-assisted pipeline management — testing, observability, quality. Meanwhile 71% name incorrect or hallucinated outputs reaching stakeholders as a top concern, and trust as an organisational objective jumped from 66% to 83% year over year.

Implication Teams bought the accelerator and skipped the brakes. This is the defining 2026 risk.

SIGNAL 04

The foundation is the constraint, not the model

Accenture's AI-ready data research finds only 7% of companies are "data reinventors" while 64% have already moved past pilots — and puts an estimated 4.5 point EBIT margin advantage on closing that gap. Deloitte and McKinsey land in the same place: legacy data architecture cannot power real-time autonomous AI, and modernisation has to produce a living, organisation-wide backbone rather than another warehouse.

Implication The bottleneck moved from model access to data readiness — permanently.

SIGNAL 05

Unstructured data entered the pipeline

Forrester's Q1 2026 Data Quality Wave notes buyers now prioritise platforms that can profile and evaluate logs, images and documents — previously underexplored data types — and that static validation is losing to real-time monitoring and diagnostics. Data quality, observability, governance, privacy, metadata and lineage are consolidating into single platforms.

Implication "Data engineering" now includes content engineering. Most teams are not staffed for it.

SIGNAL 06

The economics stopped working

The 2026 practitioner surveys put the maintenance tax between roughly a quarter and half of capacity depending on how it is measured — a 1,101-respondent survey reports 26% of time simply fighting fires, with modelling pain (90%) driven by pressure to move fast (59%) and unclear ownership (51%). Secondary analysis of the Fivetran and dbt 2026 datasets converges on ~53% of engineering capacity going to maintaining and troubleshooting what already exists. Warehouse and compute spend rose for 57% of teams; team budgets rose for 36%.

Implication More headcount cannot close this gap. Only a different operating model can.

On the evidence Figures are attributed to their source and reflect what was published between February and July 2026. Vendor-published benchmarks (latency, cost, token efficiency) are marketing artefacts measured on vendor-chosen workloads — directionally useful, not procurement-grade. The practitioner survey figures come from self-selecting samples of a few hundred to ~1,100 respondents. Where two sources disagree on the size of the maintenance tax, both ranges are shown rather than the more dramatic one.

02 — Executive Summary

Six inversions that redefine the discipline

AI does not make data engineering faster. It inverts six of its founding assumptions. Each inversion removes a constraint that shaped every architecture built between 2010 and 2024 — which is why incremental tuning of the existing stack does not get you there.

Traditional DEAI-Native DE

Pipelines you writeImperative code, hand-tuned, one engineer deep per system

→

Pipelines you specifyDeclarative intent plus contracts; agents generate, test and deploy the implementation

Rules anticipated in advanceEvery schema, mapping and validation encoded before the first record lands

→

Behaviour learned and adaptedDrift detected, classified and resolved in-flight; rules proposed from observed data, approved by humans

Tables as the deliverableMeaning lives in Confluence, tickets and the heads of three people

→

Context as the deliverableSemantic layer, ontology and policy are executable artefacts consumed by machines

Structured data onlyRows and columns; documents, images and audio routed around the platform

→

Multimodal by defaultExtraction, classification and embedding are pipeline stages like any other

Batch-anchored freshnessNightly windows; real-time bolted on as an exception path

→

Continuous by defaultStreaming ingest and incremental compute as the primary path; batch as an optimisation

Humans in every loopEngineers on call for pipeline failures; stewards in an unbounded queue

→

Humans on the loopAgents handle the routine; people set intent, adjudicate the ambiguous, own the outcome

What leadership should take away

This is a capacity reallocation, not a cost-cutting exercise. Roughly half of data-engineering capacity is currently defensive. AI does not remove that work — it moves most of it below the human threshold, and the value comes entirely from what the freed capacity is pointed at next.

What the evidence warns

The 2026 data says teams adopted AI for authoring far faster than for verification. Generating pipelines four times faster without a matching lift in testing, lineage and observability produces defects four times faster too. Sequence the trust plane first.

What does not change

Ownership, stewardship, definitions, controls, authorised sources and change management. Every autonomous capability described on this page is only defensible because a governance framework sits under it. Modernise the engine; keep the governance.

03 — Gaps in Traditional DE

Nine structural gaps, not nine bugs

These are not implementation failures. They are the designed-in consequences of building data platforms for human analysts, batch windows and known schemas. Each one is now a direct blocker to AI at enterprise scale.

GAP 01 Severe

The maintenance tax

Roughly half of capacity goes to keeping existing pipelines alive; schema drift alone is commonly the single largest maintenance category. Every new pipeline adds permanent liability, so throughput falls as the estate grows. Organisations past ~200 active pipelines report the steepest curve.

GAP 02 Severe

Brittleness by design

Imperative pipelines encode assumptions about source structure that the source never agreed to. A renamed column, a changed enum, a new nested field — and the failure surfaces downstream, hours later, as a wrong number rather than an error.

GAP 03 High

Quality as an afterthought

Tests are written for the failures someone already imagined. Coverage clusters on primary keys and null checks; semantic errors — a plausible-looking but wrong value — pass every assertion and reach the stakeholder intact. 71% of practitioners now name exactly this as their top concern.

GAP 04 High

Metadata that documents instead of executing

Catalogs describe assets to humans who mostly do not read them. Lineage is partial and stale. Definitions live in a spreadsheet. None of it is machine-consumable at runtime, so an agent asking "what is net revenue and may I see it?" gets no reliable answer.

GAP 05 Severe

Blind to most enterprise information

Contracts, tickets, emails, engineering drawings, call transcripts, policy documents — the majority of what an enterprise knows has never entered the data platform. Traditional DE has no primitive for it, so it was scoped out. AI use cases start there.

GAP 06 High

Latency ceilings inherited from batch

Nightly reload architectures cap freshness at 24 hours no matter how much real-time integration is layered on top. Agents that act — reprice, route, approve, escalate — cannot operate on yesterday's state, and the usual workaround is per-application caches that quietly create new copies of the truth.

GAP 07 High

Cost that grows faster than value

Consumption pricing plus unreviewed pipelines plus full refreshes equals compute spend rising for 57% of teams while budgets rise for 36%. Very few organisations can attribute warehouse spend to a business consumer, so optimisation becomes across-the-board throttling.

GAP 08 High

Organisational gravity

The 2026 practitioner data is blunt: organisational blockers — unclear ownership, absent leadership direction, poor upstream requirements — outweigh technical debt as the top constraint. No platform migration survives contact with an operating model that has no owner for the data.

GAP 09 High

No context layer

Between physical tables and business questions sits a translation layer that exists only in analysts' heads. Without it, both humans and agents reconstruct meaning per query — inconsistently. This is the single most common reason AI pilots pass a demo and fail production.

04 — Competencies & Tooling

The nine branches of data engineering

Data engineering is not one skill. It is nine distinct competencies that most enterprises staff unevenly — heavy on ingestion and transformation, thin on semantics, observability and platform economics. Below: what each branch actually does, how organisations typically implement it, the tools in play, and where AI changes the work.

C1 · INGESTION & INTEGRATION

Getting data in

What it is: connectors, CDC, APIs, files, events — landing source data reliably with replay, idempotency and backfill.

How it's implemented: managed connectors for SaaS; log-based CDC for OLTP; custom API extraction for the long tail; an integration pattern catalogue (batch extract, pub/sub, request-reply, two-way sync) governing which pattern a source is allowed to use.

FivetranAirbyteDebeziumQlik ReplicateADFLakeflow ConnectSnowflake OpenflowAI: schema-drift agentsAI: connector synthesis

C2 · STORAGE & MODELING

Where data lives and how it's shaped

What it is: lakehouse/warehouse design, open table formats, partitioning, and the modelling discipline — dimensional, Data Vault, One Big Table, medallion.

How it's implemented: bronze/silver/gold zones over an open format; conformed dimensions for BI; Data Vault where source volatility and auditability dominate; increasingly a single storage tier serving BI, ML and retrieval.

SnowflakeDatabricksBigQueryRedshiftFabric / OneLakeApache IcebergDelta LakeAI: model generation from profiles

C3 · TRANSFORMATION & SEMANTICS

Turning raw into meaning

What it is: business logic, conformance, derivations, metric definitions — plus the semantic layer that makes those definitions singular.

How it's implemented: ELT in SQL/Python inside the warehouse, version-controlled with tests and CI; a metrics/semantic layer over the gold zone; historically a bespoke stored-procedure estate nobody wants to touch.

dbtSpark / PySparkSnowparkSQLMeshCubeAtScaleCoalesceAI: NL → tested modelsAI: legacy logic extraction

C4 · ORCHESTRATION

Making it run, in order, on time

What it is: dependency graphs, scheduling, retries, backfills, SLAs and the on-call rota that sits behind them.

How it's implemented: a DAG scheduler owning cross-system dependencies; data-aware triggers replacing cron where maturity allows; environment promotion via CI/CD. A fifth of teams still run with no orchestrator at all.

AirflowDagsterPrefectLakeflow JobsADFArgoControl-MAI: ZeroOps repair agents

C5 · QUALITY & OBSERVABILITY

Knowing it's right — and knowing first

What it is: validation, profiling, freshness/volume/schema monitoring, incident management, reconciliation to source.

How it's implemented: assertion tests in the transformation layer, an observability platform on top for anomaly detection and lineage-based triage, DQ scorecards for the business, and a severity/escalation model.

Great ExpectationsSodaMonte CarloAnomaloBigeyedbt testsIDQAI: learned expectationsAI: semantic validation

C6 · GOVERNANCE, METADATA & SECURITY

Trust, control and provenance

What it is: catalog, lineage, glossary, ownership, access control, masking, retention, sovereignty and audit.

How it's implemented: a catalog federated across platforms; RBAC/ABAC with tag-based policy; column masking and tokenisation for regulated attributes; a stewardship workflow with approval chains and impact analysis.

Unity CatalogSnowflake HorizonCollibraAlationPurviewOpenMetadataImmutaAI: auto-classificationAI: policy-as-code

C7 · STREAMING & REAL-TIME

Data in motion

What it is: event backbones, stream processing, exactly-once semantics, windowing, late-arrival handling, materialised views on streams.

How it's implemented: a Kafka-class log as the enterprise event spine; stream processing for enrichment and aggregation; a serving store for sub-second reads. Usually a separate team from batch — which is itself part of the problem.

KafkaConfluentFlinkKinesisPulsarMaterializeSpark Structured StreamingAI: streaming enrichment

C8 · AI & UNSTRUCTURED DATA ENGINEERING

The branch that barely existed in 2022

What it is: document parsing, chunking, embedding, vector indexing, retrieval evaluation, feature engineering and serving, prompt/context assembly, eval pipelines.

How it's implemented: a parsing and extraction tier over the document estate; embeddings written to a vector index governed alongside tables; a feature store for online/offline parity; retrieval quality measured with the same rigour as pipeline SLAs.

FeastTectonpgvectorPineconeLanceDBRayMLflowAI: extraction & classificationAI: eval harnesses

C9 · DATAOPS, PLATFORM & FINOPS

Running data engineering as a product

What it is: CI/CD, environments, IaC, testing strategy, release management, self-service enablement, cost attribution and unit economics.

How it's implemented: Git-based promotion with automated tests; infrastructure as code; a golden-path template so domain teams ship without a platform ticket; showback/chargeback by consumer.

GitHub ActionsTerraformdbt CIBackstageDatafoldSelect / KeeboAI: cost & query optimisationAI: golden-path scaffolding

The staffing reality In most enterprises C1–C4 are well staffed, C5–C6 are one person's side responsibility, C7 sits in a different org, C8 was assembled ad hoc for the first GenAI pilot, and C9 does not formally exist. AI compresses effort hardest in C1–C4 — exactly where capacity already exists — which is why the migration argument is fundamentally about redeploying people into C5, C6, C8 and C9, not about reducing headcount.

05 — Project Archetype Map

Which projects use which concepts — and where AI lands

Data engineering decisions are archetype-driven. A regulatory reporting build and a RAG assistant share a lakehouse and almost nothing else. This is the map we use in engagements to scope a platform against the portfolio of work it actually has to carry.

Project archetype	Core DE concepts	Typical stack	Dominant pain	Where AI changes the game
Enterprise DWH / BI modernisationLegacy EDW or ETL estate → cloud lakehouse	C2 modelling · C3 transformation · C4 orchestration · C9 platformDimensional models, medallion zones, SCD handling	Snowflake / Databricks / Fabric · dbt · Airflow · Power BI / Tableau	Thousands of undocumented legacy mappings; nobody knows what is still used Severe	Automated logic archaeology. Agents parse PowerCenter XML, SSIS DTSX and stored procedures, reconstruct business rules, generate equivalent dbt/Spark, and produce a usage-based decommission list.
Customer 360 / MDMGolden records across CRM, billing, service, web	C1 integration · C5 quality · C6 governance · C3 conformanceMatch/merge, survivorship, cross-reference, stewardship	Informatica MDM / Reltio / Profisee · IDQ · Collibra · CDC	Rule-tuning never converges; steward queue never empties Severe	Learned matching and survivorship. Embedding-based candidate generation with a trained classifier; steward decisions become labelled training data; active learning routes the most informative cases, not the least confident.
Real-time operational analyticsFraud, pricing, logistics, personalisation	C7 streaming · C1 CDC · C8 features · C5 observabilityEvent backbone, windowing, online/offline parity	Kafka / Confluent · Flink · Materialize · feature store · low-latency serving	Two codebases for the same logic; training/serving skew High	Unified declarative pipelines. One definition compiled to both streaming and batch execution; agents monitor skew and drift continuously rather than at retrain time.
Regulatory, risk & finance reportingBCBS 239, Solvency, IFRS 17, SOX, ESG	C6 governance · C5 reconciliation · C2 auditabilityLineage to source, versioning, controls evidence, sign-off	Data Vault · warehouse · Collibra / Purview · reconciliation frameworks	Proving lineage and control effectiveness to an auditor High	Evidence generation. Column-level lineage assembled automatically; controls narrative and variance explanations drafted from actual execution metadata — reviewed, not written, by humans.
IoT / telemetry / industrialSensors, equipment, connected product	C7 streaming · C2 time-series storage · C8 MLEdge buffering, downsampling, gap-filling, anomaly detection	MQTT / Kafka · time-series or lakehouse · Spark · edge runtimes	Volume vs. cost; sensor drift silently corrupts models High	Adaptive retention and detection. Models decide what to keep at full fidelity; anomaly detection replaces static thresholds that nobody recalibrates.
GenAI knowledge assistant / RAGPolicy, contract, support, engineering knowledge	C8 unstructured · C6 access control · C5 evalParsing, chunking, embedding, retrieval, permission-aware serving	Lakehouse + vector index · embedding models · orchestration framework · eval harness	Retrieval quality and permission leakage Severe	AI is the pipeline. Extraction, classification, enrichment and evaluation are model-executed stages — but they must inherit the same lineage, access policy and SLA discipline as tables.
Agentic AI applicationsAgents that act: approve, reprice, schedule, escalate	C3 semantics · C6 policy · C7 freshness · C5 observabilityContext layer, tool/API surface, guardrails, audit trail	Semantic layer + knowledge graph · MCP/tool servers · policy engine · event backbone	Agents lack business context and act on stale or over-permissioned data Severe	This archetype only exists if DE is rebuilt. The context layer, machine-readable policy and sub-minute freshness are prerequisites, not enhancements.
Data products & monetisationInternal marketplace, partner sharing, embedded analytics	C9 productisation · C6 contracts · C2 sharingSLAs, versioning, contracts, entitlement, usage metering	Data sharing / clean rooms · catalog · contract tooling · billing telemetry	No unit economics; consumers unknown High	Autonomous product ops. Usage, cost and SLA telemetry per product; agents draft contract changes and flag consumers affected before a breaking change ships.

↔ scroll the table horizontally

06 — From the Field

What a well-built traditional platform looks like — and what it tells us

The critique above is not theoretical. Below is an anonymised composite of enterprise data platforms R2 Digital has delivered — a multi-domain MDM and warehouse programme spanning customer, membership, certification and product data across roughly a dozen source systems. It was competently built and faithful to the vendor reference architecture. Its constraints are exactly the constraints of the era that produced it.

AS-BUILTLayered topology — Landing → Staging → Base Objects → ViewsMDM / DQ suite on managed RDS · Collibra governance

LandingSource-shaped tables · truncate & load daily · zero transformation, by design

StagingCleansing, DQ rules, reference lookups · 1-to-1 mappings

Hub / base objects3NF · daily incremental · tokenize → match → merge → best version of truth

Data services viewsService & extract layer for downstream consumption

Governance toolingCatalog, lineage and stewardship workflow

DQ & reportingScorecards, address standardisation, BI quality dashboards

▼

AS-BUILTEight enterprise integration patternsBatch · real-time · asynchronous

1 · Source extractsBatch daily — the workhorse

2 · Downstream writesBatch to warehouse & marts

3 · Publish outHub publishes to queue

4 · App → hub syncReal-time service call

5 · Lookup / cross-refReal-time identity resolution

6 · Two-way syncBidirectional, loopback-guarded

7 · Hub → app pushReal-time propagation

8 · Subscribe inApps publish, hub subscribes

TELL 01

The cache fallback is the architecture confessing

The design carried a documented alternative: if performance or availability became an issue, deploy a local data cache on every consuming application, refreshed at low latency. That is a correct engineering answer — and simultaneously an admission that a batch-anchored hub cannot reliably serve real-time reads at enterprise concurrency. In the AI-native design real-time resolution is the primary path, and the cache disappears along with the copies of truth it created.

TELL 02

"Truncate and load daily" was deliberate — and it set the ceiling

Full daily reloads in landing and staging kept the platform decoupled from volatile source systems. Entirely reasonable at the time. It also fixed the freshness ceiling at 24 hours regardless of how much real-time integration sat on top. Streaming CDC into an open table format removes the constraint without giving up the decoupling — this is the single highest-leverage change in most legacy estates.

TELL 03

The trust framework was the right idea, statically configured

Per-source, per-column trust factors were genuinely ahead of their time — the conceptual ancestor of learned survivorship. But the weights were set by hand in workshops and rarely revisited. The AI-native version keeps the concept and makes the weights observed: measured against outcomes, recalibrated continuously, explainable per attribute.

TELL 04

Six tools, five consoles, one engineer

Hub console, data director, DQ analyst, DQ developer, admin console, address standardisation — plus catalog and BI. Each is competent; collectively they demand a specialist per console and make cross-tool root-cause analysis a manual craft. A single agent-mediated cockpit over the same estate is an operating-model change, not a UI preference.

TELL 05

The governance model survives intact

Guiding principles, a target operating model spanning consumers, capabilities, technology, process, governance and roles, a versioned business-rules artefact with impact analysis, defined authorised sources, and a documented chain of accountability for change. None of this is made obsolete by AI. It is precisely the control framework that makes autonomous pipelines defensible to an auditor. Modernise the engine; keep the governance.

TELL 06

The delivery model was already right

Domain-by-domain shippable increments, prioritised by business value, with a proof of concept preceding the MVP. That sequencing maps almost one-for-one onto the phased migration later on this page. Organisations that already run data work this way migrate fastest — the hard organisational muscle is already built.

On this material Details are drawn from prior R2 Digital client engagements and presented in anonymised, generalised composite form. Client names, domains, source systems, volumes and identifying specifics have been removed. Nothing here criticises the client teams or the vendors — the architectures were appropriate and correctly built for their date. The point is narrower and more useful: the constraints that shaped them are no longer constraints, and an architecture optimised against removed constraints is by definition leaving value on the table.

07 — Where AI Attaches

The leverage map

AI is not applied to "data engineering" in general. It attaches at specific, identifiable failure points. Below, each stage of the lifecycle with the pain it generates today and the agent that removes it — plus the trust plane that has to exist underneath before any of it is safe to run unattended.

HIGHEST LEVERAGE

1 · Legacy logic archaeology

Parsing decades of ETL metadata to recover business rules is the task AI does dramatically better than people, and it unblocks every modernisation programme. Vendors now ship migration agents converting 14+ legacy platforms into warehouse-native pipelines; the value is less the code generation than the comprehension.

HIGHEST LEVERAGE

2 · Autonomous incident response

Detection, correlation across lineage, hypothesis, sandbox-validated fix, human approval. This is where the maintenance tax actually lives, and it is the capability with the clearest published implementations today.

HIGHEST LEVERAGE

3 · Unstructured-to-structured extraction

Turning contracts, tickets and documents into governed tables and retrievable context creates data assets that simply did not exist before. It is the only item on this list that increases the surface area of the platform rather than reducing its cost.

LOWER LEVERAGE THAN CLAIMED

Watch: pure code generation

Generating SQL faster is the most adopted and least differentiating use. Writing the model was rarely the bottleneck — agreeing what it should mean and proving it is right were. Optimising the easy step magnifies the hard one.

LOWER LEVERAGE THAN CLAIMED

Watch: NL-to-SQL over raw tables

Ungrounded natural-language querying against a physical schema produces confident, wrong answers. It is only viable on top of a semantic layer — which is the actual investment.

GUARDRAIL

Autonomy needs an approval boundary

Define, per pipeline, what an agent may do unattended (retry, re-infer schema, re-run a partition), what needs approval (logic change, contract change, permission change), and what is never automated (data deletion, policy override, production access grants). Write it down before, not after.

08 — Target Architecture

What an AI-powered data engineering platform looks like

Two structural differences from the modern data stack of 2022. First, an agent control plane runs horizontally across every layer — agents are not a feature inside a tool, they are how the platform is operated. Second, a context plane sits between storage and consumption, making meaning and policy machine-readable. Everything else is recognisable; these two are the change.

Agent control planespans every layer · each agent scoped, permissioned, logged and evaluated

BuilderIntent → pipeline + tests

MigratorLegacy logic → native code

QualityLearned expectations, drift

ZeroOpsDetect · RCA · propose fix

GovernanceClassify, tag, enforce policy

FinOpsCost, pruning, right-sizing

Steward copilotExceptions with reasoning

L1Source & ingestion planecontinuous by default · batch as an optimisation

Log-based CDCOperational databases, sub-minute

Event streamingDirect-to-lakehouse writes, seconds

Managed connectorsSaaS estate, drift-aware

Document & media ingestContracts, tickets, images, audio

Agent-authored connectorsLong-tail APIs specified, not coded

▼

L2Multimodal storage planeone open format · one governance boundary

Open table formatIceberg / Delta — ACID, time travel, engine-neutral

Object & content storeSource documents retained with provenance

Vector indexEmbeddings governed like tables, not beside them

Graph storeEntities and relationships for multi-hop reasoning

Feature storeOnline/offline parity for models and agents

▼

L3Processing & transformation planedeclarative · incremental · model-assisted

Declarative pipelinesDefine the result; the engine plans the execution

Unified batch + streamOne definition, two execution modes

LLM stages in-pipelineExtract, classify, summarise, enrich, embed

Incremental computeRecompute what changed, not the table

Contract enforcementBreaking changes fail at build, not in production

▼

L4Context & semantic planethe layer most enterprises are missing

Semantic modelEntities, metrics, dimensions — one definition of "revenue"

Ontology / knowledge graphHow things relate; supports multi-hop questions

Data contractsSchema, SLA, ownership, versioning as code

Policy-as-codeWho and what may access which attribute, enforced at query

Runtime metadataLineage, freshness, quality — queryable by machines

▼

L5Serving planehumans, applications and agents as first-class consumers

SQL & BIGoverned metrics, not ad-hoc redefinition

Data product APIsVersioned, entitled, metered

Tool / MCP surfaceThe typed, permissioned way agents call data

Retrieval endpointsPermission-aware hybrid search

Feature & inference servingLow-latency reads for decisioning

Trust planeevery layer emits evidence · nothing autonomous ships without it

LineageColumn-level, end to end

ObservabilityFreshness, volume, schema, distribution

EvalsAccuracy of every AI-executed step

AuditWho/what changed data, and why

CostUnit economics per data product

How it fits the AI-first enterprise data architecture

The data engineering platform is not the whole picture — it is the bottom two thirds of it. The value of separating the context plane is that the intelligence layer above can change models, frameworks and vendors without the foundation moving.

Why the context layer is the control point

Models will be replaced. Agent frameworks will be replaced. The semantic model, ontology and policy definitions are the assets that compound — and they are the only part of the stack a vendor cannot supply for you, because they encode how your business defines itself.

Why open formats matter more now

When four engines — SQL, streaming, ML, retrieval — read the same data, engine-neutral storage stops being an ideological preference and becomes a cost and consistency requirement. It is also the cheapest available insurance against platform consolidation.

Why the trust plane comes first

Every autonomy claim on this page depends on being able to answer: what did the agent change, on what evidence, and how do we know it was right? Build lineage, observability and evals before delegating anything. The 2026 data shows this is where most teams are underinvested.

09 — Concept-by-Concept Redesign

Every core concept, reconfigured

The useful question is not "where can we add AI?" but "which of these concepts would we design differently if we started today?" The answer is: nearly all of them. Nine core concepts below, each with what it looks like traditionally, what replaces it, and the honest operational impact.

Traditional

Connector per source, schema pinned at build time
Upstream change breaks the run — or worse, silently drops a column
Backfills are manual, risky and usually out of hours
Long-tail APIs never get connected because the business case per source is too small

AI-native

Schema treated as observed state, versioned and diffed every run
Drift classified automatically: additive (absorb), renamed (map with confidence score), semantic change (escalate)
Agent proposes the mapping with evidence; a human approves anything ambiguous
Long-tail connectors specified in natural language against API docs and generated with tests

Operational impactSchema drift is the single largest maintenance category in most estates. Moving it from "breaks the pipeline" to "raises a reviewable proposal" is the highest-return single change available — and it is the one to implement first because it is reversible and easy to measure.

Traditional

Weeks of workshops to agree a dimensional or Data Vault design
Model degrades as sources evolve; nobody re-runs the design
90% of practitioners report modelling pain — driven by speed pressure and unclear ownership
Documentation written once, at go-live

AI-native

Agents profile sources, infer entities, keys, cardinality and candidate relationships
A first-draft model — dimensional or Vault — generated in hours, then argued over by humans
The model is continuously re-evaluated against actual data and query patterns
Documentation regenerated from the model on every change

Operational impactThe value is not speed of first draft — it is that modelling stops being a one-time event. Be clear-eyed: the agent proposes structure, but only the business can say which grain matters. Design authority stays human.

Traditional

Logic hand-written in SQL, stored procedures or a visual ETL tool
Legacy estates hold thousands of transformations whose intent is undocumented
Migration means reading mappings one by one — the true cost of every modernisation
Change impact assessed by grep and hope

AI-native

Intent expressed declaratively; the agent generates implementation plus tests plus documentation as one unit
Legacy mapping XML, DTSX packages and procedures parsed to reconstruct business rules in plain language
Semantic diffing: does the new model produce the same numbers as the old one, on real data?
Impact analysis from column-level lineage, before merge

Operational impactGenerated code is only trustworthy with parity testing against production data. The rule we apply: no legacy pipeline is decommissioned until its replacement has matched output for a full business cycle, including period-end.

Traditional

Hand-written assertions covering the failures someone already imagined
Coverage concentrated on keys and nulls; semantic errors pass cleanly
Thresholds set once and never recalibrated as the business changes
Quality discovered by the business, reported as a ticket

AI-native

Expectations learned from historical distributions per column, per segment, per season
Anomaly detection on volume, freshness, distribution and inter-column relationships
Semantic validation: does this value make sense given everything else on the record?
New rules proposed from observed data and reviewed by a steward in plain language

Operational impactLearned thresholds generate false positives early — budget for a tuning period and route alerts to a triage queue, not to on-call, for the first weeks. Teams that skip this stage disable the alerts within a month.

Traditional

Cron-style schedules; dependencies encoded by hand and drifting from reality
Failure at 02:40 pages a human who reads logs across five consoles
Root-cause analysis is tribal knowledge; the same incident recurs
Backfill order worked out from memory

AI-native

Data-aware triggers: pipelines run when their inputs are actually ready
Background agent detects the failure, correlates across lineage, forms a hypothesis
Fix validated in a sandbox against real data, then presented for approval — or applied automatically inside a pre-agreed boundary
Every incident and resolution becomes retrievable context for the next one

Operational impactThis is where the maintenance tax actually lives, and where the clearest vendor implementations exist today. Start with propose-only autonomy. Move a pipeline to auto-apply only after it has demonstrated a clean proposal record over a meaningful sample.

Traditional

A catalog populated in a project push, then abandoned
Descriptions written by whoever had least to do that sprint
Lineage partial, table-level, and out of date
Discovery happens by asking a colleague on Slack

AI-native

Metadata harvested continuously from execution, not from documentation effort
Column-level lineage derived from parsed logic and query history
Descriptions, classifications and PII tags generated then human-confirmed
Discovery by natural-language question, answered from the graph with provenance

Operational impactThe shift that matters: metadata stops being documentation and becomes runtime infrastructure. If your catalog cannot be queried by a machine at execution time, it is a wiki with a search box.

Traditional

"Revenue" defined independently in the warehouse, three BI tools and a finance spreadsheet
Reconciliation meetings as a standing calendar item
Business logic duplicated into every dashboard
Natural-language querying over raw tables produces confident, wrong answers

AI-native

One governed semantic model — entities, metrics, dimensions, time grain — compiled to SQL
An ontology above it capturing how entities relate, enabling multi-hop questions
Agents query the semantic layer, never the physical schema
Every answer traceable to a definition and a version

Operational impactThis is the least glamorous and most decisive investment on the page. Independent evaluations report materially higher agent accuracy when unified context is available versus semantic definitions alone. Without it, agentic analytics does not survive its first executive review.

Traditional

Documents live in content management, outside the data platform entirely
Extraction handled by brittle OCR templates that break on layout changes
No lineage, no quality measurement, no access policy consistency
The majority of what the enterprise knows is simply out of scope

AI-native

Parsing, extraction, classification and embedding as governed pipeline stages
Extracted fields land as tables with provenance back to the source page or clause
Vector indexes governed under the same catalog and access policy as tables
Retrieval quality measured with an eval set, monitored like any other SLA

Operational impactThis is the only redesign that expands what the platform can answer rather than reducing what it costs. It is also the most common source of permission leakage — inherit access control from the source document, always, and test it adversarially.

Traditional

Compute spend grows with pipeline count; nobody owns the total
Full refreshes because incremental logic was too hard to write
Warehouses sized for the worst hour and running all day
Optimisation arrives as an across-the-board cut after a bad invoice

AI-native

Cost attributed per pipeline, per data product, per consumer
Agents identify unused tables, redundant refreshes and expensive query patterns, and propose changes with an estimated saving
Incremental processing as the default because the engine plans it, not the engineer
Unused assets flagged for decommission with evidence of zero consumption

Operational impactAI workloads add cost as well as remove it — inference in-pipeline, embedding regeneration, eval runs. Model both sides. A business case that counts only the savings will not survive month four.

10 — Migration Roadmap

What the migration actually involves

Not a re-platforming project. A staged replacement of how data work is produced, verified and operated — run alongside the existing estate, domain by domain, with a committed decommission date attached to each. Indicative timings assume a mid-to-large enterprise with a few hundred pipelines; scale accordingly.

Phase 0 · Wk 0–6

Phase 1 · Mo 2–5

Phase 2 · Mo 5–12

Phase 3 · Mo 12–20

Phase 4 · Ongoing

Phase 0 · Weeks 0–6

Assess & baseline

Harvest metadata across the estate to build the real dependency graph — not the documented one. Establish what exists, what runs, what is consumed, and what nothing has read in a year. Quantify the maintenance tax and cost baseline. Rank domains by value and feasibility.

Exit criteriaA measured baseline (pipelines, incidents, cost, capacity split), a decommission candidate list, and one chosen lighthouse domain with a named business owner.

Phase 1 · Months 2–5

Foundation & lighthouse

Stand up the open lakehouse, the trust plane and the agent control plane — in that order. Migrate one domain end to end with agent-assisted conversion, running in parallel with the legacy path. Prove numerical parity across a full business cycle including period-end.

Exit criteriaOne domain live with column-level lineage, learned quality monitors, propose-only ZeroOps, and a signed-off parity report. Legacy path still running.

Phase 2 · Months 5–12

Industrialise

Turn the lighthouse into a factory: templated conversion, reusable contracts, a golden path domain teams can self-serve. Build the context layer for the migrated domains — semantic model first, ontology where multi-hop questions demand it. Extend to unstructured sources.

Exit criteriaMajority of mastered volume on the new platform; semantic layer serving BI and agents; decommission dates committed and diarised for every legacy pipeline.

Phase 3 · Months 12–20

Agent-ready & autonomous

Expose the typed, permissioned tool surface agents consume. Graduate proven pipelines from propose-only to bounded auto-apply. Turn on FinOps agents. Execute the legacy exit — the phase most programmes quietly skip.

Exit criteriaLegacy platform switched off and licences released; autonomy boundaries documented per pipeline; agent consumption governed and metered.

Phase 4 · Ongoing

Operating model

The permanent change. Data engineers move from pipeline authors to platform and context owners. Stewards move from queue-clearing to adjudicating the genuinely ambiguous. Evals, cost and quality get a governance cadence with the same seriousness as financial close.

Exit criteriaNot a phase with an end — the point at which the platform is run as a product with named owners, published SLAs and unit economics.

The seven workstreams that run underneath

Phases are the narrative; these are the actual work packages. Under-resourcing any one of them is the most reliable predictor of a stalled programme.

WS 1

Discovery & inventory

Automated harvesting of pipelines, schedules, dependencies, consumption and cost. Produces the graph everything else is planned against.

WS 2

Platform & landing zone

Lakehouse, open table format, networking, identity, environments, IaC, CI/CD, golden-path templates.

WS 3

Conversion factory

Agent-assisted extraction of legacy logic, generation of native pipelines, and automated parity testing at scale.

WS 4

Trust plane

Lineage, observability, learned quality monitors, eval harnesses for every AI-executed step, audit logging.

WS 5

Context layer

Semantic model, ontology, data contracts, policy-as-code. Built domain by domain alongside migration — never as a separate later project.

WS 6

Governance & risk

Autonomy boundaries, approval chains, model-risk treatment for AI-in-pipeline, privacy and sovereignty, audit evidence.

WS 7

People & operating model

Role redefinition, skills uplift, incentives, and the decommission accountability that stops the estate doubling.

WS 8

Value tracking

Baseline, instrument and report the measures below monthly. Programmes without a published number lose their funding in year two.

What to measure

Baseline these in Phase 0 or the business case is unfalsifiable. Targets shown are the direction we aim for, not a guarantee — outcomes depend heavily on estate condition and organisational readiness.

↓ 50%share of capacity spent on maintenance and firefighting

↓ 70%mean time to detect a data incident

↓ 60%mean time to resolve, via agent-proposed root cause

24h → <15mfreshness for the domains that need to drive decisions

↑ 3×share of the estate covered by column-level lineage

100%of critical metrics defined once, in the semantic layer

↓ 25%compute cost per data product, net of new AI workload

> 90%of AI-executed pipeline steps covered by an eval

FAILURE MODE 01

The permanent hybrid

Coexistence without a committed decommission date becomes two platforms, two cost lines and two versions of the truth — permanently. Set the legacy exit date in Phase 2, name an accountable executive, and defend it. This is the most common way these programmes fail, and it fails quietly.

FAILURE MODE 02

Acceleration without verification

The 2026 evidence is explicit: adoption of AI authoring is running roughly three times ahead of adoption of AI quality management. Shipping pipelines faster than you can prove them correct converts a productivity story into a trust incident — and trust, once lost with the CFO, takes a year to rebuild.

FAILURE MODE 03

Buying the agent, skipping the context

An agent over an undocumented physical schema is a very fast way to produce confident wrong answers. The semantic layer is unglamorous, hard to fund and non-negotiable. If the roadmap has it in "phase later", the roadmap is wrong.

FAILURE MODE 04

Treating it as a technology programme

The practitioner data ranks organisational blockers above technical debt. If nobody owns customer data, no architecture fixes it. Ownership, decision rights and decommission accountability are Phase 0 deliverables, not change-management garnish.

Free consultation

How can we help?

Every platform is different — the pipeline count, the legacy tooling, the state of governance, the appetite for autonomy. A 45-minute working session with R2 Digital: we look at your current data engineering footprint, identify where AI leverage is real versus marketing, and sketch a first-90-days sequence. No cost, no obligation.

Email us directly

More from R2 Digital

This piece is part of a series on building the AI-first enterprise. Each stands alone; together they cover strategy, investment, master data and now the engineering foundation.

Sources

External references used on this page

Published February–July 2026 unless noted. Vendor sources are marked; their benchmarks are self-reported and should be read as directional.

01Gartner Announces Top Predictions for Data and Analytics in 2026gartner.com · analyst 02Agent-Ready Data Will Reshape Data Management System Market Share by 2027gartner.com · analyst research note 03Gartner D&A Summit 2026 — context layer vs. intelligence layer takeawaysatlan.com · vendor summary of analyst event 042026 Hype Cycle for Agentic AIgartner.com · analyst 05Lakeflow: A new era of agentic data engineering (June 2026)databricks.com · vendor 06AI Data Engineering: New Smart Pipelines in Snowflake (June 2026)snowflake.com · vendor 07Microsoft Fabric June 2026 Feature Summary — agent-authorable Data Factory skillsmicrosoft.com · vendor 082026 State of Analytics Engineering Report (363 respondents)getdbt.com · vendor-run practitioner survey 09dbt Labs: AI-driven acceleration is outpacing trust and governanceprnewswire.com · vendor release 102026 State of Data Engineering Survey (1,101 respondents)Practical Data Community · independent practitioner survey 11AI-Ready Data for Advanced AI — the 7% "data reinventors" findingaccenture.com · consultancy research 12McKinsey Global Tech Agenda 2026 — data and AI foundationsmckinsey.com · consultancy research 13Deloitte — State of AI in the Enterprise, 2026deloitte.com · consultancy research 14The Forrester Wave™: Data Quality Solutions, Q1 2026forrester.com · analyst 15Data Management Trends in 2026: Moving Beyond Awareness to Actiondataversity.net · industry commentary 16Key takeaways from Gartner D&A Summit 2026cloudera.com · vendor summary 17Migration Agent: autonomous migration for 14 legacy ETL platforms (March 2026)maia.ai / Matillion · vendor 18GenAI partner accelerators for data engineering & migrationdatabricks.com · vendor 19Secondary analysis of the Fivetran and dbt 2026 datasets — the 53% maintenance figureabhs.in · independent analysis, not primary data 20Context layer for AI agents — enterprise guide, incl. agent-accuracy comparisonatlan.com · vendor (commercially interested) 21Semantic layer vs. context layer; knowledge graph vs. context graphenterprise-knowledge.com · consultancy 22What is data observability — the five pillarsmontecarlo.ai · vendor 2372% of data teams use AI; 71% fear bad data — report coveragebigdatawire.com · trade press 24Snowflake and Databricks Summits 2026: what actually matterspointfive.co · vendor commentary