The Agentic Turn in Data Engineering: Why Architecture Still Leads

In a previous article we built an AI agent inside Databricks that reads your data, reasons about it, and hands back recommendations you can act on. That piece was about how to stand one up. This one is about the question that shows up the morning after: where does an agent actually earn its place across the data engineering workflow, and what decides whether it helps or just makes a mess faster?

Picture the second case. An agent opens a pull request overnight. It has refactored a transformation and added a passing test, and the description reads well. It has also renamed a column to customer_id in a model where every neighbouring table has called it cust_key for three years. Nobody told the agent that convention existed. Nothing in the diff is broken. It still doesn't belong in your platform, and a reviewer now has to notice that before it merges. Multiply that across a week of agent-generated changes and the shape of the problem is clear.

What Is AI Agentic Automation? Moving Beyond Autocomplete

The useful line to draw runs between a single model call and an agent. A single call takes a prompt and returns text (a completion, a summary, an answer in a chat window) and then waits for you. An agent runs a loop instead. It can read a schema, propose a change, run the test against that change, and use the failure it gets back to try a different approach, calling tools and acting across systems until it reaches something it treats as finished.

That loop is what makes agents useful for data work. It also means an unbounded agent will eventually act somewhere you didn't want it to, so the version that holds up in production is a bounded one: the agent does the multi-step grind inside limits you set, and a human signs off at the points that matter.

Top Use Cases for AI Engineering Agents in Software Development

Across a typical data engineering workflow, agents are already good at a recognisable band of tasks:

• Documentation. Data dictionaries, column descriptions, lineage summaries, and runbook drafts that stay close to the code instead of rotting in a wiki.

• Test and quality scaffolding. Turning expectations into a first draft of test cases, anomaly checks, and validation rules: a starting point an engineer sharpens, not the final word.

• Coding and refactoring. Generating transformation logic, translating between SQL dialects, and mechanical refactors across a lot of files at once.

• Review. A first pass over a diff that flags anti-patterns, style drift, and obvious performance or governance concerns before a human reviewer spends attention on it.

• Debugging and triage. Reading logs, tracing a failure back to its cause, and proposing a fix for a person to approve.

• Modeling. Profiling a source and proposing a schema, a normalization choice, or a first-cut dimensional design.
  

These tasks are not equal, though. Documentation and test scaffolding are cheap to check and hard to break; if a dictionary entry comes out wrong, someone corrects it and little else happens. A schema change or a production fix carries real downside: the toil it removes is high, but a bad one propagates before anyone notices. Where a task sits on that spectrum, far more than how impressive the agent looks doing it, is what should decide how much rope you give it.

Watch the recording of our webinar "Bridging Business and Analytics: The Plug-and-Play future of Data Platforms"
Why Context and System Architecture Control AI Agent Performance

An agent is only as coherent as the context it works from, and on a data platform most of that context is the architecture. Naming conventions, modeling standards, a semantic layer that pins down what "revenue" actually means, an owner for each domain: those are what let an agent produce something that fits the rest of the platform instead of being merely correct on its own terms. Take that structure away and the agent still produces something. It produces confident output that reads fine in isolation and quietly disagrees with everything around it: the renamed column, the metric computed a fourth different way, the table normalized in a model where three others denormalized it on purpose. Each change looks reasonable on its own. Together they are technical debt, arriving faster than a team of people could have written it. This is why "an agent will clean up our messy platform" has the causation backwards: an agent consumes structure, it can't manufacture it. Hand one a well-modeled, well-governed platform and it compounds good engineering. Give it an inconsistent starting point and you get that inconsistency back at scale, which costs more to unwind later than getting the foundation right would have cost up front.

Best Practices for AI Governance, Guardrails and Human-in-the-Loop Delegation

You can only delegate a task you are able to check, contain, and undo, which is exactly why governance, so often treated as a brake on agents, is what actually lets you hand them real work. Access controls bound what an agent can reach. Data contracts hand it stable definitions to build against. Review gates stop plausible-but-wrong output before it lands, and when something does slip through, an audit trail and clear domain ownership tell you what changed and who answers for it. A team that trusts those boundaries can delegate more to agents, precisely because it knows where the edges are. That is also why the sensible rollout follows the blast radius. Begin where the toil is high and the downside is low (documentation, test scaffolding, first-pass review) and let agents run with a light human touch. Keep people firmly in the loop on modeling and anything that reaches production. Measure the impact instead of assuming it, then widen an agent's autonomy as your guardrails and your trust in them mature. If you cannot yet check an agent's output on a given task quickly and reliably, that task is not ready to hand over yet; more often than not, the thing blocking it is a gap in the foundation rather than a limit of the agent.

Moving towards Agentic Governance: Our Conclusion

Agentic AI is becoming a real layer on top of the modern data platform and the productivity gains are not hypothetical. But it works like a multiplier, and a multiplier only ever reflects what you give it: good engineering on a well-governed platform, and every existing inconsistency on a fragile one, both scaled up. The teams that get the most out of it won't necessarily be the earliest adopters, but the ones who did the unglamorous work first (the architecture, the standards, the governance) and then let agents move faster on top of it. What agents change is the speed of the work, never the question of who is accountable for deciding what "good" looks like in the first place.

If you would like an outside perspective on whether your data platform is in shape to delegate to agents safely, we would be glad to talk it through with you.