Data pipeline creation used to require specialized engineers and ran roughly forty hours per pipeline. The business analyst held the domain knowledge — which fields matter, what “valid” means, how the source maps to the model. What they lacked was a way to translate that knowledge into pipeline logic. So every request queued behind engineering, and the queue set the pace of the business.
What the assistant generates against
The temptation is to describe the solution as “Claude writes the ETL.” That framing invites the one question a risk committee will always ask — what stops it from writing the wrong thing? — and leaves it unanswered. So reframe it.
A business user describes the need in plain language; the assistant returns a complete pipeline configuration: schema mapping, transformations, validation. The configuration is not free-form code. It is a structured artifact that must conform to a data contract before anything runs. The contract is the guardrail. Self-serve is safe because a non-conforming request produces a non-conforming config, which the platform rejects — not because a human catches it three steps later.
That is the whole design, and it is a workflow, not an agent. Anthropic draws the line cleanly: workflows orchestrate models and tools through predefined code paths, while agents let the model direct its own process and tool use [1]. The guidance that follows is to reach for the simplest thing that works, and to place programmatic checks between steps so each stage is verified before the next [1]. Natural language in, structured config out, contract-validated, declarative engine executes — every step is inspectable and deterministic. Nothing roams the warehouse.
Rung one — the data contract
Start with one source and one target. The analyst says, in effect: take these customer records, standardize the country codes, drop rows with no account ID, land the result here on this cadence. The assistant emits the schema mapping, the transformations, and the validation rules as a configuration that conforms to the target’s contract.
On Databricks that configuration is a Lakeflow Declarative Pipeline — the framework formerly shipped as Delta Live Tables — where you declare the tables and the rules and the engine handles execution, ordering, and orchestration [2][3]. The validation rules are Expectations: declarative constraints that check each record as it flows and, by your policy, warn on it, drop it, or fail the update [2]. The success criterion is not “looks right.” It is mechanical: the output schema matches the contract, and the Expectations pass. That is what makes the forty-hours-to-one-hour collapse defensible rather than merely fast — the definition of done is a property of the system, not a matter of opinion.
Rungs two through four — product, mesh, AI-ready
The simple case wins the demo. The next three rungs win the enterprise.
Take a harder request: a revenue-by-segment dataset assembled from a billing system, a CRM, and a product-usage stream, needed by finance, by the analytics team, and — increasingly — by downstream AI. Here the assistant generates more than a pipeline. It generates a product specification: the contract, the transformations, the Expectations, and the catalog registration — name, owner, tags, description — that turns a table into a governed data product.
That registration is the hinge. In Unity Catalog, governance is not configured per application; it operates beneath every data and AI interaction, enforcing access control and capturing lineage automatically [4]. So the moment the assistant registers the product, three things follow at once. It is a data product — owned, documented, quality-checked. It is discoverable across the mesh — Unity Catalog lets domains publish products under one consistent set of policies, found through centralized metadata [5]. And it is AI-ready — because column-level lineage is captured automatically, every value traces to its origin, which is the precondition for letting a model consume it and still answer to an auditor [6].
The self-serve claim scales for exactly this reason. Each artifact a business user creates is born governed, documented, and discoverable — so the hundredth self-served product is as trustworthy as the first, and the catalog grows richer rather than messier as usage climbs. Databricks’ own metadata-driven pattern, dlt-meta, makes the mechanism explicit: you generate the metadata, and the framework generates the pipeline from it [7]. The assistant’s job is to produce good metadata against the contract. The platform does the rest.
Coda — the same pattern on AWS
The pattern is not Databricks-specific; it lands on the hyperscaler too, and the way it lands is the instructive part. AWS’s unified answer is the next generation of Amazon SageMaker — Unified Studio over a SageMaker Lakehouse, with governance built in through SageMaker Catalog [8]. Its AI-native pipeline builder is Amazon Q data integration in AWS Glue: describe the job in plain language and it returns a complete integration job across many sources and transforms — and, fittingly, it runs on Amazon Bedrock, the same place Claude lives [9]. So the assistant in this post is buildable on AWS directly: call Claude on Bedrock, generate the config, done [10].
The difference is where the guardrail sits. Amazon Q generates code that an engineer reviews and then runs — AWS’s own guidance is to check the script for accuracy before deploying [9]. The guardrail is the reviewer, and governance is assembled around the output from the catalog and access layer. The declarative path inverts that: the generated artifact must conform to a contract the engine enforces, so the guardrail is the target, not the review. Both platforms ship the AI-native front door. The property this post cares about — self-serve safe by construction — comes from making the generated artifact conform to a contract before it runs, whichever platform you build on. Databricks gives you that intrinsically through the declarative engine; on AWS you reach it by validating Claude’s config against the contract before Glue executes. The design decides it, not the logo.
The assistant never gets smarter across these four rungs. The ground it writes into gets more governed. That is the entire point, and it is the inversion most teams miss: the value is not the model’s fluency, it is the contract, the catalog, and the declarative engine it is required to write through. Put governance in the architecture and self-serve stops being a risk you tolerate for speed — it becomes a property you can defend to the board.
The next dispatch takes the same lens to a process with no clean success criterion — and why that absence is the signal to stop, not to build.
