utomation has quietly become part of our everyday lives. Many of us now live in homes where lights switch on automatically when we enter a room, the thermostat adjusts before we arrive home, and the alarm arms itself when everyone leaves. What started with a few programmable devices has evolved into connected smart homes where lighting, heating, security, and energy management work together seamlessly.

The same evolution is taking place in Data Platform Automation.

Like a smart home, automation is not simply manual or autonomous. It evolves through distinct Automation Design Levels, each representing a different architectural approach to designing and building modern data platforms—from manual implementation to AI-driven automation.

So, what do these automation design levels look like?

Level 0 - Manual

At Level 0, every aspect of the data warehouse is developed manually. Data engineers write SQL scripts, create database objects, build ETL or ELT processes, deploy code, and document the solution by hand. Requirements gathering, source-to-target mappings, data profiling, and validation are all manual activities.

There is little or no standardization, making solutions highly dependent on individual developers and difficult to maintain or scale. Every new source system or business requirement typically results in another set of custom scripts.

While this approach provides maximum flexibility, it also introduces the highest development effort, greatest maintenance costs, and largest risk of inconsistency.

Design principle: The engineer writes everything.

Level 1 - Developer-Assisted

Level 1 introduces tooling that helps engineers become more productive without fundamentally changing the way solutions are designed.

Templates, graphical modeling tools, ETL tools, reusable components, and AI coding assistants reduce repetitive work and encourage consistency. Engineers can generate basic table structures or pipeline skeletons instead of writing everything from scratch.

However, the most valuable part of the solution still depends entirely on human expertise. Business rules, transformation logic, validation rules, performance optimization, and architectural decisions remain manual.

The tools assist the engineer, but they do not understand the architecture being built.

Design principle: The tool helps you build the solution, but you still design everything yourself.

Level 2 - Metadata-Driven

Metadata becomes the primary driver of automation.

Rather than manually developing every pipeline, engineers configure metadata describing sources, targets, mappings, and load characteristics. Generic frameworks then generate much of the required SQL and pipeline logic automatically.

This dramatically reduces repetitive coding and improves consistency. A single metadata-driven ingestion framework can load hundreds of tables using the same reusable components.

However, there is an important limitation. The platform understands technical metadata, but it does not understand the architecture of a data platform. Consider a traditional data warehouse: at level 2, the platform has no understanding of concepts such as business keys, dimensions, facts, surrogate keys, slowly changing dimensions, or semantic relationships. Business rules transformations, and validation logic still need to be explicitly defined by engineers.

Design principle: Technical metadata drives implementation.

Level 3 - Rule-Driven

Level 3 marks the point where automation becomes architecture-aware.

Instead of simply generating code from technical metadata, the platform applies predefined architectural and business rules. These rules capture modeling patterns, governance standards, naming conventions, and implementation best practices.

In a traditional data warehouse, for example, the platform can automatically apply dimensional modeling techniques, Data Vault patterns, surrogate key strategies, historization rules, and standard calculations. 

Likewise, in a medallion architecture, the platform can automatically generate Bronze, Silver, and Gold layers, apply standardized transformation and data refinement patterns, enforce data quality rules, and orchestrate data movement across the layers.

Rather than developing individual pipelines, engineers define and maintain the architectural rules that drive automation.

Design principle: The platform understands architectural intent and applies design rules automatically.

Level 4 - Knowledge-Driven

Automation extends beyond technical metadata and architectural rules by incorporating enterprise knowledge into the automation process.

Data catalogs, business glossaries, governance policies, semantic models and increasingly graph-based metadata repositories provide the business context needed to guide architectural and implementation decisions.

Automation no longer focuses only on how to build the platform, but also on what the data means and how it should be governed.

Using a medallion architecture as an example, the platform can leverage enterprise knowledge to apply governance policies across Bronze, Silver, and Gold layers, enforce consistent business definitions, and maintain lineage across the platform.

In a traditional data warehouse, it can use the same enterprise knowledge to identify conformed dimensions, apply governance and data quality policies, and maintain consistent business definitions and lineage.

Design principle: Enterprise knowledge drives implementation decisions.

Level 5 - Autonomous

The long-term vision of Data Platform Automation is an autonomous platform.

Instead of defining models, rules, or metadata, users express business intent, while AI determines how to design, build, operate, and continuously optimize the data platform.

The platform moves beyond assisting engineers to acting as an autonomous engineering partner. It can interpret business requirements expressed in natural language, recommend or generate end-to-end architectures, design and optimize data models, generate pipelines and orchestration workflows, and continuously monitor and improve the platform based on changing business requirements and operational feedback.

Although fully autonomous platforms remain largely aspirational today, many of the underlying capabilities—such as AI-assisted modeling, semantic reasoning, autonomous optimization, and intelligent orchestration—are already beginning to emerge.

Design principle: AI drives implementation decisions.

Choosing the Right Automation Design Level

The automation design levels should not be interpreted as a traditional maturity model.

Each level represents a different architectural approach with its own strengths, trade-offs, and suitable use cases. A metadata-driven approach may be ideal for standardized ingestion, while a rule-driven or knowledge-driven approach may be better suited to enterprise-scale data platforms.

Even within a single platform, different workloads may adopt different automation design levels.

The objective is therefore not to reach the highest automation design level, but to select the approach that best supports the architectural and business requirements of the problem being solved.

From Smart Homes to Smart Data Platforms

The journey of a smart home provides a useful way to visualize these automation design levels.

Just as today's smart homes evolved from isolated devices into intelligent ecosystems, modern data platforms are evolving from manually engineered solutions into increasingly intelligent automation platforms.