Lakehouse Architectures: Principles and Python Implementation

Data Lakes: Evolution of Data Storage

Relational databases and data warehouses served as primary analytical data stores for decades. These systems excel with structured data, enforcing schema-on-write to ensure data quality and consistency for business intelligence (BI) reporting. Data transformation occurred during ingestion, fitting source data into predefined schemas.

As data volumes grew and new data types emerged, traditional data warehouses faced limitations. Unstructured data, such as sensor readings, application logs, and social media feeds, did not fit neatly into rigid table structures. Storing and processing these diverse formats became expensive and complex within a schema-on-write paradigm. Many organizations also needed to retain raw data for future, undefined analytical use cases, including machine learning model training, which traditional warehouses were not designed to support efficiently.

The demand for cost-effective storage for all data, regardless of structure or immediate purpose, led to the concept of a data lake. A data lake is a centralized repository that stores data in its native format, typically object storage like Amazon S3 or Azure Blob Storage. This approach enables schema-on-read, where data interpretation happens at query time, offering flexibility for evolving analytical needs.

Early data lakes, often built on Hadoop Distributed File System (HDFS), provided massive scalability and low-cost storage. Data was ingested directly from sources, including transactional databases, streaming platforms, and application logs, without prior transformation. This raw data availability supported diverse workloads, from ad-hoc analysis to advanced analytics and machine learning.

However, these early implementations introduced new challenges. The lack of enforced schema or metadata often led to “data swamps,” where data became difficult to discover, understand, or trust. Basic data management features like ACID (Atomicity, Consistency, Isolation, Durability) transactions, common in databases, were absent, complicating data updates and ensuring data quality. These issues highlighted a gap between the flexibility of data lakes and the reliability of data warehouses.

Lakehouse vs. Data Warehouse: Key Differences

Data architectures traditionally separate into data warehouses and data lakes, each designed for distinct purposes and data types. A data warehouse stores highly structured, curated data, optimized for business intelligence (BI) and reporting. It enforces a schema-on-write approach, meaning data conforms to a predefined structure upon ingestion. This design provides strong data integrity and fast query performance for analytical workloads but limits flexibility with unstructured or semi-structured data.

In contrast, a data lake ingests raw, unprocessed data in its native format, adopting a schema-on-read approach. This offers immense flexibility, accommodating diverse data types including logs, images, and sensor data, making it ideal for machine learning and exploratory analytics. However, data lakes often lack transactional consistency, schema enforcement, and the performance guarantees necessary for critical BI dashboards, leading to potential data quality issues and complex data governance.

The lakehouse architecture merges the strengths of both paradigms, providing a unified platform for all data workloads. It stores data in open, standardized formats like Parquet or ORC directly in a data lake, maintaining cost-effectiveness and flexibility. On top of this raw data, the lakehouse introduces data warehousing capabilities, including ACID (Atomicity, Consistency, Isolation, Durability) transactions, schema enforcement, and data quality features.

Key characteristics of a lakehouse include:

• Open Formats: Data resides in formats like Apache Parquet or ORC, accessible by various engines.
• ACID Transactions: Ensures data reliability and consistency, even with concurrent writes.
• Schema Enforcement & Governance: Allows defining and enforcing schemas for specific tables, improving data quality.
• Separation of Storage and Compute: Storage costs remain low in object storage, while compute resources scale independently.
• Direct Data Access: Tools can read data directly from the underlying storage layer.

This architecture enables organizations to use a single copy of data for both traditional BI and advanced analytics, eliminating data silos and reducing ETL complexity. It offers the performance and reliability of a data warehouse with the flexibility and cost advantages of a data lake.

Storage Formats: Apache Parquet and Delta Lake

Efficient data storage is fundamental for data lake and lakehouse architectures. Raw data often arrives in formats like CSV or JSON, which are not optimized for analytical queries. Specialized formats improve query performance, reduce storage costs, and enable advanced data management features.

Apache Parquet is a columnar storage format. Unlike row-oriented formats, Parquet stores data column by column. This structure allows for higher compression ratios because data within a single column is often of the same type and has similar values. Query engines can also read only the necessary columns, significantly reducing I/O operations and improving predicate pushdown efficiency.

Writing a Parquet file with pyarrow involves creating a table from data:

import pyarrow as pa
import pyarrow.parquet as pq
import pandas as pd

data = pd.DataFrame({'id': [1, 2, 3], 'value': ['Alpha', 'Beta', 'Gamma']})
table = pa.Table.from_pandas(data)
pq.write_table(table, 'sensors_data.parquet')
print("Parquet file 'sensors_data.parquet' created.")

While Parquet optimizes storage and query performance, it does not inherently provide transactional capabilities. Data consistency, schema evolution, and concurrent writes become challenges in multi-user environments. Delta Lake addresses these limitations by adding an open-source storage layer on top of Parquet.

Delta Lake stores data in Parquet format but augments it with a transaction log. This log records every change made to the data, providing ACID (Atomicity, Consistency, Isolation, Durability) properties. Features like schema enforcement prevent writes that introduce incompatible schema changes, and time travel allows querying previous versions of the data.

Creating a Delta Lake table uses a similar approach, writing Parquet files alongside the transaction log:

from deltalake import write_deltalake
import pandas as pd

new_data = pd.DataFrame({'id': [4, 5], 'value': ['Delta', 'Epsilon']})
write_deltalake('sensors_data_delta', new_data, mode='append', partition_by=['id'])
print("Delta Lake table 'sensors_data_delta' updated.")

Parquet serves as an efficient file format for immutable, batch-processed datasets where transactional guarantees are not required. It is simpler to implement for basic data storage. Delta Lake, conversely, is a transactional storage layer that uses Parquet. It adds complexity but provides data reliability, consistency, and advanced features for mutable data, streaming ingestion, and concurrent workloads. The choice depends on the specific requirements for data mutability, consistency, and concurrency within the lakehouse.

Building a Mini Lakehouse with Python

We implement a basic lakehouse architecture using Python to demonstrate core principles. This setup uses local file system directories to represent distinct data zones, enabling controlled data flow and processing.

The architecture consists of three zones:

1. Landing Zone (./data/landing): Temporary storage for newly ingested raw data. Data here is transient and unprocessed.
2. Raw Zone (./data/raw): Stores data after initial validation and conversion to an open, columnar format like Parquet. Data in this zone is immutable and retains its original form, often partitioned.
3. Curated Zone (./data/curated): Contains transformed, cleaned, and aggregated data optimized for specific analytical workloads. This zone often includes schema enforcement and versioning.

First, set up the directory structure:

import os
from datetime import datetime
import pandas as pd
import pyarrow.parquet as pq
import pyarrow as pa

os.makedirs('./data/landing', exist_ok=True)
os.makedirs('./data/raw', exist_ok=True)
os.makedirs('./data/curated', exist_ok=True)
print("Directory structure created.")

Data ingestion begins in the landing zone. We simulate an external system dropping a CSV file.

# Simulate data ingestion to landing zone
landing_data = pd.DataFrame({
    'id': [1, 2, 3],
    'name': ['Alice', 'Bob', 'Charlie'],
    'value': [100, 150, 200],
    'timestamp': ['2023-01-01T10:00:00Z', '2023-01-01T11:00:00Z', '2023-01-01T12:00:00Z']
})
landing_data.to_csv('./data/landing/sales_20230101.csv', index=False)
print("Simulated data written to landing zone.")

Next, data moves from landing to the raw zone. This step includes schema enforcement. We define an expected schema and validate incoming data against it before converting to Parquet.

# Define expected schema for raw data
expected_schema = pa.schema([
    pa.field('id', pa.int64()),
    pa.field('name', pa.string()),
    pa.field('value', pa.int64()),
    pa.field('timestamp', pa.string())
])

# Process data from landing to raw zone
landing_file = './data/landing/sales_20230101.csv'
df_raw = pd.read_csv(landing_file)

# Simple schema validation using PyArrow
try:
    table = pa.Table.from_pandas(df_raw, schema=expected_schema)
    pq.write_table(table, './data/raw/sales_data_20230101.parquet')
    print("Data validated and written to raw zone as Parquet.")
except pa.ArrowInvalid as e:
    print(f"Schema validation failed: {e}")
except Exception as e:
    print(f"Error processing data: {e}")

The curated zone holds processed data. We apply a simple transformation and implement basic versioning. Each version of the curated data resides in a timestamped subdirectory, ensuring immutability and enabling historical queries.

# Process data from raw to curated zone
df_curated = pd.read_parquet('./data/raw/sales_data_20230101.parquet')

# Example transformation: add processing timestamp
df_curated['processing_date'] = pd.to_datetime('now').floor('D')

# Implement basic versioning for curated data
current_version_dir = f"./data/curated/{datetime.now().strftime('%Y%m%d%H%M%S')}"
os.makedirs(current_version_dir, exist_ok=True)
df_curated.to_parquet(f"{current_version_dir}/sales_summary.parquet", index=False)
print(f"Transformed data written to curated zone with versioning: {current_version_dir}")

This structure demonstrates the flow from raw ingestion to schema-enforced storage and versioned, transformed data.

Why Lakehouses Fail: Common Pitfalls

Lakehouse architectures often fail when they revert to unstructured data lakes, losing the reliability of data warehouses. The primary causes are uncontrolled data ingestion, lack of transactional integrity, and insufficient data governance.

Uncontrolled data ingestion is a frequent pitfall. Without defined schemas or metadata, raw data lands in storage without context, rapidly transforming the lakehouse into a “data swamp.” Data engineers then spend significant time deciphering ambiguous datasets, often leading to data being unused because its meaning or quality is unknown. Best practice involves schema enforcement at ingestion, using tools like Apache Iceberg or Delta Lake that register schema with each write.

# Example: Enforcing schema with Delta Lake on write
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("DeltaSchemaEnforcement").getOrCreate()

data = [("Alice", 1), ("Bob", 2)]
df = spark.createDataFrame(data, ["name", "id"])

# This will fail if the schema does not match the existing table schema
# unless mergeSchema is explicitly used.
df.write.format("delta").mode("append").save("/mnt/delta/users")

A second critical failure point is the absence of ACID (Atomicity, Consistency, Isolation, Durability) properties. Without a transactional layer, concurrent writes can corrupt data, and readers may see inconsistent states. This undermines trust in the data, making the lakehouse unreliable for analytical or operational workloads. Implementing a transactional table format, such as Delta Lake or Apache Hudi, is an essential practice to guarantee data integrity. These formats provide multi-version concurrency control and transaction logs, enabling reliable updates and deletes.

Inadequate data governance compounds these issues. Many lakehouse implementations neglect centralized metadata management, granular access controls, and data lineage tracking. Data becomes difficult to discover, secure, or audit. A robust data catalog, like AWS Glue Data Catalog or Databricks Unity Catalog, is crucial for centralizing metadata, enabling data discovery, and enforcing access policies. This allows data stewards to define who can access what data, ensuring compliance and preventing unauthorized exposure.

Implementing data quality checks throughout the pipeline prevents bad data from propagating. Tools like Great Expectations or Deequ integrate validation rules directly into ingestion and transformation processes. This proactive approach identifies and quarantines poor-quality data early, preventing unreliable insights from reaching business users. A failure to implement these checks leads directly to distrust in the data assets.