Title: Lecture 5 — Storing, Moving, and Trusting Your Data
Course: AIINFRA 100: AI Systems & Development Environments > Section: Week 13 — Data & Storage Infrastructure for AI

📖 Lecture — Storing, Moving, and Trusting Your Data

Every AI model you have heard about — chatbots, recommendation engines, self-driving cars — was trained on data that had to be stored somewhere, moved somewhere, and cleaned up before a model ever touched it. It is tempting to think of that work as boring plumbing that happens "before the real AI part." In practice, the plumbing is the AI part. A brilliant model architecture trained on messy, unrepresentative, or poorly organized data will still produce a bad product. This week we look at the systems that store and move data for AI, and the quality bar that data needs to clear before it is trustworthy enough to learn from.

Warehouses, lakes, and lakehouses

A common misconception is that "a data lake is just a bigger, cheaper data warehouse." That is not accurate — they are built for different jobs, and most real organizations end up needing both (or a hybrid of the two). A data warehouse stores structured data — rows and columns, like a giant spreadsheet — and enforces a schema before the data goes in. This is called schema-on-write. Because the structure is locked in ahead of time, warehouses are extremely fast and reliable for business intelligence (BI): sales dashboards, quarterly reports, and SQL queries that finance and operations teams run every day. A data lake, by contrast, stores raw data in whatever form it arrives — structured, semi-structured (JSON logs), or fully unstructured (images, video, audio, free-text documents) — and only applies structure when someone reads it later. This is called schema-on-read. Lakes are what make modern AI/ML possible, because most interesting training data (photos, sensor logs, customer support transcripts) does not fit neatly into spreadsheet rows. Because AI teams need both the flexibility of a lake and the reliability of a warehouse, the industry has converged on the lakehouse — a single system (Databricks and Delta Lake are the best-known examples) that stores raw data like a lake but adds warehouse-style structure, versioning, and query performance on top. Lakehouses now power more than half of analytics workloads, precisely because most organizations got tired of maintaining two separate systems that constantly needed to be kept in sync.

 | Data Warehouse | Data Lake | Lakehouse
Schema applied | On write (before storage) | On read (at query time) | Both, depending on use
Data types | Structured only | Structured, semi-structured, unstructured | All types
Best for | BI dashboards, reporting | ML training, raw data archiving | Both BI and ML from one system
Example tools | Traditional SQL warehouses | Cloud object storage (raw files) | Databricks / Delta Lake

Why file format is an AI decision, not just an IT detail

Once you know where data lives, the next question is how it's packaged. Three formats come up constantly in AI pipelines:

- CSV — universal and human-readable (you can open it in any text editor or spreadsheet program), but it is uncompressed, slow to scan at large scale, and has no built-in way to describe data types.
- JSON — handles nested and semi-structured data well, which makes it the natural format for API responses and application logs, but that flexibility comes at the cost of being verbose and slower to process at scale.
- Parquet — a columnar, binary format that is compressed, fast to query, and self-describing (the schema — column names and types — is embedded in the file itself). Because ML training often means scanning millions of rows but only a handful of columns at a time, Parquet's columnar layout is dramatically more efficient than CSV or JSON. This is why Parquet is the de facto standard for large-scale ML training data.

Feature stores: compute once, use everywhere

As teams build more models, the same derived values — a customer's average order size over the last 30 days, a sensor's rolling temperature average — end up being recalculated separately by every team, and often slightly differently each time. A feature store solves this by acting as a centralized repository for these engineered features. It typically has two halves:

- An offline store, holding historical feature values used for training models (optimized for large batch reads).
- An online store, holding the latest feature values for low-latency lookups when a live model needs to make a prediction in real time.

By computing a feature once and reusing it consistently in both training and production, feature stores eliminate training-serving skew — the subtle (and hard-to-debug) problem where a model performs well in training but poorly in production because the live version of a feature was calculated differently than the training version.

Data quality: it is not just "more data is better"

The second major misconception to correct is that "more data automatically means a better AI model." Volume helps, but only if the data is good. A huge pile of stale, biased, or inconsistent data can make a model worse, not better, because the model faithfully learns whatever patterns — including bad ones — are in front of it. This is the classic "garbage in, garbage out" problem, and it applies to AI more than almost anywhere else in computing, because the model has no way to know the data was flawed. For AI systems specifically, data quality is usually judged across six dimensions:

Dimension | What it means | What goes wrong if it fails
Accuracy | Values correctly reflect reality | Model learns wrong facts or relationships
Completeness | No important missing values | Model fills gaps with bias or noise
Consistency | Same meaning represented the same way everywhere | Model treats identical things as different
Timeliness | Data is available when needed | Decisions are made on stale information
Representativeness / bias | Data reflects the real population it will be used on | Model performs worse (or unfairly) for underrepresented groups
Recency | Data reflects current conditions, not outdated ones | Model drifts away from real-world behavior over time

The first four (accuracy, completeness, consistency, timeliness) are classic data-quality dimensions that predate AI. The last two — representativeness/bias and recency — are especially important for AI because a model does not just report on data, it makes decisions and predictions from it, and those decisions can quietly encode whatever imbalance or staleness was baked into the training set.

Pipelines: where quality gets built in (or doesn't)

Data rarely walks straight from a source system into a model. It travels through a pipeline — commonly described as ETL (Extract, Transform, Load) or ELT (Extract, Load, Transform) — that pulls data from source systems, reshapes it, and lands it somewhere the AI system can consume it (often a lake, warehouse, or lakehouse). The critical lesson for AI infrastructure is that data-quality checks should be built into the pipeline itself — validating types, checking for missing values, flagging outliers — rather than bolted on afterward as a manual, one-time cleanup. A pipeline that only moves data without validating it will happily ship broken data downstream, at scale, on a schedule.

Bringing it together

Data infrastructure decisions — warehouse vs. lake vs. lakehouse, CSV vs. JSON vs. Parquet, feature store or not, and how rigorously the pipeline checks quality — are not implementation details hidden behind the model. They are architectural decisions that determine whether an AI system will actually work in production. Next week we look at how these systems get deployed across cloud, on-premises, and hybrid environments — and how to keep costs under control.