Title: Lecture 10 — The scikit-learn Workflow: Train, Predict, Evaluate
Course: AIINFRA 100: AI Systems & Development Environments > Section: Week 11 — Machine Learning Libraries: scikit-learn

📖 Lecture — The scikit-learn Workflow: Train, Predict, Evaluate

Up to now you've been working with data and tools that mostly describe or transform information. This week you meet scikit-learn, the most widely used Python library for classical machine learning. "Classical" is the key word here — before we go further, let's clear up a common mix-up. Machine learning is not the same thing as deep learning. Deep learning uses neural networks with many layers, usually needs a GPU and a lot of data, and shows up in libraries like PyTorch or TensorFlow. scikit-learn, by contrast, is built around classical ML models: decision trees, linear/logistic regression, k-nearest neighbors, support vector machines, and more. These models are often faster to train, easier to interpret, and perfectly suited to the small-to-medium, tabular datasets you'll work with in this course. Neural networks are one tool in the ML toolbox — not the whole toolbox.

The Estimator API: one interface, many algorithms

The single biggest reason scikit-learn is so approachable is its uniform Estimator API. Every model — regardless of what it does internally — implements the same two core methods:

Method | What it does
fit(X, y) | Trains the model on your features (X) and labels (y)
predict(X) | Uses the trained model to make predictions on new data

Because every estimator follows this pattern, you can swap KNeighborsClassifier() for DecisionTreeClassifier() or LogisticRegression() and the rest of your code — loading data, splitting, fitting, predicting, scoring — barely changes. This is a deliberate design choice, and it's what lets beginners experiment with several algorithms in minutes rather than rewriting pipelines from scratch each time.

The five-step workflow

Nearly every scikit-learn project follows the same shape:

1. Load and inspect the data. For learning, scikit-learn ships small "toy" datasets like load_iris() or load_digits() that need no downloading. For real projects, you'll typically load a CSV with pandas instead.
2. Split into train and test sets. Use train_test_split to set aside a portion of the data (commonly 20-30%) that the model never sees during training.
3. Choose and fit an estimator. Pick a model class, create an instance, and call .fit(X_train, y_train).
4. Generate predictions. Call .predict(X_test) to see what the trained model thinks the held-out examples are.
5. Evaluate with a metric. Compare predictions to the true test labels using a metric appropriate to the task.

Why the test set is non-negotiable

Here's the single most common beginner mistake: evaluating a model on the same data it was trained on. If you fit a model on all your data and then score it on that same data, the number you get back is almost meaningless — the model may have simply memorized the training examples rather than learned a generalizable pattern. This is called overfitting, and it's why we always hold out a test set with train_test_split before fitting anything. The test score tells you how the model is likely to perform on data it has never encountered, which is the entire point of building a model in the first place. A related and very common misconception is that high training-set accuracy means the model is "good." It doesn't — a model can reach 99-100% accuracy on training data while performing poorly on new data, precisely because it has overfit. The only way to catch this is to compare training accuracy against test (or cross-validated) accuracy. A large gap between the two is a red flag.

Cross-validation: a more reliable estimate

A single train/test split gives you one performance number, but that number depends on which rows happened to land in the test set — especially risky with small datasets. Cross-validation (via cross_val_score or cross_validate) solves this by splitting the data into several "folds," training and testing multiple times so every row gets used for both training and testing across different runs, and then averaging the results. The average of several folds is a sturdier estimate of real-world performance than any single split.

Picking the right metric

Evaluation depends on the type of problem you're solving:

Task type | Common metrics | What it tells you
Classification | Accuracy | % of predictions that were correct overall
Classification | Precision / Recall / F1 | How well the model handles each class, especially with imbalanced data
Classification | Confusion matrix | A table showing exactly which classes get confused with which
Regression | MAE (Mean Absolute Error) | Average size of prediction errors, in the original units
Regression | MSE (Mean Squared Error) | Like MAE but penalizes large errors more heavily
Regression | R² | How much of the variance in the target the model explains

Choosing the right metric for your task matters more than chasing the "best" algorithm — a model tuned to maximize the wrong metric can look great on paper and still fail at the actual job you need it to do.

Preprocessing and pipelines

Real data usually needs preparation before a model can use it: numeric features often need scaling so no single feature dominates just because of its units, and categorical features need encoding into numbers. It's tempting to do this preprocessing once on the whole dataset — but that risks data leakage, where information from the test set subtly influences training (for example, if you scale using statistics computed across the entire dataset, including the test rows). scikit-learn's Pipeline class solves this by bundling preprocessing and modeling steps into a single object that fits the preprocessing on training data only, then applies it consistently to both train and test data. Pipelines also make cross-validation safe and your code far cleaner, since "training the model" and "making a prediction" both become one .fit() / .predict() call on the whole pipeline.