Title: Lab 10 — Clean and Analyze a Real Dataset with pandas and NumPy
Course: AIINFRA 100: AI Systems & Development Environments > Section: Week 10 — Jupyter & the Data-Science Stack: NumPy & pandas

🛠️ Lab 10 — Clean and Analyze a Real Dataset with pandas and NumPy (50 pts)

Goal: Use pandas to load, profile, and clean a real public dataset in a Colab notebook, then convert part of it into a NumPy array and perform a simple vectorized calculation — previewing exactly how raw data becomes model-ready input, while practicing good notebook habits. Steps:

1. Open a new notebook in Google Colab (a local JupyterLab install also works). No paid tools are required.
2. Load the Data Carpentry ecology survey dataset directly into pandas with pd.read_csv(). (The dataset and its description are linked in this week's Resources, via the Data Carpentry lesson.)
3. Profile the data: run .head() to preview the first rows, .info() to see column types and missing-value counts, and .describe() for summary statistics. In a text cell, write 2–3 sentences describing what you notice (e.g., which columns have missing values, what the numeric ranges look like).
4. Handle missing values in at least one numeric column, using either .fillna() (e.g., the column mean) or .dropna(). In a text cell, explain which approach you chose and why.
5. Filter the DataFrame using a boolean mask — for example, keep only rows where a numeric column (such as weight) is above a threshold you choose. Print the number of rows before and after filtering.
6. Compute a .groupby() aggregation — for example, the average value of a numeric column per category (such as average weight per species). Print the result.
7. Convert your cleaned numeric column into a NumPy array with .to_numpy(), then perform a simple vectorized normalization: (arr - arr.mean()) / arr.std(). Print the first 5 normalized values.
8. Before submitting, use Restart & Run All to confirm your notebook runs top-to-bottom without hidden state. Then add a final markdown cell summarizing in 3–5 sentences how this workflow (load → clean → filter → aggregate → convert to array) mirrors the first stage of a real AI/ML data pipeline.

Deliverables: Submit your completed notebook (.ipynb file, or a shared Colab link pasted into the text entry box) showing all code cells with their output, plus your written responses from Steps 3, 4, and 8.