When most people imagine preparing a dataset for fine-tuning, they picture a race to accumulate as many examples as possible. It feels intuitive — more data, more learning, a better model. But one of the most influential results in the fine-tuning literature says almost the opposite. In the paper LIMA: Less Is More for Alignment, researchers fine-tuned a 65-billion-parameter LLaMA model on just 1,000 carefully curated instruction-response pairs — no reinforcement learning from human feedback, no massive crowdsourced corpus, just 1,000 hand-selected examples. The resulting model matched or approached the output quality of models trained on far larger, less curated datasets. This week is about internalizing that lesson and turning it into a repeatable process: how do you build a small dataset that actually works? Start with the task distribution, not the data. Before you collect or select a single example, you need to answer a concrete question: what inputs will this model see in production, and what outputs should it produce for each? If you are fine-tuning a model to answer closed-domain questions about your company's internal documentation, your task distribution is "employee questions about policy X, Y, Z answered in a specific tone, grounded in specific source documents." If you skip this step and just grab whatever data is available, you end up with a dataset that teaches the model a little bit of everything and nothing well. Defining the task distribution first lets you evaluate every candidate example against a clear standard: does this example represent something the model will actually be asked to do? Curate for coverage and correctness before volume. Once you know the task distribution, the next job is making sure your examples span it. That means representing the different sub-tasks, phrasings, edge cases, and difficulty levels your model needs to handle, and it means every example needs to be factually correct and well-formed. A single mislabeled or sloppy example does not just fail to help — it actively teaches the model the wrong pattern. Duplicate examples are almost as harmful: they over-weight one pattern in the loss function and starve the model of exposure to everything else. This is why data quality work is mostly subtractive. You are not asking "what else can I add?" You are asking "what should I remove, fix, or deduplicate?" Below a certain size, models start memorizing instead of learning. A useful mental model: below roughly 1,000 examples per task, a model has so few instances of a pattern that it tends to memorize the training examples verbatim rather than extracting the generalizable rule behind them. That memorization looks fine on the training set and falls apart the moment a real user phrases a request slightly differently. This is one reason LIMA-style curation works — 1,000 examples is close to the minimum viable size for a task, but only if every one of those 1,000 examples is doing real work: covering a distinct case, phrased distinctly, and correct. Tooling makes curation tractable at scale. The Hugging Face datasets library is built for exactly this workflow: loading a dataset, filtering it down to the categories or examples you actually want, computing summary statistics (length distributions, category balance, duplicate counts), and — when your source corpus is too large to fit in memory or on disk — streaming it example-by-example via IterableDataset so you never have to materialize the whole thing at once. This matters even for small target datasets, because your source corpus (the pool you are curating from) may be huge. Synthetic data generation is a legitimate 2026 tool, used carefully. When labeled data for your domain is scarce, a common and increasingly standard practice is to use an existing LLM to generate additional instruction-response pairs from a small set of human-written seed examples. The seed examples anchor the style, difficulty, and correctness bar; the LLM expands coverage. The risk is quality drift — synthetic examples can still be low-quality or subtly wrong, so they need the same filtering and review as human-written data, not a pass. The table below summarizes the shift in mindset this week asks you to make:
| Old assumption | What the evidence shows | What to do instead |
|---|---|---|
| More examples = better fine-tuning | LIMA reached strong quality with \~1,000 curated pairs; oversized low-quality sets can hurt performance | Set a quality bar first, then decide how much data clears it |
| You need thousands of examples to fine-tune | 500–1,500 high-quality curated examples often outperform 10x larger, noisier datasets | Budget your time for curation, not just collection |
| Data prep is mostly about gathering more | Duplicates and low-quality examples actively hurt generalization | Treat cleaning (dedup, filtering, correctness checks) as the main event |
| Volume first, task definition later | Without a defined task distribution, added data doesn't target real production needs | Define task distribution before touching a single example |
Addressing the misconceptions directly. It is worth being blunt about two beliefs that are common and wrong. First: "more training data always leads to better fine-tuning outcomes." It doesn't. Past the point where your examples cover the task distribution well, additional low-quality or duplicate examples add noise to the loss signal and can measurably degrade output quality — this is not a minor caveat, it is the central finding of the curation-over-scale literature. Second: "you need thousands of examples to fine-tune effectively." You don't, and assuming you do leads teams to skip the harder, more valuable work of curation in favor of the easier work of scraping more data. LIMA and similar studies show 500–1,500 well-chosen examples routinely outperforming datasets an order of magnitude larger. Your job this week, in the lab, is to practice exactly this kind of curation on a real dataset.