R120 · AI education › Learning paths for AI engineers

nanochat: the whole LLM pipeline in one hackable repo

Read the original

educationlearning-resourcesllm-trainingreinforcement-learningteaching

TL;DR

nanochat is Andrej Karpathy's minimal, single-node harness that trains a chat model end to end: tokenizer, pretraining, SFT, RL, evaluation, KV-cache inference and a chat CLI. Its reference speedrun beats GPT-2's CORE score (0.2565) in about 1.65 hours on one 8xH100 node, roughly $48 at $3 per GPU-hour. On 2026-11-17 it had 60,615 stars and 8,509 forks, MIT license, created 2025-10-13, last push 2026-09-07.

Study it as a readable reference implementation and an experiment loop, not as a production training stack. Mayank A.'s post (2026-11-01) gets the stage list right but skips the two ideas that make the repo distinctive: one depth dial that sets every hyperparameter, and a public time-to-GPT-2 leaderboard.

What it is: the pipeline in one script

runs/speedrun.sh runs every stage in order on a blank 8xH100 box. Each stage is one readable script.

  1. Data. nanochat/dataset.py downloads pretraining shards of about 250M characters each; the speedrun pulls 170. The dataset switched to NVIDIA ClimbMix in March 2026, which alone cut time-to-GPT-2 from 2.76 to 2.02 hours.

  2. Tokenizer. scripts/tok_train.py trains a GPT-4-style BPE with a 32,768-token vocabulary on about 2B characters; tok_eval.py reports compression.

  3. Pretraining. scripts/base_train.py trains a depth-24 model at a data-to-parameter ratio of 8, below the compute-optimal default of 10.5, so it finishes sooner. FP8 is on; the optimizer in optim.py mixes Muon and AdamW.

  4. Base eval. DCLM CORE score and validation bits per byte, a loss that does not depend on vocabulary size.

  5. SFT. scripts/chat_sft.py trains on one task mixture: SmolTalk (460K conversations), MMLU auxiliary train for 3 epochs, GSM8K train for 4 epochs (math with a Python tool).

  6. RL. scripts/chat_rl.py runs reinforcement learning on specific tasks.

  7. Chat eval and inference. ARC, GSM8K, HumanEval, MMLU and SmolTalk tasks; engine.py does KV-cache decoding, execution.py sandboxes the Python tool, and chat_cli lets you talk to the result.

A CPU and Apple Silicon path (runs/runcpu.sh) shrinks the model to run in tens of minutes, with toy results.

Design choices worth studying

What the post gets right and wrong

How to use it, failure modes, checklist

The best use for a staff engineer is a ground-truth model of where training compute and quality come from, so vendor claims about fine-tuning, precision or context can be checked against something you have run.

Failure modes to expect:

Checklist before using it for learning or a team session:

Related entries: R097 (MiniMind, the same stage-by-stage idea at 64M parameters; MiniMind goes wider on post-training such as DPO, PPO, GRPO and agent RL, while nanochat goes deeper on pretraining speed and scaling), R098 (Stanford CS329A, same post author), R024 and R063 (agents tuning training runs, the theme behind nanochat's autoresearch records), R066 (statistical gates for accepting self-improvement changes), R081 and R022 (learning lists).

Sources (as of 2026-11-17):

Connections

See R120 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.