TL;DR
nanochat is Andrej Karpathy's minimal, single-node harness that trains a chat model end to end: tokenizer, pretraining, SFT, RL, evaluation, KV-cache inference and a chat CLI. Its reference speedrun beats GPT-2's CORE score (0.2565) in about 1.65 hours on one 8xH100 node, roughly $48 at $3 per GPU-hour. On 2026-11-17 it had 60,615 stars and 8,509 forks, MIT license, created 2025-10-13, last push 2026-09-07.
Study it as a readable reference implementation and an experiment loop, not as a production training stack. Mayank A.'s post (2026-11-01) gets the stage list right but skips the two ideas that make the repo distinctive: one depth dial that sets every hyperparameter, and a public time-to-GPT-2 leaderboard.
What it is: the pipeline in one script
runs/speedrun.sh runs every stage in order on a blank 8xH100 box. Each stage is one readable script.
Data.
nanochat/dataset.pydownloads pretraining shards of about 250M characters each; the speedrun pulls 170. The dataset switched to NVIDIA ClimbMix in March 2026, which alone cut time-to-GPT-2 from 2.76 to 2.02 hours.Tokenizer.
scripts/tok_train.pytrains a GPT-4-style BPE with a 32,768-token vocabulary on about 2B characters;tok_eval.pyreports compression.Pretraining.
scripts/base_train.pytrains a depth-24 model at a data-to-parameter ratio of 8, below the compute-optimal default of 10.5, so it finishes sooner. FP8 is on; the optimizer inoptim.pymixes Muon and AdamW.Base eval. DCLM CORE score and validation bits per byte, a loss that does not depend on vocabulary size.
SFT.
scripts/chat_sft.pytrains on one task mixture: SmolTalk (460K conversations), MMLU auxiliary train for 3 epochs, GSM8K train for 4 epochs (math with a Python tool).RL.
scripts/chat_rl.pyruns reinforcement learning on specific tasks.Chat eval and inference. ARC, GSM8K, HumanEval, MMLU and SmolTalk tasks;
engine.pydoes KV-cache decoding,execution.pysandboxes the Python tool, andchat_clilets you talk to the result.
A CPU and Apple Silicon path (runs/runcpu.sh) shrinks the model to run in tens of minutes, with toy results.
Design choices worth studying
One complexity dial.
--depthsets width, heads, learning-rate scaling, training horizon and weight decay so every size comes out compute-optimal. A proposed change has to help at every depth, which guards against tuning to one config. GPT-2 capability lands around depth 24 to 26.A metric loop people can compete on. The leaderboard ranks wall-clock time to beat GPT-2 CORE on 8xH100, with the commit for each record. It fell from 3.04 hours (Jan 29, 2026) to 1.65 hours (Mar 14). Two of the six records came from "autoresearch" rounds, which the README credits to Karpathy without describing in detail.
What to watch per run. For a 5-minute depth-12 experiment the README names val_bpb against steps, time and FLOPs, the CORE score, VRAM, model FLOPs utilization and tokens per second.
A modern small transformer in 555 lines.
gpt.pyuses rotary embeddings with no learned positions, QK norm, parameter-free RMSNorm, ReLU-squared MLPs, grouped-query attention, a sliding-window pattern (three short layers, one long) and gated value embeddings. FlashAttention 3 runs with an SDPA fallback.Explicit precision, not autocast. Weights stay in fp32 for the optimizer; a custom
Linearcasts to one global compute dtype per forward pass. fp16 training gets a GradScaler for pretraining and SFT, but RL does not support it yet.No framework on purpose. No config objects, model factories or flag trees: a strong baseline meant to be forked, not configured. PRs must disclose substantial LLM-written code the author does not fully understand.
What the post gets right and wrong
Right: the stage list, the single speedrun script, and that the code is built for changing one thing and measuring it. The 59K star figure was close (60.6K two weeks later).
Overstated: "understand attention, RoPE, GQA and normalization." The code implements them; it does not teach them. The explanation lives in Karpathy's guides in the repo Discussions (Beating GPT-2 for under $100, miniseries v1, adding abilities), and the README points newcomers to DeepWiki.
Missing the cost: the speedrun needs a rented 8xH100 node for about 1.5 to 2 hours. One GPU works but takes 8 times longer, and cards under 80GB need a smaller
--device-batch-size. The CPU path shows the flow, not a useful model.Missing what makes it different: the depth dial and the leaderboard. Without them it reads like any from-scratch tutorial repo.
A commenter's question, answered: is the data mix a knob? Partly. Pretraining uses one dataset, and you choose how many shards. The SFT mix is exposed as
--mmlu-epochsand--gsm8k-epochs; anything else means editing the task list inchat_sft.py.For the student who asked where to start: run
runcpu.shonce for the whole flow, then read the stage scripts in order.
How to use it, failure modes, checklist
The best use for a staff engineer is a ground-truth model of where training compute and quality come from, so vendor claims about fine-tuning, precision or context can be checked against something you have run.
Failure modes to expect:
OOM on smaller cards until the device batch size drops; non-CUDA paths (xpu, mps) are lightly tested.
Wall-clock comparisons across hardware mean nothing; compare val_bpb against FLOPs instead.
Speedrun settings serve the leaderboard. The data ratio of 8 is deliberately below compute-optimal; do not copy it as a general recipe.
Flattering evals. SFT trains on MMLU and GSM8K training splits, so those chat scores overstate general ability. The model itself is tiny (about 4e19 FLOPs) and hallucinates freely.
Not a fine-tuning stack. It trains from scratch; it is not a tool for adapting pretrained open weights.
Checklist before using it for learning or a team session:
Run
runcpu.shlocally to see every stage once.Read
gpt.py,optim.pyandengine.pybefore renting GPUs.Budget about $15 to $50 per speedrun (spot vs on-demand 8xH100).
Make one change, rerun depth 12, and judge it on val_bpb vs FLOPs and CORE.
Check the change still helps at a second depth before believing it.
Treat MMLU and GSM8K chat scores as in-distribution.
Related and sources
Related entries: R097 (MiniMind, the same stage-by-stage idea at 64M parameters; MiniMind goes wider on post-training such as DPO, PPO, GRPO and agent RL, while nanochat goes deeper on pretraining speed and scaling), R098 (Stanford CS329A, same post author), R024 and R063 (agents tuning training runs, the theme behind nanochat's autoresearch records), R066 (statistical gates for accepting self-improvement changes), R081 and R022 (learning lists).
Sources (as of 2026-11-17):
Mayank A., LinkedIn post, 2026-11-01
karpathy/nanochat: README,
runs/speedrun.sh,nanochat/gpt.py,scripts/chat_sft.pyGitHub repository metadata (stars, forks, license, last push)