R097 · AI education › Learning paths for AI engineers

MiniMind: the whole LLM training pipeline at 64M parameters

Read the original

educationlearning-resourcesllm-trainingreinforcement-learningopen-weightsteaching

· Source: Dr. Alvaro Cintas on LinkedIn, on jingyaogong/minimind (Apache-2.0)

TL;DR

MiniMind (Jingyao Gong, Apache-2.0) is a teaching repo that runs every stage of a modern LLM recipe on a 64M-parameter Qwen3-style decoder: tokenizer, pretraining, SFT with tool calls and a thinking switch, LoRA, DPO, PPO/GRPO/CISPO, multi-turn agentic RL, and white-box distillation. Each stage is one readable script (the agentic RL trainer is 488 lines), with open datasets for every stage and weights that load in transformers, llama.cpp, vLLM and Ollama.

The value for an engineer is not the model, which hallucinates freely and scores near chance on most benchmarks. It is seeing the post-training stack as code you can step through, and the README's honest notes on why RL breaks at small scale (all-zero rewards, degenerate groups, reward hacking by the judge model). It goes far past nanoGPT, which stops at pretraining.

The "2 hours" headline is the mini pretrain plus mini SFT on one RTX 3090 (about 2.3 h, about RMB 3). The full recipe, the RL stages and the 14 GB SFT set take much longer, and the author trains on 8x 3090.

What the post claims vs what the repo says

Post claimWhat the README and code show
Train your own LLM from scratch in about 2 hours2.31 h on one 3090 for pretrain_t2t_mini (1.2 GB) + sft_t2t_mini (1.6 GB), one epoch, about RMB 3 of rental. That gives a "Zero" chat model. The full sets are 10 GB and 14 GB; tool-call SFT and RLAIF add about 1 h each on the mini data.
64M parameters, about 1/2700 of GPT-3True for minimind-3 (8 layers, d_model 768, 8 query / 4 KV heads, 6,400-token vocab). There is also a 198M-total, 64M-active MoE (4 experts, top-1).
Full pipeline, seven stagesTrue. Also MoE, YaRN context extrapolation, an OpenAI-compatible server with tool_calls and reasoning_content, and a Streamlit chat UI.
Plain PyTorch, no high-level librariesMostly. Losses, LoRA, DPO, PPO with GAE, GRPO, CISPO and the rollout loop are hand-written. transformers still supplies the tokenizer, the PreTrainedModel base class and the reward model loader. RLAIF needs a separate 1.8B reward model (InternLM2-1.8B-Reward); sglang is an optional rollout backend.
Runs on one consumer GPUTrue for the mini path. The author's own box is 8x RTX 3090.
Vision, multimodal and diffusion versionsSibling repos MiniMind-V and MiniMind-O; diffusion (dLM) and linear attention are experimental write-ups in Discussions.

Two things the post leaves out. The data is heavily distilled: about 100K tool-call samples from Qwen3-4B, R1 reasoning traces, and Qwen3 reasoning data. So the weights are from scratch, but much of the behaviour is copied from bigger models. The data also skews Chinese, so English output is weak.

Pipeline architecture

Two stages are required (pretrain, SFT); everything else branches off the SFT weights. Every script supports DDP via torchrun, resumable checkpoints (--from_resume 1, including across a changed GPU count), and wandb or SwanLab logging.

StageScriptDataWhat it teaches
Tokenizertrain_tokenizer.pyany corpusBPE + ByteLevel, 6,400 tokens with tool-call and think tokens. The README advises not retraining it: a small vocab keeps the embedding and output layers a small share of 64M params, and changing it breaks weights and tooling. Compare models by bits per byte, not perplexity, across tokenizers.
Pretraintrain_pretrain.pypretrain_t2t(_mini).jsonlPlain next-token prediction on cleaned, deduplicated text already in the later chat template.
SFTtrain_full_sft.pysft_t2t(_mini).jsonlMulti-turn chat with system/user/assistant/tool roles. Tool calls are mixed into SFT rather than a separate stage. At 14 GB this is closer to mid-training than format alignment.
Adaptive thinkingchat templatemixed into SFT/RLOne model, an open_thinking flag: the template injects an empty or an open think block; training mixes both via a thinking_ratio. Replaced a separate reasoning model in March 2026.
LoRAtrain_lora.py, model_lora.pylora_*.jsonlHand-written low-rank adapters for domain or identity tuning; convert_model.py merges them back into full weights.
DPOtrain_dpo.pydpo.jsonl (from DPO-En-Zh-20k)Off-policy preference learning: actor plus a frozen reference, no reward or value model. Good for style and safety, weak for correctness.
PPOtrain_ppo.pyrlaif.jsonlActor, critic and full GAE. Reward rises slowly because the critic has to converge first.
GRPO / CISPOtrain_grpo.pyrlaif.jsonlGroup-relative advantage, no critic. CISPO keeps gradient flowing through clipped ratios by using the clipped ratio as a weight on log-prob.
Agentic RLtrain_agent.pyagent_rl.jsonl, agent_rl_math.jsonl (with ground truth)Multi-turn tool-use rollouts with a trajectory-level reward (next section).
Distillationtrain_distillation.pyteacher outputsWhite-box: CE plus temperature-scaled KL to the teacher's token distribution, dense or MoE pairs.

The README's best teaching device is a single frame for every preference and policy algorithm: each one is a choice of policy term, advantage estimate and KL regulariser. DPO, PPO, GRPO and CISPO are then four settings of three knobs rather than four separate algorithms.

Architecture is a Qwen3-aligned decoder: pre-norm RMSNorm, SwiGLU, RoPE (theta 1e6) with YaRN, GQA. The author picked a wide, shallow 8 x 768 shape for training speed, while citing MobileLLM's finding that deep and narrow usually wins at this size.

Agentic RL design

train_agent.py is the most useful file for anyone building agent harnesses, because it is a complete tool-use RL loop small enough to read in one sitting.

One trajectory, one reward: the agentic RL loopPrompt + toolswith ground truthPolicy rolloutN samples per promptTool call?Score trajectoryanswer, tool, format, RMMock tool runsresult appendedGroup advantageGRPO or CISPO, KL to refyesnoup to 3 turnsweights update, next batch
agentic RL loop in train_agent.py

Tool calls loop back into the same rollout; only the finished trajectory is scored, and the group of N trajectories sets each one's advantage.

R(\tau) = R_{answer} + R_{tool} + R_{format} + R_{rm} - R_{unfinished}

The README is candid about where this stops. Rewards from a judge model drift from what the environment would say, so agent tasks should lean on environment rewards. And because the training data has few samples that both think and call tools, turning both on at once makes the model unstable.

Results and how to read them

The headline result is the agent task: on 20 tool-using arithmetic problems, the agentic-RL weights got 17/20 (85%) against 12/20 (60%) for plain SFT. That is the clearest evidence that the RL loop does something. Twenty items is tiny, so treat it as a smoke test, not a measurement.

On benchmarks scored with lm-evaluation-harness, the model is near chance. minimind-3 scores 24.9 C-Eval, 25.4 CMMLU, 28.5 ARC-Easy, 50.7 PIQA. SmolLM2-135M gets 58.5 ARC-Easy and 68.2 PIQA. The README says plainly these are "only for fun" and prints the standard error per benchmark, noting that gaps under about 5 points on OpenBookQA, or about 3 on C-Eval, PIQA and Social-IQa, do not separate two models. That is a good habit to copy.

Two results teach more than the scores:

The subjective ranking (a GPT-5.4 Thinking judge over a handful of Chinese prompts) placed the MoE model first for fluency and correct code but noted dense hallucinations. One judge, a few prompts, no repeats: anecdote only.

Tradeoffs, failure modes, caveats

How to use it

A weekend plan for an engineer who has only used LLMs through an API:

  1. Download minimind-3 and run eval_llm.py with --open_thinking 0 and 1, then eval_toolcall.py. Read the chat template to see how tools and think blocks become tokens.

  2. Read model_minimind.py (292 lines): attention with GQA, RoPE/YaRN, the MoE router.

  3. Train the Zero model on the mini data (about 2.3 h on a 3090, or rent one). Watch the loss curves and compare pretrain vs SFT answers.

  4. Run train_grpo.py and log group_reward_std. Then swap in a binary reward and watch it collapse; that is the reward-sparsity lesson made concrete.

  5. Read train_agent.py end to end and change one reward term. Re-run the 20-task check and see what moves.

When reviewing a training or RL setup that borrows from this repo:

Sources

Connections

See R097 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.