Genesis 1B: Run 2 Extended — 20k → 40k Steps
Genesis 1B Run 2 extended to 40,000 steps (~21B tokens, above Chinchilla-optimal). Step 38,081 / 40,000 — 95.2% complete, ETA April 7.
Research, engineering, and insights from Kroonen AI.
Genesis 1B Run 2 extended to 40,000 steps (~21B tokens, above Chinchilla-optimal). Step 38,081 / 40,000 — 95.2% complete, ETA April 7.
Same 1B parameters, 3x throughput. How torch.compile, real-valued RoPE, a deeper architecture (32 layers vs 20), and batch tuning tripled training speed on the same 2x RTX 4090 hardware.
Data sovereignty, constitutional alignment, and the case for training language models on consumer hardware. Why the future of AI is local, private, and personality-first.
A silent AdamW state bug during Run 1 that produced a false recovery on poisoned weights. The load path didn't crash or hang -- it just silently ruined the model.
How we fixed FSDP checkpoint deadlocks during Run 1 on consumer GPUs without NVLink using DCP sharded checkpoints and decoupled evaluation.