Now loading...
Character.ai, the startup behind innovative conversational AI, has opened up about its early days of training massive language models. Before the company turned its attention to fine-tuning open-source foundations, its initial team experimented with clever ways to speed up and streamline the process of pretraining large transformers. Now, those innovations are being released publicly for the first time, offering a peek into the engineering wizardry that powered their progress.
The disclosures spotlight a handful of key methods developed under the guidance of co-founder Noam Shazeer, a veteran in AI research. Among them is Squinch, a clever 6-bit technique for compressing gradients that slashes communication needs between computing nodes without sacrificing model performance. This was crucial when Character.ai ran its biggest pretraining setups on hardware with just a quarter of the bandwidth found in top-tier systems. By breaking gradients into blocks and encoding eight values into a tight 48-bit package, Squinch kept training accurate even in bandwidth-starved environments. It is still useful today for specialized setups like sparse mixture-of-experts models or cross-domain work.
Another standout is Attention Z-Reg, a regularization trick that keeps attention scores in check during training. By nudging the total activation levels toward zero, it ensures that the brain float 16 format can operate in its most precise range, avoiding the loss of detail that happens with very large numbers. This builds on ideas from mixture-of-experts research but applies it more broadly to attention and linear layers, integrated directly into the optimization without adding extra loss terms.
For handling quantization, the team devised Dynamic Clamping to stop tiny activation values from getting rounded away to nothing, which could derail training stability. In feed-forward networks, they adjusted clamping thresholds on the fly based on weight statistics, preventing overcrowding in low-value ranges and preserving accuracy across quantized steps. This simple adjustment made a big difference in keeping models steady as they scaled.
To make attention computations more efficient, especially with complex data like chat histories, Character.ai introduced the Visibility Mask. This uses two compact arrays to define which parts of the input can “see” each other, supporting everything from simple causal masking in single documents to tree-like structures in multi-turn conversations. It packs unrelated sequences together for better hardware use, enables bidirectional attention where needed, and even aids advanced inference tricks like beam search.
On the distillation front, where smaller models learn from larger “teacher” ones, the company optimized storage for offline training runs. Using Gumbel Softmax sampling, they subsampled the teacher’s output probabilities while keeping the distribution unbiased, dramatically cutting down on the data needed to store for student model training. This draws from established reparameterization techniques but tailors them for large vocabularies, balancing efficiency with fidelity.
Beyond these, Character.ai’s early playbook included staples like native 8-bit training with dynamic scaling, shared key-value caching in a interleaved pattern, and thoughtful hyperparameter tweaks inspired by scaling laws. They also warmed up batch sizes gradually for smoother large-model runs and augmented data with synthetic variations to sharpen question-answering skills.
Though Character.ai has moved away from from-scratch pretraining, these tools still shape their current efforts in refining open-source models. You can check out their ongoing projects on GitHub, such as pipelining-sft and Ovi. The company is hiring engineers to push these boundaries further in post-training reinforcement learning, with openings at their careers site.
