FBI-LLM: Scaling Up Fully Binarized LLMs from Scratch via Autoregressive Distillation

Problem: Large Language Models (LLMs) have immense parameter scales that create substantial storage and computational demands, limiting their practical deployment. Although weight binarization can alleviate these issues, extreme 1-bit quantization typically suffers from significant accuracy degradation. Furthermore, existing low-bit methods like BitNet b1.58, PB-LLM, and OneBit either utilize ternary representations, retain salient weights in full precision, or rely on existing full-precision models, which restricts flexibility in model architecture and vocabulary size.

Method: The authors present FBI-LLM, a framework to train fully binarized LLMs from scratch using FBI-Linear modules that binarize weight matrices to 1-bit while scaling them with column-wise learnable vectors a and b to form W_{b} = \text{sign}(W) \odot a + b. To stabilize training, they employ a streamlined autoregressive distillation (AD) loss where the binarized student model matches the soft predicted probability distribution of a full-precision teacher model (such as LLaMA2-7B). The embedding, causal head, and normalization layers are maintained in full precision, and backpropagation is handled using the Straight-Through Estimator (STE).

Results: Evaluated across 130M, 1.3B, and 7B parameter scales on the Amber dataset, FBI-LLM achieves competitive perplexity and downstream task accuracy compared to other binary baselines on benchmarks like BoolQ and HellaSwag. The 1.3B model achieves up to 87% of the performance of its full-precision counterpart, while the 7B model consistently improves with further training. Trajectory analysis through the flip-flop ratio demonstrates that initializing from pretrained weights is unnecessary, as training from scratch achieves similar optimization stability and equivalent training loss.

Limitations: Despite these advances, FBI-LLM still suffers from an unavoidable performance gap compared to full-precision counterparts. Additionally, the training process requires knowledge distillation from a teacher model, which introduces extra computational overhead. Finally, current hardware lacks direct support to achieve real inference speedups with fully binarized LLMs, activation quantization is not addressed, and the binarized models may still retain ethical concerns and biases from the pretraining data.