[1-Bit Quantization] Scaling Binary (BitNet) Logic for 16MB Parameter Golf Constraint

[Zzcvbnma]

Hi Team,

I'm Ranjit, an independent AI developer. I'm taking on the 16-million parameter (16MB) challenge by implementing a 1-bit Binary (BitNet-inspired) architecture.

To strictly adhere to the model size constraint while maximizing parameter count, I've designed a custom BitLinear layer that uses +1/-1 weights. This approach allows me to squeeze ~134M parameters within the 16MB artifact limit.

Current Progress:

• Initial 1-bit logic implemented in my repo: Zzcvbnma/Parameter-Golf-Binary-Beast
• Optimization for 8x H100 distributed training (XNOR-based logic).
• Final 10-minute training run (Awaiting compute grant/credits).

I'm aiming for sub-1.10 BPB loss. Has anyone else experimented with extreme quantization for this challenge? I'd love to hear your thoughts on convergence stability in the first 600 seconds!