W-1.1 is Jhelum Labs' first language-model development programme — a decoder-only Transformer trained from scratch, targeting 123M parameters, open weights, and transparent research documentation.
The project focuses on building and studying Transformer-based language models, with particular attention to training methodology, data preparation, model architecture, optimization, text generation, and reproducibility. Rather than treating model development as simply downloading pretrained weights, the project emphasizes understanding and implementing every underlying component: tokenization, Transformer blocks, attention mechanisms, training loops, validation, checkpointing, and inference.
The W-1.1 development programme has included small-scale prototype experiments (Tazi), a Spark experimental model, and the target W-1.1-125M model — intended to follow a GPT-2-small-like architectural approach with approximately 40,000-token vocabulary and 2 billion training tokens.
Architecture & Design
The W-1.1-125M target follows a decoder-only Transformer design. A decoder-only Transformer predicts the next token from the preceding context, learning statistical and semantic patterns in text through autoregressive generation.
Token Embeddings
Convert discrete token identifiers into dense numerical representations the model can process.
Causal Self-Attention
Each token incorporates information from earlier tokens only — no future token access during next-token prediction.
Feed-Forward Networks
Transform token representations through learned nonlinear functions within each Transformer block.
RMSNorm + Residuals
Maintain stable information flow and support optimization of deeper networks across all layers.
RoPE Positional Encoding
Rotary position embeddings encode token order relative to sequence position efficiently.
SwiGLU FFN
Gated linear unit activation in feed-forward layers for improved model expressivity.
The proposed vocabulary size is approximately 40,000 tokens using byte-pair encoding (BPE), representing common words efficiently while retaining the ability to process uncommon words and fragments. Tied input/output embeddings reduce total parameter count.
Experimental Phases
Tazi — Prototype Pipeline
The initial Tazi experiments verified the fundamental training pipeline on locally available hardware. Phase 2 used a 15,965,952-parameter model trained on processed OpenWebText:
- Training tokens: 120,490,409
- Validation tokens: 13,387,824
- Vocabulary: 8,000 tokens
- Effective batch size: 2,048 tokens per optimizer update
A key finding: validation loss decreased initially then began increasing after ~1,500 steps — indicating overfitting and motivating further attention to regularization, validation monitoring, and checkpoint selection.
Spark — First Experimental Model
Spark trained a ~17M parameter model on TinyStories using a Colab NVIDIA T4 GPU. This experiment established a practical text-generation workflow and surfaced a checkpoint-resume issue that highlighted the importance of preserving optimizer state, scheduler state, and RNG state alongside model weights.
| Configuration | Value |
|---|---|
| Model parameters | ~17 million |
| Vocabulary size | 8,000 tokens |
| Context length | 256 tokens |
| Hidden dimension | 384 |
| Transformer layers | 8 |
| Attention heads | 6 |
| Dropout | 0.1 |
| Max training steps | 20,000 |
| Initial learning rate | 3 × 10⁻⁴ |
Results & Roadmap
| Experiment | Metric | Recorded Result |
|---|---|---|
| Tazi Phase 2 | Parameters | 15,965,952 |
| Tazi Phase 2 | Processed tokens | 133,878,233 |
| Spark | Parameters | ~17 million |
| Spark | Best validation loss | 1.490866 |
| Spark | Training progress | Step 19,499 / 20,000 |
| W-1.1-125M | Target parameters | ~125 million |
| W-1.1-125M | Planned training tokens | ~2 billion |
| W-1.1-125M | Planned vocabulary | ~40,000 tokens |
Benchmark Comparison — W-1.1 Instruct vs GPT-2 Small
Next Steps
The immediate roadmap for W-1.1-125M:
- Finalize the 125M-parameter decoder-only architecture
- Validate the ~40,000-token BPE tokenizer
- Prepare, filter, and audit the training corpus
- Implement reliable checkpoint recovery (model + optimizer + scheduler + RNG state)
- Run a short pilot to measure throughput, memory, and loss stability
- Scale toward the 2-billion-token training target
- Run standardized evaluation on documented held-out datasets
Research attribution: Jhelum Labs · Project: W-1.1 · Document type: Technical overview and experimental research summary · Benchmark status: Internal experimental results; standardized external benchmark scores require measurement and verification.