Report field | Final record |
|---|---|
Project | w-1.1-project-spark |
Phase | Phase 3 - Complete |
Model | SparkGPT decoder-only Transformer |
Unique trainable parameters | Approximately 17.34 million |
Dataset | Approximately 500 MB TinyStories text |
Training hardware | NVIDIA Tesla T4 with CUDA and FP16 |
Completed updates | 20,000 |
Best validation loss | 1.4872 |
Final status: Phase 3 training completed successfully. The best SafeTensors checkpoint generates readable, structured short stories and is ready to be preserved as the Spark baseline for later W-1.1 experiments. |
Executive Summary
W-1.1 Spark was trained as a controlled Phase 3 experiment to measure how a compact decoder-only Transformer learns from a large, focused story corpus. The final model completed its full 20,000-step cosine-decay schedule on a Tesla T4. Its best recorded validation loss was 1.4872, and fixed-prompt testing produced fluent multi-paragraph stories with recognizable characters, settings, dialogue, emotions and conclusions.
The experiment succeeded at language fluency and narrative form. It also exposed predictable limitations of a small, single-domain model: entity drift, weak causal reasoning, occasional grammar errors, repeated templates and unreliable document-ending behavior. These findings are useful inputs for Phase 4 Spectrum and the eventual W-1.1-125M design.
1. Phase 3 Purpose
Phase 3 was designed to improve the surface quality and stability of the W-1.1 experimental model before broadening its data. Spark focused on TinyStories because the corpus provides simple vocabulary, short narrative arcs and repeated grammatical patterns that a small model can learn efficiently.
2. Experiment Objectives
- Train a compact language model on approximately 500 MB of focused story text.
- Verify stable CUDA training on a Tesla T4 using FP16 mixed precision.
- Measure training and validation behavior across a complete 20,000-step schedule.
- Protect both the best checkpoint and the latest resumable checkpoint in Google Drive.
- Evaluate grammar, story structure, repetition, semantic consistency and character tracking.
- Create a reliable baseline for comparison with Phase 4 Spectrum.
3. Model Architecture
Component | Spark configuration |
|---|---|
Architecture | Decoder-only causal Transformer |
Vocabulary | 8,000-token Byte-Level BPE |
Context length | 256 tokens |
Embedding width | 384 |
Transformer layers | 8 |
Attention heads | 6 |
Head dimension | 64 |
MLP expansion | 4x embedding width |
Activation | GELU |
Dropout | 0.10 |
Output weights | Tied with token embeddings |
Unique parameters | Approximately 17.34M |
The SafeTensors file is larger than a simple four-bytes-per-unique-parameter estimate because the tied embedding and language-model head can be represented as separate state-dictionary tensors during serialization. This does not mean the model has more unique trainable parameters.
4. Dataset and Preprocessing
The Phase 3 corpus was approximately 500 MB of TinyStories text. A custom 8,000-token Byte-Level BPE tokenizer was trained and reused for preprocessing, training and generation. Token IDs were stored as uint16 binary arrays and accessed through memory mapping to avoid loading the entire corpus into RAM.
Data component | Implementation |
|---|---|
Raw input | data/raw/data.txt |
Tokenizer | data/processed/tokenizer.json |
Training tokens | data/processed/train.bin |
Validation tokens | data/processed/val.bin |
Split | 90% training / 10% validation |
Batch sampling | Random contiguous 256-token windows |
Storage type | uint16 memory-mapped arrays |
5. Training Configuration
Setting | Value |
|---|---|
GPU | NVIDIA Tesla T4 |
Framework | PyTorch with CUDA |
Precision | FP16 automatic mixed precision |
Micro-batch size | 16 sequences |
Gradient accumulation | 8 micro-batches |
Effective tokens/update | 32,768 |
Optimizer | AdamW |
Betas | 0.9, 0.95 |
Weight decay | 0.10 |
Peak learning rate | 3e-4 |
Minimum learning rate | 3e-5 |
Warm-up | 500 steps |
Schedule | Warm-up followed by cosine decay |
Gradient clipping | 1.0 |
Validation interval | Every 250 steps |
Maximum steps | 20,000 |
6. Checkpoint and Resume Design
Spark stored checkpoints in Google Drive so a Colab interruption would not destroy training progress. The latest checkpoint restored model weights, optimizer state, completed step and best validation score. The best checkpoint was updated only when validation loss improved.
- spark_best.safetensors - final model selected by lowest validation loss.
- spark_latest.safetensors - weights from the most recent completed evaluation.
- spark_latest_optimizer.pt - AdamW state required for an accurate resume.
- spark_latest_state.json - completed step, configuration and best validation score.
7. Final Training Results
Metric | Final result | Interpretation |
|---|---|---|
Training completion | 20,000 / 20,000 | Full schedule completed |
Best validation loss | 1.4872 | Lowest recorded Phase 3 validation loss |
Best checkpoint | spark_best.safetensors | Use for generation and final evaluation |
Latest checkpoint | spark_latest.safetensors | Represents completion of the run |
Checkpoint size | 77.868 MB | Serialized SafeTensors weights |
Optimizer state size | 132.345 MB | AdamW resume state |
Training status | Complete | No remaining updates |
8. Learning Behavior
Training remained stable under a peak learning rate of 3e-4, 500-step warm-up and cosine decay toward 3e-5. Earlier observations showed validation loss falling from approximately 4.1425 at step 250 to 1.6926 at step 5,750, then improving further to 1.5026 around step 15,500 and finally reaching a best recorded value of 1.4872 by completion.
Short increases in validation loss occurred during training, but later evaluations recovered and produced new best checkpoints. This behavior indicates evaluation noise and a late-stage plateau rather than a damaging loss explosion. Saving the best checkpoint protected the strongest model throughout the run.
Result: The selected learning-rate schedule was appropriate for Spark. It supported rapid early learning, stable middle-stage improvement and smaller late-stage refinements without NaN or infinite losses. |
9. Inference Configuration
Decoding control | Test value |
|---|---|
Maximum new tokens | 200 |
Temperature | 0.7 |
Top-k | 40 |
Top-p | 0.9 |
Repetition penalty | 1.15 |
Checkpoint | spark_best.safetensors |
Device | CUDA |
10. Text-Generation Evaluation
Four fixed prompts were used to test story fluency, structure, consistency and creativity. The model produced readable multi-paragraph stories in every case.
Test | Prompt | Assessment |
|---|---|---|
1 | Once upon a time, a little boy named Tom | 8/10 - coherent mountain adventure with a complete ending |
2 | Lily found a mysterious box under her bed. | 6/10 - fluent, but the box, hat and trap become confused |
3 | One sunny morning, the little rabbit | 6.5/10 - creative plot, but the dragon later becomes a bird |
4 | The girl was afraid of the dark, but | 8/10 - coherent comforting story with dialogue and closure |
Representative Success: Tom's Mountain
The model introduced Tom's goal, followed his climb, described the view at the top and ended with a simple lesson about exploration. Character identity and the central setting remained stable through the story. This was the strongest evidence that Spark learned the basic beginning-middle-ending pattern of TinyStories.
Representative Limitation: Lily's Box
The prose remained fluent, but the mysterious box changed into hats, a trap and a cat-related incident. The response then introduced police without a convincing causal reason. This shows that Spark can imitate grammatical story form while still losing track of object identity and event logic.
Qualitative Capability Summary
Capability | Observed level | Evidence |
|---|---|---|
Grammar and punctuation | Good | Mostly correct sentences and dialogue punctuation |
Paragraph structure | Good | Multi-paragraph stories with readable progression |
Narrative style | Good | Strong TinyStories-like tone and moral endings |
Character consistency | Moderate | Usually stable in short spans; can drift later |
Causal reasoning | Limited | Events sometimes follow without a logical cause |
Entity tracking | Limited | Objects and creatures can change identity |
Long generation | Limited | Consistency weakens as more tokens are generated |
Overall qualitative generation rating: approximately 7/10 for TinyStories-style text. This score measures writing quality within Spark's training domain, not general intelligence, factual knowledge or instruction-following ability.
11. Confirmed Strengths
- Produces fluent English with mostly correct capitalization and punctuation.
- Builds recognizable story openings, conflicts, actions and conclusions.
- Maintains child-friendly vocabulary and an appropriate TinyStories tone.
- Generates multiple paragraphs and simple dialogue.
- Often provides an emotional or moral ending.
- Avoids the severe gibberish and endless loops seen in earlier prototypes.
12. Remaining Limitations
Limitation | Observed evidence | Implication |
|---|---|---|
Entity drift | Dragon changes into a giant bird | Weak internal state tracking |
Object confusion | Box, hat and trap become mixed | Form is stronger than meaning |
Causal inconsistency | Police appear after a cat-related injury | Limited reasoning capacity |
Grammar errors | Example: 'had already ate' | Language learning is incomplete |
Template repetition | Frequent 'Once upon a time' restarts | Corpus patterns are strongly copied |
No reliable story stop | A second story begins after the first | EOS handling must improve |
Single-domain knowledge | Strongest only on short stories | Not suitable for code or broad QA |
13. Important Preprocessing Lesson
The preprocessing pipeline did not explicitly insert an <eos> token between every story, and the generation loop did not stop when an end token appeared. As a result, Spark sometimes completed one story and immediately began another. Future phases should define document boundaries explicitly during preprocessing and terminate generation when the end-of-sequence token is produced.
14. What Phase 3 Proved
Phase outcome: COMPLETE. A focused, clean corpus can teach a small model fluent grammar, paragraph structure and recognizable narrative patterns. However, fluency alone does not guarantee logical consistency, entity tracking, factual reliability or broad capability. |
15. Lessons for W-1.1-125M
- Preserve warm-up, cosine decay, gradient clipping and best-checkpoint selection.
- Evaluate with fixed prompts and seeds in addition to validation loss.
- Use explicit document separators and train reliable EOS behavior.
- Track per-domain validation loss when the dataset becomes multi-domain.
- Keep clean data and deduplication as first-class parts of the training system.
- Do not confuse fluent text with reasoning; measure both separately.
- Retain Spark as a baseline instead of overwriting it during later phases.
16. Preserved Phase 3 Artifacts
Artifact | Purpose |
|---|---|
spark_best.safetensors | Best Phase 3 weights for generation and comparison |
spark_latest.safetensors | Weights at the end of the 20,000-step run |
spark_latest_optimizer.pt | Optimizer state for reproducibility or continuation |
spark_best_state.json | Best validation record and configuration |
spark_latest_state.json | Final completed step and training state |
tokenizer.json | Exact 8,000-token BPE used by the model |
meta.json / train.bin / val.bin | Processed data metadata and token arrays |
W-1.1-spark.docx | Final Phase 3 technical record |
17. Next Step - Phase 4 Spectrum
The next controlled experiment is w-1.1-project-spectrum. It will test a balanced multi-domain dataset containing education, code and programming, web text, news and research. To isolate the effect of data diversity, the first Spectrum run should keep Spark's architecture, tokenizer size, context length, optimizer, learning-rate schedule and token budget as consistent as practical.
Final Conclusion
W-1.1 Spark successfully completed Phase 3. The approximately 17.34M-parameter model finished 20,000 CUDA training updates on a Tesla T4 and achieved a best recorded validation loss of 1.4872. Its best checkpoint produces readable, structured and often engaging short stories. The model remains limited in semantic consistency, entity tracking, causal reasoning and document termination, but those limitations are now measured rather than guessed. Spark should be frozen as the Phase 3 baseline and used to guide Spectrum and the final W-1.1-125M system.