Final Report

Prepared for Jhelum Labs

Finalized 20 August 2026

Document scope and evidence

This report consolidates the recorded W-1.1 experiment lineage, final dataset preparation, custom architecture, H100 pretraining logs, saved artifacts, qualitative text-generation tests, failure recovery, and the remaining production roadmap. Numeric claims about the final run are drawn from the notebook outputs and screenshots preserved during the project conversation.

Contents

• 1. Executive summary
• 2. Experimental lineage
• 3. Final system design
• 4. Dataset engineering
• 5. H100 training run
• 6. Results and interpretation
• 7. Generation-quality research
• 8. Failure recovery and reproducibility
• 9. Artifact inventory
• 10. Production readiness
• 11. Benchmark and SFT plan
• 12. Conclusions
• Appendix A. Recorded logs
• Appendix B. References

1. Executive summary

The model is a decoder-only Transformer implemented from scratch with RMSNorm, rotary positional embeddings, causal scaled dot-product attention, SwiGLU feed-forward blocks, and tied input/output embeddings. The final run used an NVIDIA H100 and stabilized near 375,000–377,000 tokens per second after startup.

The strongest recorded checkpoint occurred at step 2,750 with validation loss 3.3804, corresponding to perplexity 29.38. Later validation measurements rose, so best.safetensors—not the final-step weights—is the recommended inference and fine-tuning starting point. The behavior is consistent with a functioning base language model: it generates readable, topical paragraphs but can hallucinate institutions, studies, quotations, and citations.

Research conclusion

The pretraining phase is technically successful, but the model is not yet production-ready. It remains a base continuation model rather than an instruction-following assistant. Production requires standardized benchmarks, test-set evaluation, Hugging Face-compatible packaging, supervised instruction tuning, safety evaluation, quantization, API serving, and monitoring.

2. Experimental lineage

W-1.1 was developed through progressively larger experiments. Each phase tested a different risk: whether the pipeline runs, whether validation behavior can be observed, whether a broader tokenizer and dataset remain stable, and whether the final architecture can train efficiently at scale.

Key experimental decisions

• Use a raw-first pipeline with deduplication, then tokenize into uint16 binary files.
• Reserve exact validation and test targets instead of relying on approximate file percentages.
• Use a modern decoder-only design rather than copying GPT-2 exactly.
• Train the full final model on H100 while keeping local 4 GB GPU inference as a deployment constraint.
• Preserve both best and final checkpoints, plus a resumable latest.pt training checkpoint.

3. Final system design

Jhelum Labs 125M is a causal decoder-only Transformer. Given a sequence of token IDs, it predicts the probability distribution of the next token at every position. Training minimizes token-level cross-entropy against the sequence shifted by one token.

Parameter accounting

The implemented architecture was checked before training. The recorded count was exactly 123,688,704 parameters, within the intended 123–125M range. Tied embeddings avoid maintaining a separate 32,000 × 768 output matrix.

4. Dataset engineering

The final raw text file was approximately 17.005 GB. It was processed document-by-document with the custom 32,000-token tokenizer. Exact validation and test token quotas were filled while all remaining tokens were assigned to training.

Figure 1. Exact token allocation after the interrupted preprocessing run was finalized from the existing binary files.

Preprocessing interruption and recovery

Cell 9 stopped after creating valid binary files. Instead of restarting the entire 17 GB preprocessing job, the existing train.bin, val.bin and test.bin files were inspected and finalized. The recovered counts were written to meta.json and manifest.json. This preserved 2.825B training tokens and avoided an expensive rebuild.

Data-quality limitation

Qualitative generations resemble web articles and occasionally invent studies, authors, journals and funding sources. This indicates that the corpus taught useful grammar and topical associations but retained noisy web-style patterns. Future corpus versions should strengthen source filtering, document quality scoring, language identification, near-duplicate removal, citation-pattern filtering, and personally identifiable information checks.

5. H100 training run

The final job used a detached Modal function with one NVIDIA H100. Dataset files were staged from Google Drive to a persistent Modal Volume, copied to local SSD, verified, and then consumed by the training loop. The notebook itself could remain on CPU because the detached training function owned the H100 container.

Figure 2. Recorded learning rate and throughput. Startup was slow, then throughput stabilized around 375k–377k tokens/s.

The difference between nominal scheduled tokens and available train tokens is 139,760 tokens—less than one optimizer step. The loader’s modulo behavior means the final step can wrap to the beginning of the binary dataset, effectively completing about one epoch.

Compilation failures and repair

1. The first H100 attempt failed because CUDA Graph outputs were overwritten across repeated compiled invocations during gradient accumulation.
2. A subsequent torch.compile change supplied both mode and options, which PyTorch rejects.
3. The compile configuration was repaired by disabling CUDA Graphs through a single valid compile configuration.
4. The resumed job started correctly, reached step 1, then sustained full H100 throughput and completed training.

6. Results and interpretation

Figure 3. Selected observed loss points from the recorded console logs. The green marker is the best validation checkpoint.

Figure 4. Validation perplexity computed as exp(validation loss). Values are specific to this tokenizer and validation corpus.

Recorded validation checkpoints

What the curve suggests

• The model learned rapidly: loss fell from 10.5419 at step 1 to below 4 by roughly step 900.
• The best generalization point appeared around the midpoint of the one-epoch run at step 2,750.
• A noisy instability band appeared around steps 2,790–2,880, where loss and gradient norms rose sharply.
• Later train losses sometimes improved while validation remained worse than the best checkpoint, so final-step weights should not automatically replace best weights.
• The validation series is sparse. More frequent, fixed-batch evaluation would separate genuine drift from validation sampling noise.

7. Generation-quality research

The final model generated coherent paragraphs from open-ended prompts. It maintained local topic consistency, used headings and bullet formatting, and produced mostly grammatical English. These are strong signs that next-token pretraining succeeded.

Sample A — artificial intelligence

The model continued with claims about learning from less data, an MIT experiment, Nature Communications, speech recognition, a named researcher and research funding. The prose was fluent and structurally plausible, but the named study, quotation and funding claims were not verified.

Sample B — education

The model produced sections titled “What is the future of education?” and “How do I get started?”, followed by bullets. It demonstrated formatting competence but included vague reasoning and a grammar error (“it’s needs” instead of “its needs”).

Sample C — young scientist

The continuation began as a narrative, then shifted into a fabricated biography and web-post language about homemade drugs. This shows topic drift, data-mixture artifacts and a safety risk. The model should not be publicly exposed without moderation and post-training.

Qualitative scorecard

8. Failure recovery and reproducibility

The project encountered infrastructure and code failures that are part of the final research record. Each failure produced a concrete engineering lesson.

Reproducibility controls now present

• Tokenizer and tokenizer metadata stored separately from weights.
• meta.json and manifest.json record finalized binary split information.
• config.json records final architecture and training configuration.
• metrics.json preserves the training/validation record.
• best, final and resumable checkpoints serve different recovery purposes.
• A local VS Code CUDA notebook reconstructs the exact architecture and verifies the parameter count before loading weights.

9. Artifact inventory

Figure 5. Displayed Google Drive file sizes for the final model artifacts.

Precision and deployment size

10. Production readiness

Pretraining completion is not equivalent to production readiness. The following gate separates work already completed from work still required.

Recommended production architecture

• Immutable model registry: base, instruct, quantized and deployment versions with checksums.
• Inference service: a single model worker with bounded context, token limits and request batching.
• API layer: authentication, quotas, timeouts, schema validation and audit logging.
• Safety layer: input classification, output moderation, blocked-token/pattern controls and escalation.
• Observability: latency percentiles, GPU memory, generation length, error rate, safety flags and version tags.
• Rollback: one-click return to the previous known-good model package.

11. Benchmark and instruction-tuning plan

11.1 Base-model evaluation

The first post-training phase should measure the frozen base model before any fine-tuning. The custom architecture must be packaged for Hugging Face Transformers or connected to EleutherAI’s lm-evaluation-harness through a custom adapter.

11.2 Supervised instruction tuning

After freezing and reporting the base scores, fine-tune from best.safetensors rather than the final checkpoint. Preserve the original checkpoint permanently.

11.3 Local 4 GB GPU target

The current FP32 checkpoint is approximately 472 MB and FP16 parameters are approximately 236 MiB, so the 123.7M model comfortably fits in 4 GB VRAM for inference. The main inefficiency is generation without a key/value cache: each new token recomputes the entire prefix. A production inference rewrite should add KV caching before latency optimization or deployment claims.

12. Conclusions

W-1.1 progressed from small experimental models to a complete 123.69M-parameter open-domain base language model. The final system successfully learned next-token prediction over 2.825B training tokens, sustained high H100 throughput, produced coherent English continuations, and saved a complete set of best, final and resumable artifacts.

The strongest evidence of success is not a single attractive sample. It is the combination of a decreasing training loss, a measurable best validation checkpoint, stable high-throughput execution, exact artifact preservation, and repeatable local loading. The strongest evidence of remaining work is equally clear: hallucinated citations, topic drift, unsafe continuations, sparse evaluation and the absence of instruction tuning.

Immediate next actions

• Calculate exact test.bin loss and perplexity.
• Create the Hugging Face-compatible Jhelum Labs 125M package and model card.
• Run a 100-example benchmark smoke test, then full benchmark suites.
• Build and audit the SFT dataset before starting instruction tuning.
• Add KV-cache generation, quantize, and measure GTX 1650 latency and VRAM.

Appendix A. Recorded final-run logs

The following table transcribes representative console lines that were visible in the recorded project outputs. It is not a complete every-step log.

Validation/checkpoint events

• Step 1,000 — validation 3.7752; new best; checkpoint committed.
• Step 2,750 — validation 3.3804; new best; checkpoint committed.
• Step 3,000 — validation 3.8948; checkpoint committed.
• Step 4,000 — validation 4.0864.
• Step 5,000 — validation 3.6045.
• Step 5,389 — training completed and final artifacts synchronized.

Appendix B. References

Model identity