Tiny Prototype

1. Purpose of this prototype

The W-1.1 tiny prototype is a small GPT-2-style decoder-only Transformer created to verify the complete language-model pipeline before spending time and GPU money on the final 125-million-parameter model. It is a pipeline test, not a production language model.

2. Current prototype specification

3. Working environment

All commands and prototype edits should be performed in the following Windows folder:

C:\Users\MSI-LAPTOP\Downloads\W-1.1-tiny-prototype

4. Exact setup and test sequence

1. Open the extracted W-1.1-tiny-prototype folder in Visual Studio Code.
2. Open Terminal → New Terminal.
3. Move into the project folder.

cd "C:\Users\MSI-LAPTOP\Downloads\W-1.1-tiny-prototype"

1. Create the Python virtual environment if it does not already exist.

python -m venv .venv

1. Activate the virtual environment.

.venv\Scripts\activate

1. Upgrade pip and install the project requirements.

python -m pip install --upgrade pip
pip install -r requirements.txt

1. Run the automated model test.

python test_model.py

1. Train the tiny prototype.

python train.py --steps 300

1. Generate text from the saved checkpoint.

python generate.py --prompt "Wasif " --tokens 200

5. GPU training verification

Before assuming that training is using the GPU, run this command inside the activated virtual environment:

python -c "import torch; print('CUDA available:', torch.cuda.is_available()); print('GPU:', torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'Not detected'); print('PyTorch CUDA:', torch.version.cuda)"

Expected GPU-ready output:

CUDA available: True
GPU: NVIDIA ...
PyTorch CUDA: <version>

• If CUDA is True, run train.py normally; the script should select device: cuda.
• Use nvidia-smi in another terminal to observe GPU memory and utilization.
• If CUDA is False, first check the installed PyTorch build and NVIDIA driver before changing code.

6. Understanding the training results

6.1 Why loss falls very quickly on tiny data

A very small dataset is repeated many times. The tiny model can memorize these repeated characters and sequences, so training loss decreases rapidly. This proves that optimization works, but it does not prove that the model has learned general English.

6.2 Why loss fluctuates on a larger document

When the architecture document is used as training data, batches contain more varied and difficult sequences. Individual batch loss can decrease and then increase. The moving average over 50–100 steps is more useful than one loss value.

7. Why the generated text looks imperfect

Outputs such as random-looking character sequences are expected because this prototype has very few parameters, a character tokenizer, a tiny dataset, and only a small number of steps. Its purpose is engineering validation rather than fluent generation.

• Increase the dataset to at least 1–10 MB of clean, varied text for a stronger prototype experiment.
• Train for more steps only after confirming that the data and checkpoints are correct.
• Use a lower generation temperature, such as 0.7, to reduce randomness if the script supports it.
• Do not judge the final W-1.1-125M architecture from the quality of this 110K-parameter character model.

8. Dataset rules for prototype experiments

• Use clean UTF-8 plain text.
• Include varied sentences and paragraphs instead of repeating one passage excessively.
• Keep a separate validation portion, ideally about 10% of the data.
• Do not change or rebuild the tokenizer in the middle of one experiment.
• Start a fresh checkpoint when the dataset or tokenizer changes materially.
• Record the dataset name, size, learning rate, steps, and final losses for every run.

9. Recommended stability settings

If training becomes unstable on the larger prototype data, test the following controlled changes one at a time:

python train.py --steps 3000 --learning-rate 0.0001

• Use gradient clipping with a maximum norm of 1.0.
• Shuffle samples or randomize starting positions.
• Log averaged training loss and validation loss every 50–100 steps.
• Confirm that the text contains enough tokens for the configured context length.

10. Prototype experiment record

11. Files in the prototype

12. Current status and next milestone

• Python 3.11.9 confirmed as compatible.
• Virtual environment created and dependencies installed.
• Automated model test passed.
• Training loss decreases on the tiny dataset.
• Checkpoint-based text generation works.
• Loss fluctuation on a larger document has been observed and explained.

13. Decision gate before scaling

Proceed to the 10–20M prototype only when all of the following checks are complete:

• CUDA detection is confirmed or the chosen CPU/GPU device is documented.
• Training and validation losses are logged separately.
• Checkpoint save and resume produce consistent results.
• Generation works from the saved checkpoint.
• The training dataset is clean, split, and reproducible.
• No NaN loss or unexplained long-term divergence occurs.