Project case study · Flagship model · Persian NLP · From-scratch training
Persian GPT from scratch: a 110M parameter language model
An end-to-end Persian language-modeling experiment, from preparing a cleaned Wikipedia corpus and building a tokenizer to training a decoder-only Transformer and comparing held-out perplexity against n-gram baselines.
01 · Context
Research question
Persian has fewer openly documented from-scratch training examples than high-resource languages. The goal here was to build a transparent, reproducible baseline workflow and document what the resulting language-model metric does and does not establish.
02 · Methods
Approach
- Cleaned and prepared 855,724 Persian Wikipedia documents, totaling about 244 million training tokens.
- Trained a custom 32,000-token byte-pair encoding tokenizer on the prepared corpus.
- Implemented and trained a 110.4M-parameter decoder-only Transformer in PyTorch for two epochs (7,450 training steps).
- Evaluated the held-out split with loss and perplexity, then compared perplexity with unigram and interpolated 5-gram baselines.
03 · Evaluation
Reported results
- Held-out test perplexity: 14.52; validation perplexity: 14.02.
- The held-out interpolated 5-gram baseline perplexity was 171.60, and the unigram baseline was 1,590.26.
- The reported language-model metrics apply only to the documented corpus split and tokenizer.
04 · Scope
Limitations
This is a small research portfolio model trained on Wikipedia text. It is not an instruction-following or factual question-answering system, has no retrieval grounding, and has not been evaluated for broad Persian fluency, safety, or downstream usefulness. Perplexity does not establish factuality or real-world performance.