rihla-llm is a domain language model I built for a hospitality-tech client in Saudi Arabia, a freelance engagement in December 2025. It answers booking and guest-support questions in Gulf Arabic, Modern Standard Arabic, English, Urdu and Roman Urdu, often within a single message. The code is open on GitHub. This article walks through the brief, the 15-stage pipeline, and the decisions I would defend or change. It is my own project, so read it as a design write-up, not an independent review.
The brief: four requirements that no single technique covers
The client ran booking and guest support for hotels in Makkah and Madinah. Off-the-shelf models handled English well and everything else poorly. The requirements were:
- Answer from the client's own policies, rates and availability, and never invent a price or a room.
- Keep guest ID numbers, phone numbers and emails out of training data, logs and responses.
- Run on hardware the client could afford, fast enough for live chat.
- Keep improving after launch without a manual retraining project each time.
Fine-tuning alone fixes the first one only partly and does nothing for the other three. That is why the result is a pipeline, not a fine-tune.
Part A: data
Most of the quality in a domain model is decided before training starts. Five stages sit here, and one helper, split.py, freezes the evaluation set.
01 Ingest and 02 PII scrub
Chat and ticket exports in JSONL or CSV become question and answer pairs. PDFs and text become documents for retrieval. Every record gets a language tag. The PII scrubber then runs rules for Saudi national IDs and Iqama numbers, Saudi and Pakistani mobile numbers, international numbers and emails, including numbers written in Arabic-Indic digits. An optional NER model catches names. The scrubber runs before deduplication on purpose: once every phone number is replaced by the same placeholder, near-identical tickets collapse into one template.
03 Dedup with MinHash-LSH
Support logs repeat the same templates thousands of times. Left alone, duplicates skew the model toward boilerplate and leak into the evaluation set. I wrote MinHash with locality-sensitive hashing from scratch, using 128 permutations in 16 bands of 8 rows, streamed so it does not need the whole corpus in memory. The first occurrence wins.
04 Synthetic data
Real tickets seed a Self-Instruct style generator. A teacher model judges each example from 1 to 5, and only 4 and above are kept. The same stage writes preference pairs for later, where the rejected answer has an invented price, the wrong language, or a dodge instead of an answer.
05 Tokenizer extension
The base tokenizer split many Arabic and Urdu words into three or more pieces, which wastes context and slows training. This stage adds the most expensive Arabic-script words as single tokens that match only at the start of a word, and initialises each new embedding as the mean of the pieces it replaces. A suffixed form such as بالمسجدين still uses the new بالمسجد token, and decoding returns exactly the original text. One pitfall is worth knowing: the Hugging Face single_word=True option silently stops these space-prefixed tokens from ever matching inside a sentence. A test in the repository covers it.
Part B: training and a release gate
06 SFT with QLoRA
Supervised fine-tuning uses QLoRA with rank 64 on all linear layers, via TRL, with sequence packing and FSDP for multi-GPU runs. When the vocabulary was extended, the embedding and output layers are trained as well, because LoRA adapters alone cannot move rows that did not exist before. An optional mix of general data limits forgetting. The method is from Dettmers et al., and I chose it because the budget was tight and adapters let one base model serve several clients.
07 DPO
Direct Preference Optimization (Rafailov et al.) runs on the merged SFT model with beta 0.1. With adapters, disabling them gives the reference model, so no second copy of the weights is loaded. I picked DPO over PPO-based RLHF because there is no reward model to train and keep stable, and preference data, not the optimiser, was the bottleneck.
08 Quantize, 09 Eval, 10 Red-team
The model is quantized to int4 with AWQ (Lin et al.) through llm-compressor, calibrated on the client's own Arabic, Urdu and English conversations instead of English web text. The evaluation harness then runs every suite against the int4 build that will ship, so quantization loss is measured and not assumed away. A gate fails the build if any suite drops below the accepted baseline. Red-team suites attack PII extraction and jailbreaks in all three languages, because public safety sets are mostly English-only.
Part C: serving and operations
11 Registry
Every model file gets a SHA-256 entry in a manifest, and the manifest carries an HMAC signature. make serve refuses to start if the files on disk do not match. Eval scores and the training config are logged to MLflow, so any running model can be traced to the data and settings that produced it.
12 Serving with vLLM
vLLM provides continuous batching, built on PagedAttention. I also turned on n-gram speculative decoding, also called prompt lookup decoding. Answers grounded in retrieved passages copy a lot of text from those passages, so the drafts are accepted often. It needs no separate draft model, which matters here because a draft model would have to share the extended vocabulary.
13 Retrieval
Prices and availability change daily, so they are retrieved and never trained in. Retrieval combines BM25, written from scratch with Arabic article stripping, with multilingual dense embeddings. The two rankings merge by reciprocal rank fusion, and a cross-encoder reranks the top results.
14 Guardrails
PII is masked on the way in and on the way out. The output check is the part I like most: any number in an answer that appears in neither the retrieved passages nor the guest's question is treated as invented, and the whole answer is replaced with a fallback in the guest's language. A model can still be wrong in words, but it cannot quote a rate that nothing supports.
15 Observability and the feedback loop
The gateway records p50 and p95 latency, tokens and cost per chat, the share of questions with no matching passage, guardrail flag rates, thumbs-up rate, canary against stable traffic, and language-mix drift between this window and the last. It also routes a fixed share of sessions to a candidate model for canary testing, and a session always hits the same model.
The loop closes in make feedback. A thumbs-down with a staff correction makes the correction the chosen answer. The same question rated up once and down once makes a ranked pair. Prompts are rebuilt with the exact system prompt and passages the model saw. Then make train eval release produces a new candidate, and it ships only if stages 09 and 10 pass again.
What a single request does
- The guest sends a message in any mix of Arabic, English and Urdu.
- The gateway masks PII, identifies the language and assigns a canary bucket.
- The retriever returns the top passages through BM25 and dense search, fusion and rerank.
- vLLM answers from the shared system prompt, the passages and the question.
- The guardrails check every number and mask PII again, then the guest gets the answer with a trace ID.
- Latency, tokens and flags are logged, and a later rating can come back through
/feedback.
Five hard problems
Arabic tokenization. Covered in stage 05: word-initial tokens, mean-initialised embeddings and a lossless decode test.
Code-switching. One message can mix Arabic script, English and Roman Urdu. lang.py labels each run separately and splits Arabic-script runs at sentence ends, because Arabic and Urdu share a script and a single label per message would be wrong half the time.
Eval contamination. A naive split leaks near-duplicates into the test set. The evaluation split is frozen by hash on the first run, only real pairs go into it, and training data is deduplicated against it before every run.
Hallucinated prices. No fine-tune knows today's rates. Prices come only from retrieval, DPO pairs penalise invented numbers, and the output guard replaces any answer containing an unsupported number.
Latency on a budget. int4 AWQ fits one mid-range GPU, vLLM batches continuously, and n-gram speculative decoding speeds up the copy-heavy answers.
Results: read this table carefully
The repository README ships a results table, and I need to be straightforward about it. The file carries a note telling the reader to replace every number with their own measured values before publishing. Treat the figures below as the illustrative numbers in the template, not as an audited benchmark, and run make eval on your own data before quoting any of them.
| Metric | Base model | rihla-llm |
|---|---|---|
| Arabic-script tokens per word | 2.9 | 1.7 |
| Domain QA pass rate (judge 4 or higher) | 41% | 78% |
| Grounding suite pass rate | 52% | 97% |
| PII red-team pass rate | 89% | 99.6% |
| p95 gateway latency | 1.9 s | 0.6 s |
| GPU memory at serve time | 28 GB | 7.8 GB |
Design decisions in one place
- QLoRA over full fine-tuning: budget-bound, and one base model can serve several clients through adapters.
- DPO over PPO-based RLHF: no reward model to maintain.
- Retrieval for anything that changes: the model learns how to answer, not what today's rate is.
- Evaluate the artifact that ships: the gate runs on the int4 build, not the bf16 model it came from.
- One prompt format:
prompts.pyfeeds SFT data, DPO pairs, feedback pairs, evals and the gateway, so the model is never served a format it was not trained on.
Running it and testing it
Install with pip install -e ".[ingest,train,quant,serve,ops,dev]", put the client exports in raw/, point the teacher setting in configs/rihla.toml at a strong model, then run make data, make train, make eval, make release and make serve. The first eval run needs you to review the report and accept it as the baseline. After that, make metrics gives a weekly health report. Twenty-two tests run on CPU only: PII patterns, dedup, the frozen split, language ID, retrieval, guardrails, the gate, signing, canary routing and the gateway through FastAPI's test client. Stages 05 to 07 are tested on a tiny model built inside the test, and the vocabulary extension has to lower tokens per word and decode losslessly.
What I would do next
- Put retrieved passages into the SFT examples so training matches the RAG prompt even more closely.
- Distil the int4 model into a smaller student for a CPU fallback.
- Replace thumbs ratings with pairwise comparisons, which give a cleaner DPO signal.
Frequently asked questions
Which base model does rihla-llm use?
The default in configs/rihla.toml is Qwen2.5-7B-Instruct. It is a single setting, and the code is built on Hugging Face transformers, TRL and PEFT, so another compatible open-weight model can be swapped in, though the tokenizer stage and the tests were written against the default.
Can I reuse it for another language or industry?
Yes. The pipeline is domain-neutral apart from the example evaluation suites and the PII rules, which target Saudi and Pakistani formats. Replace those, supply your own exports, and rerun the gate.
Why not just use a large hosted model with RAG?
For this client the answer was cost, latency and data control. A quantized model on one GPU keeps guest data in-house and answers quickly. A hosted model with good retrieval is a reasonable choice when those limits do not apply, and the retrieval and guardrail stages of this repository would still be useful in front of it.
If you want something like this built, or a review of an existing multilingual or RAG system, see my services or get in touch. For more of my own builds, read about PrintDrop and RAG pipelines and AI agents.
Sources
- Dettmers, Pagnoni, Holtzman, Zettlemoyer. QLoRA: Efficient Finetuning of Quantized LLMs, 2023.
- Rafailov et al. Direct Preference Optimization: Your Language Model is Secretly a Reward Model, 2023.
- Lin et al. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration, 2023.
- Kwon et al. Efficient Memory Management for Large Language Model Serving with PagedAttention, 2023.
- Wang et al. Self-Instruct: Aligning Language Models with Self-Generated Instructions, 2022.
- Cormack, Clarke, Buettcher. Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods, SIGIR 2009.
- Saxena. Prompt Lookup Decoding.
- Faizan. rihla-llm on GitHub.