Back to Blog
Projects

How I Built rihla-llm: A Multilingual Arabic, Urdu and English LLM From Data to Production

A guest writes in Gulf Arabic, switches to Roman Urdu, and asks for a price. Here is the 15-stage pipeline I built to answer from real data, protect personal details and keep improving after launch.

Cover card reading A multilingual LLM for Arabic, Urdu and English, with 15 numbered pipeline stages in three lanes
rihla-llm in one picture: 15 stages across data, training and serving.

rihla-llm is a domain language model I built for a hospitality-tech client in Saudi Arabia, a freelance engagement in December 2025. It answers booking and guest-support questions in Gulf Arabic, Modern Standard Arabic, English, Urdu and Roman Urdu, often within a single message. The code is open on GitHub. This article walks through the brief, the 15-stage pipeline, and the decisions I would defend or change. It is my own project, so read it as a design write-up, not an independent review.

The brief: four requirements that no single technique covers

The client ran booking and guest support for hotels in Makkah and Madinah. Off-the-shelf models handled English well and everything else poorly. The requirements were:

  1. Answer from the client's own policies, rates and availability, and never invent a price or a room.
  2. Keep guest ID numbers, phone numbers and emails out of training data, logs and responses.
  3. Run on hardware the client could afford, fast enough for live chat.
  4. Keep improving after launch without a manual retraining project each time.

Fine-tuning alone fixes the first one only partly and does nothing for the other three. That is why the result is a pipeline, not a fine-tune.

Diagram of the rihla-llm pipeline: 15 stages across data, training and serving, with a feedback loop from observability back to DPO
The pipeline diagram from the repository. A marker walks the 15 stages, then follows the dashed path from observability back to DPO.

Part A: data

Most of the quality in a domain model is decided before training starts. Five stages sit here, and one helper, split.py, freezes the evaluation set.

01 Ingest and 02 PII scrub

Chat and ticket exports in JSONL or CSV become question and answer pairs. PDFs and text become documents for retrieval. Every record gets a language tag. The PII scrubber then runs rules for Saudi national IDs and Iqama numbers, Saudi and Pakistani mobile numbers, international numbers and emails, including numbers written in Arabic-Indic digits. An optional NER model catches names. The scrubber runs before deduplication on purpose: once every phone number is replaced by the same placeholder, near-identical tickets collapse into one template.

03 Dedup with MinHash-LSH

Support logs repeat the same templates thousands of times. Left alone, duplicates skew the model toward boilerplate and leak into the evaluation set. I wrote MinHash with locality-sensitive hashing from scratch, using 128 permutations in 16 bands of 8 rows, streamed so it does not need the whole corpus in memory. The first occurrence wins.

04 Synthetic data

Real tickets seed a Self-Instruct style generator. A teacher model judges each example from 1 to 5, and only 4 and above are kept. The same stage writes preference pairs for later, where the rejected answer has an invented price, the wrong language, or a dodge instead of an answer.

05 Tokenizer extension

The base tokenizer split many Arabic and Urdu words into three or more pieces, which wastes context and slows training. This stage adds the most expensive Arabic-script words as single tokens that match only at the start of a word, and initialises each new embedding as the mean of the pieces it replaces. A suffixed form such as بالمسجدين still uses the new بالمسجد token, and decoding returns exactly the original text. One pitfall is worth knowing: the Hugging Face single_word=True option silently stops these space-prefixed tokens from ever matching inside a sentence. A test in the repository covers it.

Part B: training and a release gate

06 SFT with QLoRA

Supervised fine-tuning uses QLoRA with rank 64 on all linear layers, via TRL, with sequence packing and FSDP for multi-GPU runs. When the vocabulary was extended, the embedding and output layers are trained as well, because LoRA adapters alone cannot move rows that did not exist before. An optional mix of general data limits forgetting. The method is from Dettmers et al., and I chose it because the budget was tight and adapters let one base model serve several clients.

07 DPO

Direct Preference Optimization (Rafailov et al.) runs on the merged SFT model with beta 0.1. With adapters, disabling them gives the reference model, so no second copy of the weights is loaded. I picked DPO over PPO-based RLHF because there is no reward model to train and keep stable, and preference data, not the optimiser, was the bottleneck.

08 Quantize, 09 Eval, 10 Red-team

The model is quantized to int4 with AWQ (Lin et al.) through llm-compressor, calibrated on the client's own Arabic, Urdu and English conversations instead of English web text. The evaluation harness then runs every suite against the int4 build that will ship, so quantization loss is measured and not assumed away. A gate fails the build if any suite drops below the accepted baseline. Red-team suites attack PII extraction and jailbreaks in all three languages, because public safety sets are mostly English-only.

Part C: serving and operations

11 Registry

Every model file gets a SHA-256 entry in a manifest, and the manifest carries an HMAC signature. make serve refuses to start if the files on disk do not match. Eval scores and the training config are logged to MLflow, so any running model can be traced to the data and settings that produced it.

12 Serving with vLLM

vLLM provides continuous batching, built on PagedAttention. I also turned on n-gram speculative decoding, also called prompt lookup decoding. Answers grounded in retrieved passages copy a lot of text from those passages, so the drafts are accepted often. It needs no separate draft model, which matters here because a draft model would have to share the extended vocabulary.

13 Retrieval

Prices and availability change daily, so they are retrieved and never trained in. Retrieval combines BM25, written from scratch with Arabic article stripping, with multilingual dense embeddings. The two rankings merge by reciprocal rank fusion, and a cross-encoder reranks the top results.

14 Guardrails

PII is masked on the way in and on the way out. The output check is the part I like most: any number in an answer that appears in neither the retrieved passages nor the guest's question is treated as invented, and the whole answer is replaced with a fallback in the guest's language. A model can still be wrong in words, but it cannot quote a rate that nothing supports.

15 Observability and the feedback loop

The gateway records p50 and p95 latency, tokens and cost per chat, the share of questions with no matching passage, guardrail flag rates, thumbs-up rate, canary against stable traffic, and language-mix drift between this window and the last. It also routes a fixed share of sessions to a candidate model for canary testing, and a session always hits the same model.

The loop closes in make feedback. A thumbs-down with a staff correction makes the correction the chosen answer. The same question rated up once and down once makes a ranked pair. Prompts are rebuilt with the exact system prompt and passages the model saw. Then make train eval release produces a new candidate, and it ships only if stages 09 and 10 pass again.

What a single request does

  1. The guest sends a message in any mix of Arabic, English and Urdu.
  2. The gateway masks PII, identifies the language and assigns a canary bucket.
  3. The retriever returns the top passages through BM25 and dense search, fusion and rerank.
  4. vLLM answers from the shared system prompt, the passages and the question.
  5. The guardrails check every number and mask PII again, then the guest gets the answer with a trace ID.
  6. Latency, tokens and flags are logged, and a later rating can come back through /feedback.

Five hard problems

Arabic tokenization. Covered in stage 05: word-initial tokens, mean-initialised embeddings and a lossless decode test.

Code-switching. One message can mix Arabic script, English and Roman Urdu. lang.py labels each run separately and splits Arabic-script runs at sentence ends, because Arabic and Urdu share a script and a single label per message would be wrong half the time.

Eval contamination. A naive split leaks near-duplicates into the test set. The evaluation split is frozen by hash on the first run, only real pairs go into it, and training data is deduplicated against it before every run.

Hallucinated prices. No fine-tune knows today's rates. Prices come only from retrieval, DPO pairs penalise invented numbers, and the output guard replaces any answer containing an unsupported number.

Latency on a budget. int4 AWQ fits one mid-range GPU, vLLM batches continuously, and n-gram speculative decoding speeds up the copy-heavy answers.

Results: read this table carefully

The repository README ships a results table, and I need to be straightforward about it. The file carries a note telling the reader to replace every number with their own measured values before publishing. Treat the figures below as the illustrative numbers in the template, not as an audited benchmark, and run make eval on your own data before quoting any of them.

MetricBase modelrihla-llm
Arabic-script tokens per word2.91.7
Domain QA pass rate (judge 4 or higher)41%78%
Grounding suite pass rate52%97%
PII red-team pass rate89%99.6%
p95 gateway latency1.9 s0.6 s
GPU memory at serve time28 GB7.8 GB

Design decisions in one place

  • QLoRA over full fine-tuning: budget-bound, and one base model can serve several clients through adapters.
  • DPO over PPO-based RLHF: no reward model to maintain.
  • Retrieval for anything that changes: the model learns how to answer, not what today's rate is.
  • Evaluate the artifact that ships: the gate runs on the int4 build, not the bf16 model it came from.
  • One prompt format: prompts.py feeds SFT data, DPO pairs, feedback pairs, evals and the gateway, so the model is never served a format it was not trained on.

Running it and testing it

Install with pip install -e ".[ingest,train,quant,serve,ops,dev]", put the client exports in raw/, point the teacher setting in configs/rihla.toml at a strong model, then run make data, make train, make eval, make release and make serve. The first eval run needs you to review the report and accept it as the baseline. After that, make metrics gives a weekly health report. Twenty-two tests run on CPU only: PII patterns, dedup, the frozen split, language ID, retrieval, guardrails, the gate, signing, canary routing and the gateway through FastAPI's test client. Stages 05 to 07 are tested on a tiny model built inside the test, and the vocabulary extension has to lower tokens per word and decode losslessly.

What I would do next

  • Put retrieved passages into the SFT examples so training matches the RAG prompt even more closely.
  • Distil the int4 model into a smaller student for a CPU fallback.
  • Replace thumbs ratings with pairwise comparisons, which give a cleaner DPO signal.

Frequently asked questions

Which base model does rihla-llm use?

The default in configs/rihla.toml is Qwen2.5-7B-Instruct. It is a single setting, and the code is built on Hugging Face transformers, TRL and PEFT, so another compatible open-weight model can be swapped in, though the tokenizer stage and the tests were written against the default.

Can I reuse it for another language or industry?

Yes. The pipeline is domain-neutral apart from the example evaluation suites and the PII rules, which target Saudi and Pakistani formats. Replace those, supply your own exports, and rerun the gate.

Why not just use a large hosted model with RAG?

For this client the answer was cost, latency and data control. A quantized model on one GPU keeps guest data in-house and answers quickly. A hosted model with good retrieval is a reasonable choice when those limits do not apply, and the retrieval and guardrail stages of this repository would still be useful in front of it.

If you want something like this built, or a review of an existing multilingual or RAG system, see my services or get in touch. For more of my own builds, read about PrintDrop and RAG pipelines and AI agents.

Sources

More Articles