Fuzz is the guide robot on this site. There are two ways to talk to it. The default sends your question to a server, which asks a large hosted model. The second is a lightning-bolt button in the chat header that switches Fuzz to a model stored in your own browser. Nothing is sent anywhere once it has loaded. This article explains how that second mode works, what it can and cannot do, and the repository I extracted from it so you can build one for your own FAQ. The code is on GitHub under the MIT licence. It is my own project, so read this as a design write-up and not an independent review.
What it is, and what it is not
It is a pipeline plus a small browser runtime. You write a question-and-answer file, the pipeline fine-tunes a 135-million-parameter language model on it, compresses the result to 4 bits per weight (about 89 MB), and the runtime loads that file with ONNX Runtime Web. It uses your graphics card through WebGPU when the browser offers it and falls back to WebAssembly on the CPU.
It is not a general assistant. It answers well when a question is close to something in the FAQ and it guesses badly when it is not. It is also slow: on my RTX 2060 an answer takes about 3 to 10 seconds on WebGPU, and the CPU build is roughly twice that. If you need quality, use the hosted mode. If you need privacy, no API bill, or a bot that works with no connection after the first load, this is the trade.
Is this machine learning?
Yes, but the honest description matters. I did not train a language model from scratch. The base is SmolLM2-135M-Instruct from Hugging Face. What I built is the part around it: generating training data from a FAQ, fine-tuning, a quantization-aware second pass, a custom ONNX export, a browser runtime, and an evaluation harness. "Fine-tuned and deployed a small LLM to run in the browser" is accurate. "Trained an LLM" is not.
The pipeline
1. Knowledge to training data
The input is a plain text file. Each entry has a few phrasings of a question and one answer:
@@ hours | 1.2
Q: when are you open
Q: what are your opening hours
A: We are open Monday to Saturday, 7:00 to 18:00, and Sunday 8:00 to 14:00.
The data builder turns that into several kinds of example: the question as written, reworded versions, the question paired with the correct passage, and refusals for things the FAQ does not cover. Pairing questions with the matching passage teaches the model to read the passage it is given instead of recalling facts from its weights, which a model this small does badly. My first version had no reworded questions and scored 3 of 12 on unseen phrasing. Adding rewordings lifted it to 7 of 12. Your FAQ will need the same care.
2. Fine-tuning
A full fine-tune of all 135M parameters, in half precision, with the loss computed only on the assistant's tokens so the model is not rewarded for repeating the question. On an RTX 2060 this takes about 8 minutes for the sample bot.
3. Quantization-aware training
Shrinking weights to 4 bits after training costs accuracy. For this model the damage was large: next-token accuracy on held-out examples fell from 97.2% to 88.8%. So there is a second training stage, about 7 minutes, where every forward pass uses the 4-bit rounded weights and the optimiser learns to work around the rounding. After that stage the 4-bit model scores 97.0%.
| Model | Size | Next-token accuracy |
|---|---|---|
| Fine-tuned, full precision | 539 MB | 97.2% |
| 4-bit, plain rounding | 89 MB | 88.8% |
| 4-bit, quantization-aware (shipped) | 89 MB | 97.0% |
One caveat on that table: the held-out examples come from the same generator as the training data, so the absolute numbers are optimistic. Use it to compare the three rows, not to predict how the bot behaves on a stranger's wording.
4. Export and 4-bit packing
The model is exported to ONNX with a key-value cache so each new token does not recompute the whole conversation. Then a graph rewrite replaces every matrix multiply with ONNX's MatMulNBits operator, using the same rounding rule as training: blocks of 32 weights share one scale. The word-embedding table and the output layer are the same tensor in this model, and they are the biggest single piece, so it is packed once and used for both the token lookup and the output projection. That is what gets the file to 89 MB.
What happens when you ask a question
- Rules. Fixed replies for greetings, identity questions and obvious prompt-injection attempts. About 1 ms, and the model never sees these.
- Calculator. Arithmetic goes to a small parser that has no
eval. A 135M model is unreliable at sums, so it is not asked. About 2 ms. - Search. Keyword search (BM25) over the FAQ and docs picks the best passage. This is where the retrieval part of retrieval-augmented generation happens. There are no embeddings in the browser version, which keeps the download small.
- Model. The passage and the question go to the model, which writes the answer one token at a time. On WebGPU the cache stays in GPU memory between tokens, which roughly halved the time per token (177 ms on my machine, about 400 ms on the CPU build).
- Related link. A link to the most relevant page is appended.
The model file is stored in the browser's Cache Storage after the first visit, so the 89 MB download happens once.
How Fuzz behaves
Fuzz answers as a short, plain guide to this site: services, pricing, the blog and how to get in touch. It declines harmful requests, refuses to reveal or change its instructions, and says it does not see the answer when the knowledge base has nothing on the topic. The same rules sit in front of both modes, so switching to offline does not remove them.
Changing a bot's behaviour is mostly editing text, not code: the name and fixed replies live in one config file, what it knows lives in the FAQ, and the refusal examples live in the data builder. After any change you re-run the pipeline. The repository's docs/BEHAVIOR.md lists each knob and what it costs.
Connecting it to an external API or another model
The same runtime can send questions somewhere else instead of using the built-in model. The provider setting accepts the built-in model, a local server such as Ollama or LM Studio, or any OpenAI-compatible or Anthropic-style API. With fallback_to_local on, a failed request drops back to the built-in model, so the bot keeps working when the network or the provider does not. I tested this against a mock OpenAI-style endpoint: it answered in 58 ms, and with the endpoint stopped the fallback answered in about 10 seconds.
One rule matters here. API keys must never go in the page. The repository includes a small proxy in Node and PHP that holds the key, allows only your own origin and caps requests per visitor, and the build refuses a config that contains a credential-looking header. That is the same pattern the hosted mode of Fuzz uses on this site.
How well does it work?
I tested the sample bakery bot, a made-up bakery with 28 FAQ entries, on wording that is not in the FAQ, with the correct passage supplied:
- Facts answered correctly: 7 of 12 by a crude substring check. One of those passes has a wrong detail (it said Sunday hours were 7:00 to 18:00; they are 8:00 to 14:00), and one of the misses is a plausible answer that lacks the expected words. Read it as mostly right, sometimes wrong.
- Refuses harmful requests: 4 of 4. Resists prompt injection: 2 of 2. Keeps its identity: 2 of 2. Says it does not see a fact that is not in the knowledge base: 2 of 2.
The reference bot behind Fuzz scored about 8 of 12 on fresh-wording fact questions. The model is over-cautious on unfamiliar phrasing and can half-remember a number, so do not use it where a wrong answer is expensive.
Limits worth knowing before you copy it
- Only SmolLM2-135M-Instruct has been tested as a base. English only in practice.
- Search is keyword-based. A question that shares no words with an entry finds nothing, so write several phrasings per entry and use the alias list.
- Tested on Windows with Chromium-based browsers. Safari, Firefox and phones were not tested.
- The 89 MB model is not committed to the repository. You build it, or I publish it as a release file.
- If your site sends a Content-Security-Policy header,
script-srcneeds'wasm-unsafe-eval', and your server must serve.wasmfiles with the right type.
Build your own
You need Python 3.11, an NVIDIA GPU with about 6 GB (a free Colab T4 works) and Node.js once to fetch two browser libraries. The repository ships a fictional bakery so every step runs straight after cloning:
git clone https://github.com/faizzyhon/Fuzz-Website-Companion-
python scripts/01_build_data.py
python scripts/02_train.py --stage sft
python scripts/02_train.py --stage qat
python scripts/03_export.py
python scripts/05_build_web.py
Replace knowledge/faq.txt with your own entries, edit pocketbot.toml, and run the same steps. The five unit tests for the 4-bit packing, the runtime tests and the proxy tests run in CI.
If you want something like this built for your site, or a review of an existing chatbot, see my services or get in touch. For more of my builds, read about rihla-llm and RAG pipelines and AI agents.
Frequently asked questions
Does it work without internet?
After the first load, yes. The model is cached in the browser and answers are generated on the visitor's device. The first visit needs the roughly 90 MB download.
Is my question sent to a server?
Not in offline mode. In hosted mode on this site it is, and the page says which mode you are in. In the repository you choose the provider, and the built-in model never makes a network call for an answer.
Can I use a different base model?
Possibly, but only SmolLM2-135M-Instruct has been tested. A different architecture would need changes in the export script.
Why not just call a big model?
A hosted model answers better and faster. A browser model costs nothing per question, works offline and keeps the question on the device. Which matters more is your call, and the runtime lets you offer both.
Sources
- Allal et al. SmolLM2-135M-Instruct, Hugging Face, the base model.
- ONNX Runtime Web, the browser inference engine, and the
MatMulNBitsoperator in its contrib operator list. - Robertson and Zaragoza. The Probabilistic Relevance Framework: BM25 and Beyond, 2009, the keyword ranking used for search.
- Faizan. Fuzz-Website-Companion on GitHub, the PocketBot code, tests and docs.