I only wanted a vector
I was building part of a voice assistant. A customer says "I want to order pizza" and the system has to work out what they want. An order? A complaint? A question? My plan was small. Turn every sentence into a list of numbers, called an embedding, then find the nearest sentence whose meaning I already know.
An embedding is a point on a map with 768 directions. Sentences that mean the same thing land close together, even across languages.
My test set is 202 short sentences in Hindi, Marathi, Telugu, Tamil, Hinglish and English, covering intents like ordering food, checking the weather and booking a ride. On it, EmbeddingGemma picked the right intent first time 96.5% of the time. The smaller multilingual-e5-small got about 91%. The catch is the budget. A voice assistant gets roughly 10 ms for this step, and only the small model fit inside it.
A few lines of Rust, I thought. Then I stepped on a loose board.
I should say where I come from. I grew up on AtCoder, CodeChef and Codeforces, writing C++ against the clock, where fast means a better algorithm. Go from O(n²) to O(n log n) and you win. This project gave me no such lever. The maths of a neural network is fixed. Everything I learned came from the constant factor, the boring part I had always skipped.
The first measurement for Gemma was 150 to 170 ms per sentence, far outside that budget and too slow for something that sits in the middle of a live conversation. So I started digging, one layer at a time.
Depth 1: the gate of tokens
Before the model sees a sentence, it chops it into pieces called tokens. And before it lets you in, it asks you to wear a name tag. EmbeddingGemma wants every query to start with task: search result | query: .
For a short sentence that tag is about 7 of the 16 tokens, and every token costs about 1.3 ms of real computation. Chopping the text up is cheap, about 0.07 ms. So the time was not hiding in my Rust code.
Depth 2: the hall of 24 layers
Next came a long hall. This is the model itself: about 300 million adjustable numbers arranged in 24 layers. Each layer lets every token look at every other token and mix their meaning. At the far end everything is squeezed into one point on that 768-direction map.
Meaning turns into location. "I want to order pizza" and its Hinglish cousin land near each other. "What is the weather" lands far away. That is why intent matching becomes a simple question: which known point is nearest?
What that map looks like
I squashed the sentences of three models onto a flat page, one dot per sentence, coloured by intent. Squashing 768 directions into two loses a lot, so read it as a picture, not a measurement.

In the two small-model panels, full precision on the left and 8-bit in the middle, the eight intents mostly group together, but several of them break into two or more patches. The 8-bit panel looks as tidy as the full-precision one, which is the picture version of "8-bit cost me nothing". In the EmbeddingGemma panel on the right each intent sits in a tighter group with more space around it, which fits its higher score.
I also checked which sentences the small model treats as alike. Each cell below is a cosine similarity, where 1.00 means the same direction.

Two things stood out. Pizza in Hindi, Marathi, Telugu and Tamil scores 0.95 to 0.98 against each other, so the model sees them as nearly the same request. But English pizza scores only 0.88 against Hindi pizza, and a Hindi joke scores the same 0.88 against Hindi pizza. So the small model leans on the language as well as the meaning. Unrelated sentences still score between 0.73 and 0.89, so the scale is squeezed and the cut-off for "same intent" has to be chosen with care.
Depth 3: the managers
The model is just a file, a recipe of operations plus the weights. Something has to cook it. I met two managers, ONNX Runtime from Microsoft and OpenVINO from Intel.
They only worked with me when their version numbers matched exactly. One pairing refused to load and gave a Windows error 127. I also ran the same job from Python to compare. Rust and Python tied, because more than 99.8% of the time is spent inside the model. The language was never the problem.
Depth 4: the workshop
The managers do not multiply matrices themselves. They hand the job to craftsmen who spend years tuning one thing, matrix multiplication. ONNX Runtime has one called MLAS. OpenVINO uses oneDNN. NumPy and PyTorch often use MKL.
To see what they do, I wrote my own matrix multiply in C++ six ways. The sums are identical every time. Only the way they are done changes.
| Version | Time (1024 x 1024) | Speed-up |
|---|---|---|
| Naive triple loop | 2122 ms | 1x |
| Loops reordered so memory is read in order | 610 ms | 3.5x |
| AVX2, 8 numbers per instruction | 131 ms | 16x |
| Keep a small block in CPU registers | 51 ms | 41x |
| Plus 4 threads | 22 ms | 96x |
Swapping two loops gave 3.5x with no new instructions. That one stung, because it is the kind of thing a contest trains you to see. One of my ideas did nothing: splitting the matrix into cache-sized tiles did not beat the simple reorder. The real libraries reach about 200 GFLOPS on my laptop. My toy reached about 97.
Depth 5: the vault of lanes
At the bottom of the workshop is the CPU. A normal instruction adds two numbers. An AVX2 instruction adds eight at once, one per lane. A fused multiply-add does the multiply and the add together, and each core has two units that can do it.
My i5-9300H has no AVX-512, which would give 16 lanes, and no VNNI, which speeds up 8-bit integer maths. On paper the chip peaks at about 450 GFLOPS. The best I measured was about 200. A gap that size is normal, because small matrices and memory speed hold the lanes back.
I also tried compiling the Rust service for this exact CPU. It changed nothing, because the hot loop is inside the libraries, not in my code.
Depth 6: the gnome with the sack
This is where it got interesting. My first report said OpenVINO was 6.4 times faster than ONNX Runtime. I did not trust it, so I asked ONNX Runtime to show me where its time went.
The profiler pointed at one operation, DequantizeLinear, using 62% of the time. The Gemma file stores its weights as small 8-bit integers to save disk space. But on every single call, ONNX Runtime converted all of them back to full 32-bit numbers before using them. That is a fixed 145 to 175 ms of hauling a 1.2 GB sack up the stairs, however short the sentence.
I fixed it by doing the conversion once, offline, and saving the model with the full-size weights.
| One sentence at a time | |
|---|---|
| Before | 147 to 171 ms |
| After | 34 ms |
The embeddings came out identical (cosine similarity 1.000000). The price is a bigger file, 295 MB growing to 1.18 GB.
I also owed a correction. Most of the OpenVINO "speed-up" in my earlier report was this avoidable cost, not OpenVINO being cleverer. I went back and fixed the report.
Depth 7: shortcuts through the market
With the gnome gone, what was left was real arithmetic, about 1.3 ms per token. So I went looking for fewer tokens and cheaper arithmetic.
| Experiment | Time for one sentence | Right intent first time |
|---|---|---|
| Folded model (baseline) | 41 ms | 96.5% |
Shorter tag, query: |
37 ms | 98.0% |
| 4-bit weights | 55 ms (slower) | 95.5% |
| Dynamic 8-bit integer maths | 34 ms | 97.0% |
| 8-bit plus the shorter tag | 19 ms | 98.5% |
| 8-bit on OpenVINO | 190 ms (much slower) | 98.0% |
A few things I would not have guessed. Dropping bits to 4 made it slower on this chip. Dynamic 8-bit maths made it faster even without the special integer instructions. And OpenVINO, the supposed speed champion, was eight times slower on the same model.
Be careful with the accuracy column. Differences of one to three sentences out of 202 are noise. All I can say is that I saw no loss. The 19 ms figure came from Python. In the Rust service, 8-bit maths gave 31.6 ms for one sentence but got slower for batches of 16, so it only suits single requests.
Here is the part that changes the conclusion. Even at 19 ms, Gemma still misses the 10 ms budget. For the tightest real-time path the small model, at about 5 ms, is still the right pick. Gemma earns its place where accuracy matters more than the last few milliseconds.
Depth 8: the wall
At the very bottom there is a wall. It is the hardware. Four cores, eight lanes, about 200 of a possible 450 GFLOPS.
I thought about writing my own kernels with SYCL, a way of writing C++ that runs on different chips. But on this laptop it would land on the same four cores and the same ceiling, so I expect nothing from it. I have not tried it. The real next steps are different hardware: a CPU with AVX-512, VNNI or AMX, or a GPU or NPU.
The scoreboard from my first report
Before the deep dive I had already written a benchmark report, comparing my service against the usual tools on the same laptop, with the same 202 sentences. This is its master table, with the corrections I made later.
| Setup | Precision | Dimensions | One sentence | Texts per second (16 at once) | Right intent first time | Memory |
|---|---|---|---|---|---|---|
| My Rust service (ONNX Runtime), e5-small | 8-bit | 384 | 5.05 ms | 278 | 91.1% | 430 MB |
| Hugging Face TEI, e5-small | 8-bit | 384 | 6.20 ms | 264 | 91.1% | 680 MB |
| FastEmbed, e5-small | 8-bit | 384 | 8.23 ms | 220 | 89.6% | 1,019 MB |
| Plain PyTorch, e5-small | 32-bit | 384 | 20.49 ms | 167 | 91.1% | 2,110 MB |
| EmbeddingGemma on OpenVINO | 8-bit | 768 | 37.99 ms | 54 | 96.5% | 580 MB |
| EmbeddingGemma, default ONNX Runtime | 8-bit | 768 | 165.80 ms | 27 | 96.5% | 710 MB |
| EmbeddingGemma, unconstrained | 8-bit | 768 | 245.00 ms | 19 | 96.5% | 740 MB |
All of this ran on an Intel i5-9300H, with 4 cores and AVX2, and no GPU. A few things from the same report:
- 8-bit instead of 32-bit made the e5-small model four times smaller (118 MB against 470 MB) and lost no accuracy in Recall@1. When I later compared the two on my Rust pipeline, the embeddings had a mean cosine similarity of 0.9967.
- 4-bit was a bad idea for the small model in the report: it cost 8 to 11 points of accuracy. An embedding depends on angles between vectors, and 16 levels is too coarse. (My own 4-bit test on Gemma lost little accuracy but was slower.)
- Shorter vectors. EmbeddingGemma can be cut from 768 numbers down to 128 without retraining. That made searching 5.8 times faster, shrank the index by 83%, and Recall@1 went from 96.5% up to 98.5%. The model itself takes the same time, because the cut happens at the end. With 202 sentences a gain of 2 points is a few sentences, so I would call it "no loss" and not "better".
| Vector length | Right intent first time | Search time for 1,000 queries | Index size |
|---|---|---|---|
| 768 | 96.5% | 24.1 ms | baseline |
| 512 | 97.0% | 10.8 ms | 33% smaller |
| 256 | 97.5% | 6.3 ms | 67% smaller |
| 128 | 98.5% | 4.2 ms | 83% smaller |
- Many callers at once. With 4 clients the service handled 284 sentences a second instead of 136, but the slowest requests rose to about 30 ms, because 4 clients with 2 threads each are too many for 4 cores.
The report's advice came out in two parts. For a real-time voice path under 10 ms, use the Rust service with the small model, two sessions of two threads each, and a cache of the 5,000 most recent sentences. A cache hit costs under 0.01 ms. For harder routing where accuracy matters more, use Gemma, cut to 128 or 256 numbers.
What I got wrong in that report. I wrote it before I profiled anything, and three claims did not survive.
- It said OpenVINO was 6.4 times faster. Most of that was the avoidable weight conversion in Depth 6. The 38 ms figure only reproduces with native OpenVINO, and going through ONNX Runtime it was 75.6 ms.
- It quoted 5.05 ms for my service. When I rebuilt and re-measured on Windows I got 6.96 ms and could not pin down why, so I would re-measure on an idle machine before quoting either.
- It described Gemma as a 12-layer model. The graph I inspected has 24 layers.
Everything above is in the full report, including those corrections in section 9.
Everything I tried, in order
The story above skips a lot. Here is the full list, so you can see how deep each layer went and what came out of it. Most of these were dead ends, and I kept them in because a dead end tells you where not to dig.
| # | Layer | What I tried | What I found |
|---|---|---|---|
| 1 | Code | Reviewed the Rust service and fixed what was wrong, then got it building on Windows (a MinGW toolchain, after Smart App Control blocked unsigned build scripts) | It builds and runs, and the notes list every snag so I can rebuild it later |
| 2 | Language | Same job in Rust and in Python, same model and settings | A tie. More than 99.8% of the time is inside the model |
| 3 | Compiler | Rebuilt with target-cpu=native so the compiler could use every instruction my CPU has |
No gain |
| 4 | Runtime | Added OpenVINO as a second back end, and Gemma support | Works, but only with exactly matching versions. Three pairings tested, one failed outright |
| 5 | Runtime | Swept threads and hints | 4 threads was best on 4 cores. 8 threads did not help, and OpenVINO's throughput hint was worse (717 vs 500 ms for a batch of 16) |
| 6 | Hardware | Measured the real matrix-multiply ceiling with NumPy | About 200 GFLOPS, not the 450 on the spec sheet |
| 7 | Runtime | Profiled ONNX Runtime operation by operation | DequantizeLinear took about 62% of the time |
| 8 | Runtime | Fitted a line to latency against token count | ONNX Runtime: 145 ms fixed plus 1.30 ms per token. OpenVINO: about 0 fixed plus 1.30 ms per token |
| 9 | Model file | Folded the dequantization into the weights offline | 147 to 171 ms down to 34 ms, same embeddings |
| 10 | Tokens | Tried four different prefixes | A shorter one saved about 10% with no visible accuracy loss |
| 11 | Model file | 4-bit weights | Slower than full precision on this chip |
| 12 | Model file | Dynamic 8-bit integer maths | 34 ms, or 19 ms with the shorter prefix. Slower for large batches in Rust |
| 13 | Runtime | Dynamic 8-bit on OpenVINO | 190 ms. Much slower |
| 14 | Kernels | Wrote a matrix multiply in six versions | 2122 ms down to 22 ms. Showed which tricks matter |
| 15 | Competition | Ran my service against Hugging Face TEI, FastEmbed and plain PyTorch (first report) | Mine was fastest at 5.05 ms, then TEI at 6.20 ms |
| 16 | Vectors | Cut Gemma's vectors from 768 numbers to 512, 256 and 128 (first report) | Search 5.8 times faster, index 83% smaller, no visible accuracy loss |
| 17 | Quality | Drew t-SNE maps and a similarity heatmap (first report) | Gemma's intents sit in tighter groups. The small model leans on language as well as meaning |
| 18 | Load | Ran 4 clients at once (first report) | 284 sentences a second, but the slowest requests rose to about 30 ms |
Things I did not try, so nobody assumes I did: SYCL, the laptop's built-in graphics, a padding experiment, checking whether the smaller default model hides the same dequantization cost, and any real customer data.
My original report, with the first round of numbers, is here. Some of its early claims turned out to be wrong, and section 9 of it holds the corrections.
What I carried back up
I came back with a map of the whole stack, bottom to top, and a service that went from 171 ms to about 19 to 32 ms. But the thing I value most is a habit. Profile first, then change something. The biggest win of the whole trip was a profiler and one small script, not a new language and not a clever instruction.
Some honest limits. Everything here was measured on one laptop. The test set is 202 sentences, not real customers. The facts about newer chips come from reading, not from testing. And I have not measured the shorter tag in the Rust service itself.
Next I want to add a cache for common phrases like "yes" and "cancel", try batching, and test on a bigger, messier set of real sentences.
The code, notes and the little matrix multiply are in my rustem repo. The full write-up is in REPORT.md. The experiment notes are in BUILD_NOTES.md and LEARNING_NOTES.md, and the matrix multiply is in learning/matmul.