DEV Community: InferenceKey

Serving a 1B math model on vLLM: the memory cap sets how many answers fit, not how fast

InferenceKey — Sun, 01 Nov 2026 06:34:10 +0000

Limite 1B Violetto is a math model that ships as one 2.1 GB file, and Paradigma's table puts it at 94.01% on AIME 2026, a US exam for top high school students. We put it on one RTX 4090 on our cloud and gave it the 30 problems once each. The result: 27 correct, 2 wrong, 1 with no final answer. The per-problem breakdown is in our full test of Limite 1B on AIME 2026. This post covers the serving choices, which differ from a chatbot's.

Credit first. Prignano, Cirillo, Morosini, Cerovaz, Bartolocci, Rodolà, Starace and Pappone at Paradigma trained the model from scratch. They released it under Apache 2.0 with their evaluation code, and wrote the vLLM plugin that serves it. MathArena maintains the AIME 2026 problem set on Hugging Face. We heard about Limite in the AI Search news video of 27 September. What we added: a deployable package around the plugin, runs on two different cards, and a grade for every answer.

Stock vLLM will not load it, and the error points the wrong way

vLLM is the open-source program that serves models behind an HTTP API. Limite has its own architecture, and vLLM does not know it out of the box. The code that teaches it lives in Paradigma's plugin, which vLLM only loads when an environment variable names it:

VLLM_PLUGINS=limite vllm serve paradigma-inc/limite-1b-violetto

Leave the variable out and you get "Transformers does not recognize this architecture". It sends you off to upgrade Transformers, which will not help. The plugin can only copy the whole model onto each GPU (data parallelism), never split one copy across cards. And Paradigma's verified stack pins Python 3.12, vLLM 0.26.0 and a PyTorch build for CUDA 13.0, which means an NVIDIA driver of 580 or later on the host.

Our packaged vLLM option installs plain vLLM only, so we wrote a custom backend: a small Python file that installs the versions Paradigma tested, launches vLLM with the plugin and relays requests.

The memory cap is about concurrency, not speed

The flag to understand is --gpu-memory-utilization, the share of the card's memory vLLM is allowed to claim. We ran the 4090 at 0.90, then at roughly 6 GB and 4 GB. Speed stayed flat: 211 tokens per second at 0.90, 213 and 209 at the 4 GB cap. Problems 1 and 5 came back correct each time.

A token is a chunk of text, roughly a short word. vLLM keeps a scratch record of every token in each open answer, and the cap sizes that area. At 0.90 it held 1,465,844 tokens, about eleven full-length answers. At the 4 GB cap it had room for 141,408, which is about one. Real usage ran about 1 GB above the cap, so plan for it.

At 0.12, about 3 GB, vLLM would not start at all. It needs room for one answer at the full 131,072 tokens, and proposed a limit near 40,000 instead. We left that untried: several answers ran longer.

One caveat: a capped 4090 imitates a small card's memory, not its slower chip. Only two short problems ran under the caps, and no real 8 GB or 12 GB card was tried. A real 16 GB RTX 2000 Ada answered correctly at 85 tokens per second.

Measure answer length before you size anything

Limite reasons out loud. It writes out attempts, checks them and discards the failures before answering, and you wait for all of that text. Problem 1 needed 3,203 tokens. Problem 10 needed about 101,000. The median sat near 21,000.

Log that spread first: it drives the memory cap and the batching. Batching means sending many problems at once so the GPU works on them together. With all 30 in flight, throughput peaked around 17 times one answer's speed. But short answers finished in under a minute while the batch ran 16 minutes, waiting on problem 10. Sorting problems by expected length should shorten that wait. We have not measured it.

On a 4090 at about 0.80 euros an hour, a full batch comes out under one euro cent per correct answer. One at a time, most of the card sits unused.

A fixed prompt, and an SDK gap of ours

The chat template, the format that wraps each request before it reaches the model, only accepts Paradigma's standard opening instruction. A custom system prompt or a tool definition gets rejected. Our backend strips any non-standard one, so a generic chat client still works. Non-math questions come back reframed as math, so this is no fit for chat or tool-calling agents.

One gap is ours: the current SDK release has no working chat path here, and the client hung three minutes until its timeout instead of erroring. The newer code in the SDK repository fixes it, and the issue stays open until the next release.

What the test does and does not show

Paradigma averages 32 attempts per problem. One attempt is noisier, so 27 of 30 places the model in their range, not at their exact figure. The AIME 2026 problems were public months before training, and we did not test for contamination. On ArXivMath May 2026, a newer set, Paradigma reports 25%. Our grading looked at the final boxed answer only, never at the steps that led to it.

The serving setup, the full memory ladder and each problem's outcome are in the research post, with the SDK example that deploys and grades it.

To size a card for Limite yourself, start from the memory ladder and token counts in the Limite 1B AIME 2026 write-up on inferencekey.com.

Fitting Flint's coding agent on a 24 GB RTX 4090: context is the knob that decides

InferenceKey — Sat, 31 Oct 2026 06:34:09 +0000

Flint is a new coding-agent model from Recursive, trained from scratch and released under Apache 2.0. Its official compressed file weighs 22.9 GB, and Recursive recommends at least 48 GB of GPU memory for it. On a 24 GB RTX 4090, with an agent loop and four tools, it still fixed a planted bug in five turns. The deciding flag was the context length, meaning how many tokens (word pieces) the model can hold in view at once.

A 39.8B model that computes like a 2.5B one, and still loads whole

Flint has 39.8 billion parameters, the learned numbers that make up the model. About 2.5 billion work on any given token. It is a mixture of experts: a router picks 6 of 256 expert blocks per token, and one shared expert always joins in. Compute stays small, memory does not: idle experts still live on the card.

On the 4090 the loaded file takes 21.3 GiB of the roughly 22.5 GiB the card exposes. Everything else has to squeeze into the gap.

The trap: context length is a memory reservation

"Everything else" is mostly the KV cache. The model stores a key and a value per token in its attention layers, so earlier tokens are not recomputed at every step. That space is reserved at launch, sized to the context you ask for.

Only 14 of Flint's 49 layers attend to the full conversation. The others only look back 513 tokens (a sliding window). Each of the 14 keeps 4 KV heads (sets of keys and values). The memory arithmetic in the Flint deployment notes puts the cache at 28,672 bytes per token: roughly 0.9 GiB at 32K and 1.75 GiB at 64K.

On the 4090, 16K fit with the whole model on the GPU, at 22,316 MiB in use out of 23,028. At 32K the weights loaded, and then the cache reservation was refused:

allocating 87.50 MiB on device 0: cudaMalloc failed: out of memory
failed to allocate buffer for kv cache

That was 87.5 MiB, under a tenth of a GiB. The driver also holds memory outside the process, so the box overstates what you get. With 32 GB, an RTX 5090 ran 64K at about 23.3 GiB.

Recursive's example command passes the full 262,144-token context, the length their 48 GB figure is meant for. On 24 GB you pick a smaller one yourself. 16K covered our one-bug task, and a long session over a big repository will outgrow it.

What the agent loop did

The project: an invoice module that handled a discount percentage as a fraction, two tests that exposed it, and four tools (list files, read file, write file, run tests). Across seven runs on three machines its route never changed. First it listed the files and ran the tests to watch them fail. Then it read the module and its test file in one turn, wrote the fix, reran the suite and got three passing tests. It never touched the tests, and every tool call was valid JSON.

The 4090 wrote about 167 tokens per second, about 20% slower than the 5090's 210. The full loop finished in 5.7 and 6.3 seconds. Below 24 GB we only emulated cards, by capping memory and moving some experts to system RAM. Near 16 GB it answered at 4.6 tokens per second, too slow for interactive use.

Seven runs on one planted bug are a smoke test, not a benchmark. Recursive reports 74.2% on SWE-bench Verified (real bug reports from open source projects), a figure we did not reproduce. We also did not measure what compression costs in quality. Each configuration, down to an emulated 15 GB card, is in the Flint VRAM ladder with peak memory per context.

Two checks before you trust your numbers

Your llama.cpp build. llama.cpp is the engine that runs the file, and Flint's architecture was new to it. Support landed in PR #30928 by alexey-rsi, merged on 21 October, and release b11803 ships it. The chat template also relies on PR #30929, from the same author. A build from before Flint cannot load the model. If your tooling fetches llama.cpp by tag, as our worker (the program on the GPU machine that starts the engine) does, give it a tag such as b11803: a commit hash will not work there.

Your GPU. Our first RTX 5090 generated 27 tokens per second. Same file, same build, a different machine: 210. We could see a 400 W power cap and an older driver; the real cause is still open, and the slow card reported healthy. Run llama-bench (llama.cpp's speed test) on each new machine before you blame the model.

On the client side, this build puts Flint's reasoning inside the answer text, with no reasoning_content field, and those tokens count against max_tokens. Coding tools set up as Recursive suggests look for the separate field. Status: open.

Where to go from here

For short sessions, a 24 GB card at 16K or a 32 GB card at 64K is enough. For long ones, budget more memory.

Recursive built Flint and published its compressed files. alexey-rsi wrote the llama.cpp support. CISC and ggerganov approved PR #30928, and CISC and danbev approved PR #30929. ggerganov also wrote the per-expert scaling (#30861) that the Flint support uses. Our part was deploying it on InferenceKey, measuring it, and writing an SDK example that reruns the bug-fixing loop on your own NVIDIA worker. The example defaults to 64K, so on a 24 GB card change -c 65536 to -c 16384 in main.py.

The serve command, start-up timings and every open issue are in the Flint RTX 4090 deployment report on InferenceKey.

Serving MOSS-Image: step count, memory mode and the traps that change the words you ship

InferenceKey — Fri, 30 Oct 2026 06:37:39 +0000

If your product makes menus or posters through an image API, what matters is that the price on the menu is the price you sent. We served MOSS-Image-SFT, an Apache 2.0 model from OpenMOSS at MOSI.AI built for long text in pictures, on one A40 on our cloud. Two settings decided almost everything: the step count, and where the weights (the model's learned numbers) sit while it runs.

With the authors' 50-step recipe, every word was correct on every text test, at 78 to 81 seconds per picture of about one megapixel. A community turbo add-on from lodestones took 8.6 to 9.2 seconds and got two words wrong. The full MOSS-Image test on one A40 has all 22 pictures and the deploy steps. This piece covers the choices.

Why 50 steps cost 100 passes

The model begins with random static and removes some of it per step. Each of the 50 steps calls it twice, with your prompt and with an empty one, and the gap between the answers steers the picture toward your words. That is about 100 runs of a 5-billion-parameter network per picture.

The lodestones 8-step add-on is a LoRA: a 1.2 GB set of small corrections laid over the original weights, trained to finish in 8 calls with no empty-prompt call. Its card says the correction is smaller than bf16's rounding step (bf16 is the 16-bit format of the weights), so folding it in would erase it. It stays a separate layer at strength 0.7, as in the hugging-apps demo Space. That strength is shared by every request, so our backend (the server that loads the model and answers requests) runs one picture at a time. One loaded copy serves both modes:

{"prompt": "A clean conference poster on a white wall. Large bold title: OPEN WEIGHTS DAY",
 "size": "1024x1024", "seed": 42, "mode": "turbo"}

The body follows OpenAI's image API, plus mode (sft or turbo). The seed fixes the starting static: keep it the same when you compare modes, or you are comparing luck.

The trap: running the 8-step LoRA at 4 steps

We tried the 8-step add-on at 4 steps. It took 5.2 seconds and the lettering broke.

The title read "OPEN WEIGHT DAYDY" and times looked like "1(.C0". The add-on was trained for 8 steps. Adapters built for 4 exist, and we did not test them. At 8 steps, turbo repeated a word in the poster title, wrote "PNG bock" for "PNG back" and half-garbled street signs, while the chalkboard menu, the Chinese note and both layouts were right. Use it for drafts and proofread each picture.

Memory mode: pick the one that streams the right part

MOSS-Image is a pair: a 17 GB reader (the Qwen3.5-9B text encoder from the Qwen team, which turns the prompt into numbers) and a 10 GB painter. The reader is busy only briefly per prompt, so it can live in system RAM and move to the GPU on demand. To mimic smaller cards, we limited PyTorch's share of the A40 with torch.cuda.set_per_process_memory_fraction:

Memory cap	Peak on the card	50 steps	Turbo
None (real A40, 48 GB)	about 30 GB	80 s	9 s
16 GB, reader in main memory	about 14 GB	86 s	13 s
12 GB, reader and part of the painter in main memory	about 11 GB	160 s	18 s

The generic enable_model_cpu_offload() from diffusers fit 24 GB but not 16 GB and ran slower than moving only the reader, so our automatic mode never picks it. A cap copies a card's memory size, not its speed, so trust the fit more than the timings. Neither the fit on a real card nor the automatic choice has been tested yet: we set each mode by hand.

What to measure before you trust a setup

The memory ladder and the per-picture measurements list everything. These checks carry over:

Byte equality. The static is drawn on the CPU from the seed, and offloading only moves weights. All 8 ladder pictures matched the uncapped run byte for byte (cmp). If yours differ, something besides placement changed.
Peak memory as the card reports it. nvidia-smi, NVIDIA's monitor, counts memory the CUDA runtime keeps for itself, which the PyTorch cap ignores. The 16 GB setup leaves about 2 GiB spare, less with a display attached.
System RAM. The 16 and 12 GB setups used up to 22 GB. Plan for 32.
Time to ready. Ours took 63 seconds, 29 of them downloading 28 GB. A custom backend gets 300 seconds, and at 100 MB/s the download alone takes about 280. On machines we rent for you that limit is fixed for now. Status: open.
Every word, by eye. 22 pictures, one seed each, show behaviour, not a ranking.

The authors' technical report claims 96.3% average text coverage on their own benchmark, the best among open-weight models. We read only the abstract. Our menu was right in both modes, the kind of prompt where Iris-3B, the last image model we tested, dropped decimal points.

Credits and the full write-up

OpenMOSS at MOSI.AI made the model, pipeline, user guide, layout format and report. The Qwen team made the text encoder, Black Forest Labs the FLUX.2 small VAE that decodes the painter's output into pixels. lodestones trained the 8-step add-on and Kromoss, darask0 and akhaliq the 4-step and 3-step adapters. The ComfyUI team added native support, Comfy-Org made quantized repacks, the hugging-apps Space showed how to apply the LoRA. We spotted the model in a video by AI Search. Our part: the backend, the SDK example and the measurements. Deploy commands and open issues are in the MOSS-Image hands-on test with every timing and memory figure.

Nemotron 3.5 Bolt on one GPU: what the 24 GB covers, the drafter flag and the llama.cpp trap

InferenceKey — Thu, 29 Oct 2026 06:20:56 +0000

NVIDIA's Nemotron 3.5 Bolt 74B-A9B 24GB is a 74B agent model that reads text and images. Only about 9B of its parameters (the numbers a model learns in training) work on each token, and a token is roughly a piece of a word. Most are stored at about 2 bits each, so the file is 22.0 GB.

We served it on one A40, a 48 GB data-center card on our cloud, with llama.cpp, the open-source program most people use to run models on one machine. The image reader was on, and so was the drafter, a small helper model that speeds up writing. It used about 27 GB. For one user it wrote 136 tokens a second with the drafter and 70 without.

Three choices matter more than the model: what the 24 GB covers, which llama.cpp you compile, and when the drafter pays off. Every figure sits in our Nemotron 3.5 Bolt deployment write-up with the full measurements.

The 24 GB is a text-only budget for one user

NVIDIA's sum comes to 23.83 GB. That is 22.01 GB for the parameters, 0.87 GB of working memory for roughly 128,000 tokens of conversation, and 0.96 GB for the drafter. The image reader, another 1.68 GB, is deliberately outside the sum.

Two things eat into that margin. First, the unit: a card sold as 24 GB has 24 GiB. Read NVIDIA's figure as decimal and roughly 1.9 GB stays free; read it as GiB and only about 0.2 does. Second, the drafter took close to 2.4 GB on our A40, not 0.96. So a fit on a real 24 GB card stays unproven: we never put Bolt on one.

The recipes NVIDIA publishes for llama.cpp include the image reader, and the smallest of them asks for a 32 GB card. Our 27 GB (25.2 GiB), with the quick-start 98,304 tokens of conversation, counts the image reader too, so it does not check NVIDIA's text-only number. Measure your own server with the drafter on and off, in the unit your card is sold in.

Build the right llama.cpp, and build it in parallel

Bolt needs llama.cpp code from release day, 15 October. Kartik Gulia, an NVIDIA engineer, added the model and its image reader to llama.cpp, and ggerganov merged the change in llama.cpp pull request #30520 that same day. Older builds refuse the file.

We hit exactly that. Our worker (the agent that sets up each GPU machine for InferenceKey) pins a llama.cpp release 1,920 commits older than the merge, so we set v0.7.0 by hand.

Then compiling ate the clock. A fresh machine took 764.8 seconds from request to ready, and 705 of those went into building llama.cpp on 9 CPU cores. Our worker calls cmake --build with no parallel jobs unless CMAKE_BUILD_PARALLEL_LEVEL is set. Even with 12 jobs set by hand, the build was 92% of the wait. On a 31-core machine the same build finished in 2 minutes 41 seconds. A second copy on an already built machine was ready in 15.6 seconds. If first starts matter, count CPU cores, not only the GPU.

The drafter is a server flag, not a request option

Bolt comes with the LiLiCorr drafter, about 1.1B parameters in a 1.0 GB file. The drafter proposes several tokens ahead, Bolt verifies the whole batch in a single step, accepts the run up to its first mismatch and writes that token itself. You turn it on when the server starts:

--spec-draft-hf nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr-GGUF \
--spec-type draft-dflash --spec-draft-n-max 6 --spec-draft-ngl 999

On short and medium prompts it about doubled the speed: over 25 requests the median was 136 tokens a second against 70, with a floor of 110, and 70% of the drafted tokens were accepted. The per-prompt speed and acceptance table shows where it gains most.

Long documents flip the result: the drafter slows prompt reading by about a quarter. A question over a 25,000-token document took 17.5 seconds without the drafter and 21.5 seconds with it. And v0.7.0 has the per-request drafter control compiled out, so the choice is made once per server. Mixed traffic? Benchmark both shapes first.

Two traps outside the model

A card can report fine and still be useless. Our first two machines had RTX 5090 cards. nvidia-smi, NVIDIA's status tool, answered, but CUDA, the layer programs use to reach the GPU, failed to start. Our platform still marked the model ready, so we have no 32 GB numbers yet.
Hidden reasoning burns time. Bolt thinks before it replies, out of your sight. A one-sentence answer cost 374 tokens of writing time. NVIDIA's model page documents a switch to turn thinking off, and InferenceKey does not forward it yet.

What this does and does not show

The quality numbers are NVIDIA's: Bolt beats Qwen3.6-35B-A3B on most agent benchmarks (Terminal-Bench 2.1: 50.98 against 44.38) and loses on factual recall. NVIDIA's speed table puts it behind Qwen3.6 and Gemma 4. Our side was about 50 requests, one GPU, one request at a time; on a two-tool refund task it called both tools correctly in all 5 runs. We did not test several requests sharing the card, where drafters tend to help less, or vLLM, the data-center server software that is the other route.

NVIDIA built, compressed and released Bolt and its drafter under OpenMDW-1.1, which allows commercial use. Kartik Gulia and ggerganov made llama.cpp able to run it. AI Search's news video pointed us to the family. Our part was wiring it end to end and measuring it. The exact command, per-request timings and worker settings are in the complete Nemotron 3.5 Bolt article on inferencekey.com.

Price your coding agent from its own logs before you self-host DeepInfer

InferenceKey — Wed, 28 Oct 2026 06:20:34 +0000

Before you argue about self-hosting a model for your coding agents, read your own logs. Claude Code keeps a JSONL transcript of every model call under ~/.claude/projects, token usage included. We took one developer's transcripts from 21 September to 20 October: 129,407 calls and 26.2 billion tokens (tokens are the word fragments a model reads and bills by). Priced three ways, that month is 400 dollars on the two Claude Max 20x subscriptions he pays, 12,365 dollars at Claude API list price, and 274 dollars on DeepSeek's API.

The full breakdown of that month and the DeepInfer break-even math lives on inferencekey.com. This piece covers the trap you hit doing the same count, and the one measurement that decides whether self-hosting makes sense.

The first streaming line undercounts output by 2.4x

The answer arrives in pieces (streaming), and Claude Code logs a new line for the same call as those pieces land. The earliest line has no stop_reason yet, and its output_tokens holds only what was generated so far. Our first pass trusted it and counted 36.4 million output tokens. The final line of each call says 86.0 million.

Output is the priciest token on the list: 20 dollars per million on Opus 5.5, against 4 for fresh input. Deduplicate by message id and request id, and let the last line win:

import json
usage = {}
for line in open(path):  # each ~/.claude/projects/**/*.jsonl
    rec = json.loads(line)
    msg = rec.get("message") or {}
    if "usage" in msg:
        usage[(msg.get("id"), rec.get("requestId"))] = msg["usage"]  # last line wins

Do this before any pricing: every later figure inherits the error.

Your agent is a reader, so compare the cache price

On every step the agent resends the whole conversation. Providers keep recently seen text in a cache, a store that lets them skip work on text they processed a moment ago and bill it at a lower rate. In this month, 25,253 million tokens came back out of that cache. 822 million were new material written into it, and 86 million were output. That is about 303 tokens read per token written, and the median call carried 138,000 tokens of context (everything sent along with that call).

So the headline input and output prices are the wrong columns. The cache-hit price carries the bill. Opus 5.5 charges 0.20 dollars per million cached tokens; DeepSeek off-peak charges 0.003, 67 times less, and that explains almost the whole gap between 12,365 and 274. With no cache at all, the same month on Claude would come to about 109,900 dollars.

Two details if you reprice your own month. DeepSeek doubles its rates at peak hours, so use your team's real hours (here 9.4% of usage fell in peak and paid 17.2% of the bill). And keep it a price comparison: Flash and Opus are different models with different tokenizers, that difference is unmeasured, and nobody replayed this month of work on Flash. On quality, Flash topped LiveBench agentic coding among 69 models on 10 October, the same table we used when we ran DeepSeek-V4.1-Flash on two GPUs. Opus 5.5 leads the broader Artificial Analysis Intelligence Index, 58 to 39.

DeepInfer ships the engine, not the cache

DeepSeek open-sourced DeepInfer, the engine behind its own API, under Apache 2.0. It grew out of vLLM and takes ideas from SGLang and TensorRT-LLM. Credit to all of those teams, and to DeepSeek for documenting its production setup with real numbers in a post on Zhihu.

This release does offline inference: you feed it prompts from a Python script, and there is no documented HTTP server for your agents to call. The project page also lists three missing pieces of a complete service. One is the KV cache system, the store for already-processed context (the cache from the previous section). Another is the split between prefill machines, which read the prompt, and decode machines, which write the answer token by token. The last is load balancing across copies of the engine.

For this workload the cache is the whole game. With one, an 8-GPU server at DeepSeek's production rates holds about 293 developers like this one. Without it, each step processes the full context from scratch, and the same server holds about 15. The article budgets two full-time engineers to build the missing parts.

The one number nobody has measured yet

Every break-even figure hangs on one ratio: DeepInfer's speed per GPU on a single 8-GPU server, against DeepSeek's production, where decode runs on 64 GPUs or more. The README charts prefill on 4 GPUs and decode on 64. The 8-GPU case is missing.

Eight rented B200 at 7.99 dollars per GPU per hour cost 46,662 dollars a month. Below 58% of DeepSeek's per-GPU speed, renting never beats the API, whatever your headcount. At full DeepSeek speed, one server wins from about 262 people, engineers included, and is 11% cheaper at 293.

DeepInfer ships the benchmark (examples/sweep/run_sweep.sh), and a few hours of prefill and decode sweeps at 128K context on one 8-GPU node would settle it. We still owe that run, so treat every rented-server figure as an assumption. The own-rack estimate, about 465 people, rests on a 3.5 million dollar rack price that is our assumption, not a quote; the team-size table and every formula behind the DeepInfer break-even show what moves it.

Start with your own month

Count your team's month with the final-line rule above, then compare it with our month of coding agent costs and the DeepInfer payoff analysis.

Serving LFM2.5-24B-A2B on one 48 GB GPU: the flags, the traps and what to measure

InferenceKey — Tue, 27 Oct 2026 06:20:50 +0000

Liquid AI built LFM2.5-24B-A2B for agents on your own hardware. It has 24.0B parameters in total, the learned numbers that make up the model. Only 2.45B of them work on each token, the word piece a model reads and writes. Liquid ships an official FP8 checkpoint, a copy that stores most weights in 8-bit numbers, of 24.8 GB. The context, meaning the longest input plus output it accepts, is 131,072 tokens. On paper that fits one 48 GB card with room to spare. Three settings and one assumption decide whether the paper holds.

A plain limit first: our planned test on a rented 48 GB GPU never ran, so we have no numbers of our own. Benchmarks and throughput below are Liquid's, and memory figures are arithmetic on the config file. Our notes on LFM2.5-24B-A2B, its memory math and its license keep measured and computed apart.

The license comes before the GPU

The LFM Open License v1.0 sets a ceiling of 10 million US dollars of annual revenue for free commercial use. Parents, subsidiaries and sister companies under the same control all add to the count. A small team inside a large group is measured by the group, and indirect advantage counts, so even internal tooling can qualify. Settle that with your lawyer before you write serving config.

Why long contexts stay cheap

Mixture of experts (MoE) explains the A2B in the name. Each MoE block holds 64 experts, smaller sub-networks, and a router sends each token to 4 of them while the rest stay idle. Idle experts still occupy VRAM, the GPU's own memory, which is why the file weighs 24.8 GB even though about a tenth of the model works on each token.

While generating, a model stores keys and values for every past token so it never recomputes them. That store, the KV cache, grows with the conversation. Here only 10 of the 40 layers are attention layers that need one. The remaining 30 are short convolution blocks with a small state that does not grow. The article computes about 2.7 GB of cache for one conversation at the full 131,072 tokens. With every layer attending, you would pay four times as much. That assumes a head size of 64 (2048 divided by 32 heads), since the config omits it.

That leaves about 23 GB free after the weights. What vLLM, the serving engine, reserves for itself, plus CUDA context and activations, was never estimated, so "several long conversations at once" is a hypothesis to check.

Three settings that will bite you

The chat template opens every assistant turn with <think>, so the model always reasons first, and it writes tool calls as Pythonic lists between its own special tokens. vLLM runs it natively from 0.23.0, but needs telling how to split that output:

vllm serve LiquidAI/LFM2.5-24B-A2B-FP8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser lfm2

Version. Anything older than vLLM 0.23.0 does not know LFM2.5. Our platform defaults to 0.20.0, so the plan pinned a newer release for this workload.
Parsers. The reasoning parser is what separates the thinking from the final reply. The tool parser is what turns those Pythonic lists into tool calls your agent loop can act on.
max_tokens. Liquid advises leaving it unset unless you want a hard ceiling. Reasoning spends the same budget, so a tight limit can stop the model mid-thought.

Sampling needs no work: temperature 0.1, top_k 50 and repetition_penalty 1.05 ship in generation_config.json, which vLLM applies by default. Overrides for top_k, min_p and repetition_penalty go in extra_body.

The 2B saving is per token, not per server

Producing each new token, a step called decode, mostly means reading weights from GPU memory. A single user reads 4 experts per block per step, and the saving is real. A server batches many requests into each step, and each token in the batch may want a different set of 4. Read enough distinct experts and the step costs what a dense 24B model would, one where every parameter works on every token. That is reasoning, not measurement. It is also why Liquid's over 21,000 output tokens per second on one NVIDIA B300 under heavy concurrency is a total across requests, not one user's speed.

For a single user, Liquid reports an Apple M5 Max writing 184 tokens per second and an AMD Ryzen AI Max+ 395 writing 114, both in llama.cpp with a 4-bit GGUF file. Those rates include hidden reasoning that precedes the first visible word.

What I would measure first

On Liquid's benchmarks the model leads on agent workflows, tool use and instruction following. It ties Qwen3.6-35B-A3B on BFCLv4, 65.01 to 65.00, and trails it on SWE-Bench Verified, 51.00 to 66.20. Every rival is another MoE model, and no dense baseline is published.

The planned LFM2.5-24B-A2B benchmark on a 48 GB card is a script, bench.py. It covers streamed reasoning, code and Spanish prompts, a full tool-call round trip, checks that a fact buried at 8K, 32K and 100K tokens is found, and 4 and 8 parallel requests. The parallel runs decide whether the model behaves like 2B or like 24B. I would watch time to the first visible token as closely as tokens per second.

For a MoE model we did run, the Flint research post shows a coding agent measured on one RTX 4090 and one RTX 5090.

Credit

Liquid AI built the model on a 36T token training budget and refined it with reinforcement learning inside OpenClaw, Hermes Agent, Pi and OpenCode. It shipped with day-one support in llama.cpp, MLX, vLLM, SGLang and ONNX. The vLLM project maintains the recipe with the parser flags. AI Search's news roundup put the release in front of us. Our part was checking the files, doing the memory arithmetic and writing the test plan.

Checkpoint hashes, the full config.json architecture, quantization details, planned GPUs and the whole benchmark table are in the LFM2.5-24B-A2B article on inferencekey.com.

Building a speech editing API on AuK-Flash: the knobs it ignores and the wording it doesn't

InferenceKey — Mon, 26 Oct 2026 06:40:15 +0000

Speech pipelines usually chain separate tools: text to speech, cloning, noise removal, speaker separation. AuK-Flash replaces the chain with one model driven by a sentence. You send Replace 'twelve percent' with 'eighteen percent'. and a recording, and the same voice comes back saying eighteen. On an NVIDIA RTX A5000 that edit kept the graphics card (GPU) busy for 0.73 seconds.

AuK and AuK-Flash come from Ziyang Ma and 32 co-authors at Shanghai Jiao Tong University, Tencent Hunyuan and the Shanghai Innovation Institute. Our AuK-Flash test on a 24 GB card has the measurements. This piece covers the decisions you face wrapping it as an API.

Size the card for two models, not one

The AuK-Flash page lists about 6.8 GB. That is not what you deploy. AuK understands your instruction through a second model, Alibaba's Qwen2.5-Omni-3B (the "encoder"), which weighs about 12 GB more. A working install pulls 18.75 GB.

The memory on the card (VRAM) follows. We saw about 15.6 GB in use once loaded and about 20.3 GB at peak. Our own example's .env.example says "about 17 GB of free VRAM", which invites a 16 GB card. Plan for 24 GB.

A CPU offload setting, which parks part of the model in ordinary system RAM, cut the peak to about 11.3 GB. Each request then took 13 to 15 seconds, longer than the audio it produced. It might fit a 16 GB card, untested.

The encoder also carries the licence question. AuK's code and weights are MIT, but Qwen2.5-Omni-3B is under the Qwen Research Licence, which does not allow commercial use. On issue #13, project collaborator ZhikangNiu replied that the team plans a permissively licensed encoder in a future release. Settle this before a product depends on it.

The sampling knobs are locked, on purpose

AuK sharpens audio from a blurry sketch in repeated passes. The base model does 32, each also computing an unguided version to push the result toward your instruction (that push is "guidance"). AuK-Flash was trained to need 4 passes and no extra version: 4.5 times faster, by the authors' measurement.

The inference code (what runs the trained model on each request) hard-codes that recipe and ignores any step count or guidance strength you pass Flash. A source comment says re-enabling guidance "blows up the amplitude". If your wrapper exposes those parameters, hide them or label them base-model only.

Prompt wording is part of your API contract

In voice design you describe a voice in text and give it a line. Two phrasings that look interchangeable behave very differently:

Based on the following description: "...", generate speech content "...".
Generate speech based on the following description: "...". The content to speak is: "...".

The first comes from the project's command-line example: 7 of 8 English voice designs came out exactly right. The second is the cookbook's English template, and in 8 of 8 tries the model spoke part of the description out loud. The technical report recommends a separate language model in front to rewrite requests into the trained phrasing. We ran without one.

So never forward free-form text. Keep fixed templates on the server, fill the slots, and test each template like an endpoint.

One model is one queue

A single model is one artifact to ship and patch, and adding a task means writing an instruction, not integrating a tool. The price is concurrency (requests running at once). Our adapter, the code that loads the model and answers requests, allows one GPU job at a time: four edits fired together came back in turn, the last after 5.8 seconds. Warm calls (model loaded, first request done) used 0.4 to 0.8 seconds of GPU, so short edits queue fine. A long job blocks the next caller.

The cold start, the time from asking for the model to it being ready to serve, was 71.6 seconds on a GPU machine already connected to InferenceKey with no model files on it. The cold start and per-task timings for AuK-Flash break that down step by step.

Measure the words, then admit what a transcript can't see

We transcribed every output with OpenAI's Whisper (large-v3-turbo) and diffed the text. No edit touched the words around the change: a cheap regression test for any speech editing service.

Text cannot tell you whether the pitch went up, the whisper sounds like one, or the swapped word has an audible join. That needs an ear or a pitch measurement, and we used neither. Our inputs were synthetic voices from Kyutai's Pocket TTS (see our Pocket TTS Spanish voice on CPU), the friendliest possible case. Noisy rooms and overlapping speakers remain untested. Training covered Chinese and English only; our Spanish attempt was partly garbled. Benchmark scores (87.5 correct edits in 100 on SpeechEditBench) are the authors', not reproduced. Our run was around 60 requests, nearly all single attempts, none crashed.

What our part was

No ready-made InferenceKey backend handles AuK yet, so we wrote that small adapter ourselves. It pins Tencent's code, caps outputs at 30 seconds and answers on the chat address, because our text-to-speech address cannot yet return audio from a custom adapter, a gap on our side.

The AuK team built both models and wrote the report and the cookbook in the AuK repository. SGLang-Omni, vLLM-Omni and 0xShug0's audio.cpp each added ways to run AuK. Alibaba's Qwen team made the encoder. Issue reporters raised the licence, Polish and shouting questions. We spotted the release in AI Search's news video.

The full AuK-Flash write-up on inferencekey.com has the per-task table, every audio sample before and after, the platform issues we hit and the SDK example that chains a design, clone, edit and speed-up.

Serving a 6 GB ternary 27B model: the settings that decide whether it works

InferenceKey — Sun, 25 Oct 2026 06:37:52 +0000

A 27B reasoning model in a 5.95 GB file sounds like a download problem solved. Mostly it moves the problem into configuration. We served Ternary Bonsai 2 27B, Prism ML's build of Alibaba's Qwen3.8-27B, on one RTX 4090 with 24 GB. Serving means running the model behind an API that answers requests. It wrote about 104 tokens a second for one user (tokens are the word pieces a model reads and writes, about four for every three words) and peaked at about 12 GB of the card's memory. Four traps stood in the way, none of them about model quality.

The full command and every benchmark figure are in our write-up of Ternary Bonsai 2 27B on one RTX 4090.

Why the file is so small

A model is mostly a long list of learned numbers called weights. Prism ML stores almost every one as -1, 0 or +1, plus one ordinary 16-bit scale for each group of 128. That costs under 2 bits per weight instead of 16, so the file is about nine times smaller than the roughly 54 GB original. Before that conversion, the weights are rotated in a fixed pattern. At run time, the program serving the model must apply the same rotation to the values flowing between its layers.

Trap one: your usual runtime cannot read it

llama.cpp is the open-source program most people use to run model files like this locally. Prism ML's model card is explicit: "Stock llama.cpp will not run these files." Both Bonsai formats fail as unknown, and the known issues page says the popular desktop apps fail too. Older llama.cpp builds could load a related Bonsai file and then reply with gibberish, since they never apply the rotation. We did not try the stock program ourselves.

What you need is Prism ML's fork, its own modified copy of llama.cpp, which ships prebuilt downloads. On Linux, the known issues page says to take the build made with CUDA 12.8 (NVIDIA's GPU toolkit), because the newest version crashes on some machines. That advice predates the fork's 14 October release, so the crash may be fixed. We used 12.8 and saw none. Also, the Linux build needs two NVIDIA libraries it does not ship, so we added them from NVIDIA's Python packages.

Trap two: thinking eats your token budget

The model reasons in hidden text before it answers, and those thinking tokens count against max_tokens, the cap on how much it may write per reply. Set a small cap and the reply can come back empty. The card recommends 16,384.

{
  "max_tokens": 16384,
  "reasoning_effort": "medium"
}

A low client default is how a working model looks broken. With that cap, all 17 of our timed requests finished cleanly: no empty replies, no loops, nothing cut short.

Trap three: pick the thinking level on purpose

The chat template, the file that formats a conversation for the model, accepts low, medium and xhigh. Per the known issues page, high returns an error, so our custom backend (the small adapter that starts the model on InferenceKey) maps it to xhigh. Prism ML's whitepaper measures the headline 98.2%, Bonsai's score as a share of the original's, with maximum thinking. At medium, Bonsai scores 79.3 against 83.9 at maximum, and the original scores 82.6 at medium.

The card says medium gives shorter answers. Our two samples disagree: the seating puzzle took 1,447 tokens at medium and 670 at maximum. Measure on your own prompts before picking a default.

Trap four: the second user waits in line

On InferenceKey, a model pinned to a single machine handles one request at a time by default. Four questions sent together finished in 22.4 s, the last one stuck in the queue. After raising one setting (max_in_flight to 8), the same four finished in 9.6 s at about 159 tokens a second in total. The same default bit us in our AstaBrief-8B test, so check it wherever you serve.

On the server, the context (how much text the model holds in view) is 65,536 tokens. With kv_unified, that is one pool shared by four slots, one per simultaneous conversation, not 65,536 tokens each.

What to measure, and what the numbers do not say

Memory is the number people get wrong. Serving took about 12 GB, roughly double the file, because the conversation memory and the image reader live on the card too. The per-request timings and memory readings are in the article.

Quality is the number people test wrong. A conventional 2-bit shrink of similar size still scores 88.93 on an easy knowledge quiz, yet drops to 57.5 on a hard math test where Bonsai scores 95.83. On a benchmark of real bug fixes in real repositories, Bonsai scores 60.8 against 80.6 for the original. The known issues page also reports malformed or looping tool calls (the model asking your code to run a function), which we did not test.

Every quality figure here is Prism ML's. Ours came from roughly 20 requests, each question tried once or twice. Unsloth's standard 4-bit build (17.6 GB) also fits a 24 GB card and keeps a little more quality, 98.7%. The case for the 6 GB file is the spare memory it leaves, not the fit.

The Qwen team trained the base model. Prism ML designed the format, borrowed the rotation from Meta's SpinQuant work, built the fork and documented its issues in detail. Mobius Labs' compression method made the image reader smaller. Unsloth published the 4-bit build we compared against. The llama.cpp project is the foundation of the fork. An AI Search video is where we first saw the release. Our part was the custom backend, the SDK example and the measurements.

The launch command, the per-GPU speed table and the platform issues we hit are in the complete Bonsai 2 27B deployment notes.

Muse Spark Open vs DeepSeek V4.1 Flash in Claude Code: what to measure before you pick

InferenceKey — Sat, 24 Oct 2026 05:31:38 +0000

"Verified data.json is fetched live and all four requested metrics are shown." Muse Spark Open sent that line from inside Claude Code 16 seconds after we asked for a small dashboard page. The verification was a text search for a few expected words. Muse never rendered the result, and every bar on it was empty. DeepSeek V4.1 Flash needed 2 minutes 29 seconds, loaded its page in a headless browser (Chrome with no window, driven from a script), studied the screenshots and delivered a dashboard that worked.

Meta made the Muse Spark Open weights public on 15 October, and both models went through identical coding tests on the 16th. The full Muse Spark Open vs DeepSeek V4.1 Flash comparison has the video, every table and the run records. Here we look at which numbers fool you when you plug models into coding agents.

A 16-second "done" with an invisible bug

The bug survives a quick read of the code. The fill only gets a colour from rules written as descendant selectors, CSS that matches an element only when it sits inside another one. Muse placed the bar beside the row instead of within it:

<div class="row c0">...</div>
<div class="bar"><div class="fill"></div></div>
<style>.c0 .fill { background: ... } /* matches nothing */</style>

The width animates on an element with no background. Open the page and you notice within a second. Grep the file for strings and you never will.

DeepSeek slipped too, in a milder way. One card footer calls two-task totals "each". The figures are right; the wording is not.

Tokens per second measures a service, not a model

Models read and write in tokens, chunks of a word, so speed is quoted in tokens per second. In the speed round, Muse on Baseten put code on screen in 2.5 seconds, streaming 455 tokens per second. DeepSeek streamed 197 per second and burned its whole 1,500-token allowance on private reasoning, ending with no code. Artificial Analysis, taking the median across every provider, finds the opposite: 179 tokens per second for DeepSeek, 102 for Muse.

Baseten serves an NVFP4 copy of Muse, the weights squeezed into 4-bit numbers so they take less memory and move faster. Its usage data also reports speculative decoding, where cheap guesses at upcoming tokens get checked by the big model in one pass. Each probably explains part of the lead; the test cannot separate them. Benchmark a hosted model and you are benchmarking its host.

The other trap is when you start the stopwatch. Time to first token, the usual latency number, counts reasoning tokens that never reach your screen. In a separate morning test, DeepSeek's first token showed up in under a second, yet its first readable words took 3 to 6 seconds. Muse needed about one. If your agent should feel responsive, time the first answer token on its own.

Cost per task, not price per token

LiveBench, a public test set, supplied eight coding problems from AtCoder. Each model solved seven, and both failed on the same problem. Muse produced roughly 88,000 output tokens against 59,000 for DeepSeek, at a higher rate per token. At list prices that put the eight problems at 0.38 USD for Muse and 0.07 USD for DeepSeek. In small Claude Code jobs on a toy banking library, both passed every hidden test, at 0.148 USD per task for Muse and 0.016 USD for DeepSeek. The DeepSeek numbers assume peak pricing, so treat them as ceilings.

Reasoning is billed as output, so a low token price means little until you know how long a model thinks.

The shim's settings are part of your experiment

Claude Code sends requests in Anthropic's format, while Baseten accepts only OpenAI's. A LiteLLM proxy translated between the two, and its config set Muse's reasoning effort, the knob for how long a model thinks, to medium. DeepSeek has its own Anthropic-compatible endpoint and kept its default effort, high. The dashboard round was therefore uneven. Whether Muse at high effort would have tested its own page is an open question, and we did not run it.

Record every setting a translation layer applies and count each as a test variable. The proxy config and the Claude Code setup for each model sit in the engineering appendix of the Muse vs DeepSeek article.

What to collect on your own shortlist

Before committing to a coding model, we would measure:

time to the first answer token, not the first token
output tokens per solved task, reasoning included
cost per task at the price and hour you will actually pay
whether the agent opens what it built before it says it is done

Limits: each test ran once, on a single day: eight problems and three Claude Code jobs, on public problems either model may have seen in training. The speed result describes Baseten, not Muse in general. Public weights do mean you can run Muse on your own hardware, but the FP4 weights take about 408 GB, and we have not tried. We had already run DeepSeek V4.1 Flash on two graphics cards in an earlier test.

Credit

Meta built Muse Spark Open and opened its weights, and Baseten had it serving on test day. DeepSeek made V4.1 Flash, released the weights and runs it on its own API. The problems and their tests come from AtCoder; the LiveBench team assembled the set on top of LiveCodeBench. Artificial Analysis maintains the independent scores. Anthropic builds Claude Code, and without the LiteLLM project's proxy, Claude Code could not have reached Muse. We wrote the test scripts, graded the answers and configured each model.

For the per-problem timings, the price breakdown and the exact prompts, read the complete Muse Spark Open and DeepSeek V4.1 Flash test on inferencekey.com.

Self-hosting AstaBrief-8B for cited reports: check every bracket before anyone reads it

InferenceKey — Fri, 23 Oct 2026 05:36:23 +0000

You have search over your own documents. Next comes a report with a reference after every claim, and the Allen Institute for AI (Ai2) released AstaBrief-8B for that writing step. The weights (the trained model file you download) are Apache 2.0, and the base is the Qwen team's Qwen3-8B. We ran it on one NVIDIA RTX A5000, a 24 GB card, and counted every reference it wrote.

Serving it is easy. One check in the code around it is not optional.

Decision one: the prompt is not yours to write

Ai2 trained the model on one fixed instruction file. The model card warns that swapping it may give "degraded or inconsistent behavior". So your code fills two blanks, the question and the excerpts, and keeps even the line "Note that the current year is 2025". No report stated the wrong year; whether it ranks 2025 papers as the latest work, we left untested.

Each excerpt carries a label: an ID, the first author, the year and a citation count (how many other papers cite it). The model copies it into square brackets after each claim, and should write "(LLM Memory)" when it relies on its own recollection.

Check the licence first. Apache 2.0 covers the weights only; the instruction file is CC-BY-NC-4.0, non-commercial. A paid product must clear that licence or write its own prompt in the same format, with the risk the card describes. Our SDK example ships no copy: it fetches the file from Ai2 at run time, under Ai2's terms.

Decision two: size the context, then let it batch

vLLM, the open-source program that loads a model onto the GPU and answers requests, treats AstaBrief as an ordinary Qwen3 model, with no custom code. Text is counted in tokens, about three quarters of a word each. The weights fill 16.4 GB of the 24 GB. We capped each request at 16,384 tokens (question, excerpts and report), the length of Ai2's final training round. The leftover memory holds two requests at that cap side by side. Our real requests were nearer 7,000 tokens, so about four fit.

Alone on the card, one report took 41.5 to 64.7 s, at roughly 39 tokens a second. Sending four is more telling.

Sent together, four reports took 85 s, against 209 s one after another. The first try missed that, through a bug of ours. Concurrency is how many requests a model may work on at once, and pinning a model to one GPU machine through our SDK set it to one. The four waited in line: 198 s, and one report vanished. One settings call raises the limit today. The vanished report revealed a second, unfixed bug. If the first token (the opening piece of the answer) takes over 3 minutes, the request closes empty with no error. The card still finishes a report nobody receives. The AstaBrief-8B test notes on inferencekey.com cover both, plus the workaround.

On your stack, test the concurrency limit at each layer, first-token time under load, and whether a timeout fails loudly.

Decision three: the label check is part of the pipeline

The 11 reports held 380 references, and 363 (95.5%) match an excerpt we sent. Good, and also the trap: the other 17 look just like the genuine ones.

Eight of the 11 reports cited 1 to 3 labels the prompt never contained. Part of them were our own excerpts with the label garbled. We sent [W4411306417 | Thapliyal et al. | 2025 | Citations: 45] and got back [4411306415 | Thapliyal et al. | 2025 | Citations: 111]. The rest were recalled papers, dressed in the same format and missing the tag ("(LLM Memory)" showed up once across 11 reports). Plain numeric IDs cured the missing W, not the wrong labels.

You know every label you sent, so the check is a set difference:

import re
sent = set(labels_in_prompt)                      # exactly as you formatted them
cited = set(re.findall(r"\[([^\[\]]+)\]", report))
unknown = cited - sent                            # flag before a human reads it

Two cleanups belong there too. Each report opens with an empty <think> block, the slot Qwen3 models reserve for reasoning when that mode is enabled (turning it off was not part of our test). And the per-section summaries use slightly different markers than the prompt requests.

What the 95.5% does not tell you

The score proves each reference exists, not that claims are correct. We did not compare each sentence with its excerpt, yet drift turned up anyway. One report gave a paper 25 citations instead of 27, seven times over. Another turned "one eighth" of a cost into "1/4 cost". Ai2 flags the same weakness: the right study, cited for more than it found. Have a person review figures taken from excerpts.

The test was small: 4 questions, 15 OpenAlex abstracts each, inputs around 5,000 tokens. Ai2's quality score (87.0 out of 100, against 77.3 for plain Qwen3-8B) covers computer science questions only. We graded none of the reports, and ran no head-to-head against another model.

For the retrieval side, see our EmbeddingGemma 2 post on multimodal search. The full AstaBrief-8B write-up with the vLLM command, SDK example and per-request timings also has the complete citation tally.

Credit: Ai2's Asta team built, trained, evaluated and released AstaBrief and its prompt, using reports from Anthropic, OpenAI and DeepSeek models, with GPT-4.1 and DeepSeek-R1 as graders. The Qwen team made Qwen3-8B, the vLLM project serves it unchanged, Hugging Face hosts the weights, OpenAlex supplied the abstracts, and an AI Search video pointed us to the release. Our part: the SDK example, the run on our own cloud, the counting. Read the complete AstaBrief-8B report on inferencekey.com before you wire it into your stack.

Three settings that decide whether EmbeddingGemma 2 search works on your GPU

InferenceKey — Thu, 22 Oct 2026 05:32:11 +0000

EmbeddingGemma 2 is an embedding model Google DeepMind released on 6 October under Apache 2.0. An embedding model turns an input into a list of numbers (768 here) so that similar meanings get nearby lists, and search becomes "find the nearest list". This one takes text, images, video and audio into the same space, from a 1.5 GB download of 740M parameters.

We served it on a 16 GB NVIDIA RTX 2000 Ada on our own cloud. Loading it was easy. Three settings decided whether the vectors were correct and latency reasonable, so settle them before you write the indexing job.

The number format that fails without an error

GPUs can store numbers in compact 16-bit formats, float16 and bfloat16, to halve memory. Google's model card (the documentation shipped with the weights) allows only bfloat16 or float32. In float16, some values inside the network get too large to represent, and the card says you get NaN or subtly degraded embeddings with no exception. In a search index that means no crash, just worse rankings.

The card's suggested selection fits in one line:

# never float16: NaN or quietly wrong vectors, no exception
dtype = torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float32

On a GPU without native bfloat16, the float32 fallback needs twice the memory. All our runs used a safe format, so the float16 failure is Google's warning, not something we reproduced. That run is our next test. Log which dtype your stack actually loads.

Photos were slow because the container saw 64 cores

Our first backend handled images two to five times slower than it should have. A profiler showed about 72 ms of GPU work per image against 0.33 to 0.44 seconds of CPU preprocessing. The container's CPU quota (the share of the host's processors it may use, set through Linux cgroups) was about 7 cores, but PyTorch counted the host's 64 and started 64 threads to fight over 7.

Limiting threads to the quota took a single image from 531 to 111 ms, eight images from 4,333 to 867 ms, and a 6-second video clip from 754 to 307 ms. Text and audio stayed almost flat, so a text-only benchmark never shows this.

Our example backend derives the thread count from the cgroup quota. Without a quota, you set it by hand. We verified this on one container only, so time CPU preprocessing and GPU encoding separately, per input type, on yours.

Freed GPU memory stays reserved

The complete model occupied 1.4 GB of GPU memory, or 0.5 GB with only the text part loaded. Single-input requests peaked at 2.6 GB. Batching is where it grows: eight documents of roughly 6,000 tokens each (a token is about a word or part of one) in one request peaked near 10 GB.

The surprise came afterwards. PyTorch caches released GPU memory for reuse, so the process held 11.6 GB until a restart. Calling torch.cuda.empty_cache() after very large requests fixed it: the same batch now leaves the process at 1.6 GB. For bulk indexing, size the GPU for your biggest batch.

Shorter vectors: same speed, some quality loss

Google trained the model with Matryoshka Representation Learning, so the first 512, 256 or 128 numbers of each vector work as a smaller embedding. Truncating was not faster for us (34 to 35 ms per query at every size). Only storage and payload shrink.

The quality cost depends on the input type, per the model card:

Benchmark	768	256	128
MTEB multilingual v2 (text)	61.36	60.41	57.89
MMEB v2 (images and video)	59.01	56.24	45.65
MSEB retrieval (speech)	69.54	66.76	56.71

The card calls 256 close to lossless. At 128, text holds, but image and video search drops by almost a quarter, so the card keeps 128 for text. Re-normalize each vector after cutting it (scale it back to length 1), and embed queries and documents at the same size. Our EmbeddingGemma 2 measurements of VRAM and latency per input type show what each option costs on a 16 GB card.

What the search test does and does not show

Our test index had twelve files and zero text: photos, short videos and spoken recordings. Six English queries each ranked the correct file first. "How to knead bread dough" matched a Spanish recording of a bread recipe. We generated all twelve files, so this confirms the cross-modal pipeline runs end to end, not how the model compares with others. The images came from Sperid Labs' Iris-3B, which we ran in an earlier test of the Iris-3B image model, and the voices from Pocket TTS.

On serving: vLLM, the open-source inference server, did not support this model in version 0.31.0 when we checked, not even for text, so we wrote a small Python backend on sentence-transformers. Google's launch post names vLLM, SGLang, Ollama and llama.cpp as supported tooling, so check current versions first. The complete EmbeddingGemma 2 deployment write-up, with latency and VRAM tables for every input type, covers the request format, start-up times and all five problems we ran into.

Credits

Google DeepMind built EmbeddingGemma 2: the model, its image, video and audio components, the benchmark results and the float16 warning. Sahil Dua and Henrique Schechter Vera wrote the launch post. The truncatable vectors rely on Matryoshka Representation Learning by Kusupati and colleagues. Hugging Face transformers supports the model from version 5.19.0, and sentence-transformers runs it. PyAV decoded our audio and video, and ffmpeg cut the clips. We wrote the backend, took the measurements and fixed the thread count and the memory cache.

The backend code, the SDK example and every number above are in the EmbeddingGemma 2 deployment article on inferencekey.com.

Fifty API keys, one queue, ten GPU workers: the parts of a shared LLM that are not the model

InferenceKey — Wed, 21 Oct 2026 05:40:36 +0000

If you are the engineer asked to put one self-hosted model behind a whole company, the model choice is the short meeting. The long tickets are about keys, queues, alerts and the invoice. We recorded a demo of a private ChatGPT shared by fifty teams with exactly that setup: fifty keys, one endpoint that speaks the OpenAI API format, ten GPU servers, and a cost screen that splits an hourly bill per key.

One disclosure first, because it shapes everything below: the ten "H100" machines (an H100 is a high-end Nvidia GPU) were test-mode processes on one ordinary host, returning filler text instead of running a model. The panel, keys, queue and accounting were the real product. So this run says a lot about the management side and nothing about how fast a real model answers.

Scope every key to one endpoint

The demo issues a key per team, often per use case: roadmap, competitive intel, security policy Q&A, board reporting. Fifty in total, and each one is allowed on a single endpoint, nexora-private-gpt/enterprise-chat.

Two reasons for that. A leaked key can only reach the company chat, not anything else the project runs. And the key is the unit of accounting, so a key per use case is what lets you tell finance that board reporting, not "finance", is the heavy user. Offboarding a team is unticking its key. How keys, scopes and environment variables work is in the Tokens & API keys guide.

Because the endpoint accepts the same request format as OpenAI, existing clients only change a base URL and a key.

Let workers pull, do not route

In the panel, the served model is a workload and each machine serving it is a worker. Ten workers serve one workload, and each takes up to 8 requests at the same time (its concurrency), so the fleet has 80 slots. All fifty keys feed one queue and whichever worker has a free slot claims the next job. You decide which machines serve the model, and the queue decides which one answers a given request.

That pays off when you scale: you pick a registered machine or a new cloud GPU, set its concurrency, and no team changes its key or URL. Nobody clicked "Add worker" while the traffic was running, and the write-up of the fifty-team demo lists that, and a worker dying mid-load, among the scenes it has not filmed yet.

Read the backlog trend, not the backlog

The backlog is the number of requests waiting for a free slot, and throughput is how many get finished per minute. A backlog of 20 says little alone. One that keeps climbing is the signal to add a machine. The drawer turns that into a label. It stays on "Healthy flow" as long as throughput is above zero and the backlog, compared over the last complete minutes (five at most), has grown by no more than 25% or 2 jobs, whichever is larger. A falling backlog with throughput going up shows "Recovering". A rising one shows "At risk", and "Critical" when throughput falls at the same time.

A trap on the same screen: the jobs-per-minute counter in the header only counts completions your open tab has seen in the past minute, so it restarts at zero on every page load. In the take it climbed from 0 to about 3,700. The real load is the drawer's five-minute average, 4,252 finished per minute, from a script pushing about 70 requests a second.

The bill is two lines of arithmetic

Nothing is priced per word. The panel prices reserved worker time and only uses tokens (the chunks of text a model reads and writes, roughly three quarters of a word each) to split the total:

project_cost = reserved_hours * hourly_price          # 134.53 h * $2.4306/h = $326.97
key_cost     = key_tokens / project_tokens * project_cost
               # 17.37M / 93.07M = 18.7% -> $61.03 for finance-board-reporting

The hourly price is $1,750 a month over 720 hours. Ten workers cost $24.31 an hour, about $17,500 a month if they never stop. A worker accrues hours for the whole time it stays assigned to the project, busy or not. The total moves by about a cent every 1.5 seconds, so screenshots taken seconds apart differ by a few cents. Do not write a test that expects them to match.

Do not quote the per-token number

The panel shows $3.51 per million tokens. Do not put that next to a public API's price. Each worker was reserved for about 13.5 hours, and at its steady rate the test traffic would have filled only around 72 minutes of that, so the figure mostly measures machines waiting. When you pay by the hour, that unit cost drops the busier you keep the fleet, and on real GPUs it comes down to how many tokens per second each machine can produce.

What you would still have to measure on real GPUs:

tokens per second per machine for your model and your answer lengths
how close to the 80 slots you can run before latency hurts (the script never sent more than 32 at once)
error behavior, since the 0 errors and the 250 ms p95 latency (95% of requests answered at least that fast) come from the test mode
each team's real share, since the script sent most traffic from a few keys and tokens were counted on filler text

Like a soundcheck with the PA on and no band, it tells you the cables and the desk work. It does not tell you how loud the room gets.

The full article on one private assistant for fifty teams has the video, every panel screen and the full cost table per key.