Sending each conversation home.

Why it matters

Claude Code and Codex work in turns. Each turn sends everything so far: the instructions, the tools, every file read and every answer. A model server such as llama.cpp keeps a prompt cache. If it has seen the start of a prompt before, it skips reading that part again, and the reply starts much sooner.

That works on one server. Run two copies of the same model, to serve more people or more agents at once, and a simple balancer sends turns to whichever copy looks least busy. On an idle install that means every other turn lands on the copy that has never seen the conversation, and reads it all again.

We measured that first: one Claude Code session on two copies of an 8B model reused 52.8% of its prompt tokens and took 6.0 seconds a turn. Sent back to the same copy each turn, the same session reused 75.3% and took 3.2 seconds (record, §7 and §11).

Then Troy, who builds Eugene Plexus, asked the harder question: what about dozens of people, across dozens of servers? This page is what we found.

How we measured

• Real sessions. We captured 32 Claude Code (2.1.283) and 32 Codex CLI (0.130.0) sessions, each eight tool turns over real source files. A Claude Code session grows from about 19,700 to 31,000 prompt tokens.
• Replayed at once. Each session runs as its own loop: send a turn, wait for the answer, pause a moment as an agent does, send the next. Up to 64 run at the same time.
• Real servers, real install. Two to eight llama-server processes (llama.cpp b11211) on one RTX 5090, behind a real Eugene install: the gateway, a driver for each server, the agent.
• The server's own count. The headline number is how many prompt tokens the servers actually computed, read from their own counters, not estimated by us.
• Every placement on fresh servers. Each policy ran on newly started servers and a new gateway, with the same sessions and the same pauses.

Two runs of the same policy differ by about 1.5 points, so smaller differences below are not results. The full method and every table are in the measurement record.

What we found

1. Sending a conversation back to the same server is the first win

Three copies of a model, twelve agents at once. Choosing the least busy copy reused 67.7% of prompt tokens, with the first token after 1.49 seconds at the median. Sending each conversation back to the copy that served its last turn reused 80.8%, after 0.71 seconds. The servers read 41% fewer tokens.

2. At eight servers, where a new conversation starts matters as much

"Least busy" counts requests in progress. An agent spends much of its time thinking between turns, so a server whose four agents are all thinking looks empty, and new conversations pile onto it. Counting the conversations a server holds instead fixed that. (Once the token budget in finding 3 was in place, most of this gap closed; see what we built.)

3. Under load, the failures were ours

The last column is the surprise. A llama.cpp server shares one block of memory between its slots. When the prompts in progress add up past it, the server refuses the next one, or cuts a reply off part-way.

Our gateway treated that refusal as a broken server and stopped sending it anything for a while. Then the next server got the turn cold, read more, filled up, and was cut off in turn.

On an 8B model with four agents per copy, that turned a busy moment into an outage: 46 of 204 turns answered, and most failures were healthy servers being refused as "cooling down" (gateway #8).

The fix has three parts. Eugene now reads each server's memory size from the server itself. It keeps a running total of what each turn in progress needs, and keeps a tenth spare. A turn that will not fit waits a moment for room, rather than failing somewhere cold. And a full server is reported as busy, not broken. Every turn on that 8B now answers: 204 of 204.

4. Pinning conversations to slots can freeze the server

llama.cpp lets a request name the slot it runs in. Pinning each conversation to its own slot looked like an easy gain, and it is, by about 3 points, when the server has memory for a whole conversation per slot. Where it does not, it costs up to 10.

Pinning a request to a slot that is still busy sometimes froze the server: nine turns in 200 waited ten minutes, and once a server answered nothing for half an hour.

The rule that removed it: never name a busy slot, and hold the turn in front of the server until its slot is free. With that rule, none of 612 turns stalled. Pinning stays off by default, as a per-model setting.

5. The cleverer ideas measured worse

We tried three placements that sound smarter. None beat the simple one.

Prefix hashing groups conversations by how their prompts begin. It does well when users share a long common start, but agent prompts share less than you would think. Two Claude Code sessions in the same project share about 17,900 tokens. In two different projects they share about 670, because the project's folder name appears early in the system prompt. Two Codex users share about 9,600 of 11,000, whatever their project.

What we built, and what it measured

All of this is on by default in Eugene's gateway in v0.1.0 (design):

• Conversation affinity (conversationAffinity, on). A conversation goes back to the copy that served it, for 30 minutes, unless that copy is saturated.
• Spread (loadBalancing: spread, the default). A new conversation goes to the copy holding the fewest conversations per slot.
• A token budget per server, read from the server's own settings, with a short wait for room.
• A full server is busy, not broken. It does not trip the circuit breaker, and Claude Code and Codex are told to retry rather than stop.
• Evictions on the Metrics page. A turn that went home and found its history gone is counted per server: a sign that model needs more memory or another copy.
• Slot pinning, off by default, with the never-a-busy-slot rule.

Then we ran the same tests against the built code. The honest summary:

• The 8B outage is gone: 204 of 204 turns, none refused as "cooling down".
• On eight copies, spread reused 78.3% on average over four runs, with no turn failed for lack of memory in 2,176.
• With the budget in, spread and "least busy" place new conversations about equally well (78.3% against 79.4%, inside the noise). The budget itself sends a new conversation where there is room, which is most of what spread did. Spread keeps one measured edge: no memory overflow in four runs, against eight in one of three for least busy.
• Three turns in 3,808 still failed. In each, llama.cpp paused for 6 to 10 seconds while it saved and restored its cache, enough times in a row to trip our 30-second stall check. Placement cannot fix that one.

So the two things that matter most are the simplest: send a conversation home, and do not send a server more than it can hold.

What we have not measured

• Separate graphics cards. Every copy shared one RTX 5090, so a policy that reads less is faster for everyone. Separate cards would show less of that.
• Bigger models at scale. The eight-copy runs used a 1B model; the 8B ran on three copies.
• Long sessions. Eight tool turns, up to about 31,000 tokens. Real agent sessions run far longer.
• vLLM and MLX under load. vLLM shares cached prefixes across requests by itself, and MLX keeps several caches; neither was run concurrently here.

Who else does this

We are not the first to route by cache, and we want to say so plainly. Checked on 2 October 2026:

In the data centre, it is an established idea

The SGLang Model Gateway routes by an approximate prefix tree by default. llm-d, the Kubernetes Gateway API Inference Extension, NVIDIA Dynamo, the vLLM production stack, AIBrix and Ray Serve all do some form of it, and research such as Preble studies it. Dynamo and llm-d go further than we do: the engine reports exactly which blocks it holds.

They are built for vLLM, SGLang or TensorRT-LLM, mostly on Kubernetes. They mostly place by prompt prefix, which suits many users sharing one long system prompt. Our agent sessions share less (finding 5), so on this workload prefix placement did not win. On theirs it may well.

Sending a conversation home exists too

LiteLLM has session affinity that pins a conversation to one deployment, reads Claude Code's own session header, and falls back when that deployment is cooling down. GPUStack, which runs llama.cpp-based servers, turned session affinity on by default for models with several copies on 30 September 2026. Small projects such as llama-balancer keep a client on one llama-server to reuse its cache.

What we think is ours

We found no other project that does these together:

• Balancing built for llama.cpp's llama-server on ordinary machines, with no Kubernetes.
• A per-server budget read from the server's own settings, so turns wait for room instead of overflowing it.
• A full server treated as busy, not broken, so one busy moment does not take healthy servers out.
• Slot pinning that never names a busy slot.
• The same gateway in front of Ollama, LM Studio, vLLM and cloud services, with failover between them.

That is a search that came up empty, not a proof. If we missed a project, tell us and we will add it here.