Ten million tokens. Four gigabytes.
memfree is an add-on for vLLM that serves conversations of a million tokens and more in about 4 GB of GPU memory each. Your customers get long context. Your cards hold 13 times as many conversations.
Drawn to scale. memfree's memory, and the standard figure at 128K, were measured on one NVIDIA H200 on 7 November 2026. The standard figures at 1M and 10M are the model's own cache for that many tokens, worked out, because no single card can hold them.
Measured on one H200, side by side
The same card, the same model and the same 48 questions at every length. One setup keeps the whole text in GPU memory. The other runs memfree.
| Measure | Standard serving | memfree |
|---|---|---|
| Right answers at 128K tokens | 45 of 48 | 48 of 48 |
| Right answers at 1M tokens | Does not fit on the card | 48 of 48 |
| Right answers at 10M tokens | Does not fit on the card | 48 of 48 |
| GPU memory per conversation | 44 GB at 128K | About 4 GB, any length |
| Conversations per H200 at 128K | 2 | 26 |
| Card time to make a text answerable | 207 s for 128K | 17 s, up to 10M |
Long context is a memory bill.
A model keeps a cache for every token of a conversation, and that cache sits in GPU memory, the scarcest thing in a data center. For a 70B model it comes to about a third of a megabyte a token.
So clouds cap context, ration it, or price it out of reach, and a card that could serve two dozen long conversations serves two.
- 128K tokens
- 44 GB
- 1M tokens
- 344 GB
- 10M tokens
- 3.4 TB
- What one H200 holds
- 141 GB
The cache for one conversation with Llama 3.3 70B at 16-bit, beside the memory of one H200.
memfree takes the memory bill out of long context.
Memory stays flat
About 4 GB per conversation, whether it holds 128 thousand tokens or 10 million. Length stops deciding what fits on a card.
Answers stay right
48 of 48 right at a million tokens and at ten million, on one H200. A standard setup holding just 128K got 45.
Your stack stays yours
memfree runs in front of vLLM on your machines. vLLM, your models, your API and your customers' data stay where they are.
Ready in under a minute. A ten-million-token text becomes answerable in about 50 seconds, using 17 seconds of card time. Reading just 128K the standard way takes 207.
Survives restarts. A ten-million-token conversation reloads from disk after a restart and answers right in 17 seconds.
One H200, two ways
With the model loaded, the rest of the card is room for conversations. At 128K tokens each, standard serving fits 2. memfree fits 26.
One H200's memory, filled with 128K-token conversations
Conversation blocks drawn to scale from the memory measured on 7 November.
Fits in front of the stack you already run
memfree is a compiled add-on that sits between your customers and vLLM. Nothing inside vLLM is patched or replaced.
Install it beside vLLM
We hand you a compiled add-on for your GPU nodes. Your engineers keep control of the deployment.
Route long conversations through it
Your endpoint stays the same. memfree takes the long part of each conversation and keeps its memory flat.
Sell longer context on the same cards
Offer a million tokens and more without buying memory for it, and fit more paying conversations on every card.
Run a pilot on your own cards
We install memfree on one of your GPU nodes, run your long-context work with and without it, and hand you the numbers: memory per conversation, conversations per card, right answers and speed.
What we'll ask you
- Which GPUs you run, and how many
- Whether you serve with vLLM, and which models
- How long your customers' contexts get
- What the pilot should prove to you
- Who we should talk to
Questions clouds ask us
Does memfree change vLLM?
No. memfree sits in front of vLLM. vLLM, its scheduler and its own cache run exactly as you run them today.
Which models does it work with?
Open models you serve with vLLM. The numbers on this site use Llama 3.3 70B Instruct. In a pilot we measure the models you actually serve.
Which GPUs?
NVIDIA Hopper first: H100 and H200. Blackwell, starting with B200, comes next.
Do answers get worse when the context gets long?
In our tests the 70B with memfree answered 48 of 48 at one million tokens and at ten million. A standard setup holding the whole text answered 45 of 48 at 128K.
Where does our customers' data go?
Nowhere new. memfree runs on your machines, and your customers' text never leaves them.
How is it delivered?
As a compiled add-on with a thin, open connector to vLLM. You run it on your nodes, and we support it.
What does it cost?
That depends on your fleet and on how you sell long context. Tell us about both in a pilot request and we'll talk numbers.