An add-on for vLLM. Nothing in vLLM changes.
memfree sits between your customers and vLLM on your own GPU nodes. It keeps long conversations small in GPU memory, so the cards you already run serve far longer context to far more people.
Where it sits
Install it beside vLLM
We hand you a compiled add-on for your GPU nodes. Your engineers keep control of the deployment.
Route long conversations through it
Your endpoint stays the same. memfree takes the long part of each conversation and keeps its memory flat.
Sell longer context on the same cards
Offer a million tokens and more without buying memory for it, and fit more paying conversations on every card.
What you get
A small piece of software on your nodes that changes what a long conversation costs you.
- A compiled add-on
- Built for your GPUs and delivered ready to run. You install it next to vLLM.
- A thin, open connector
- The small piece that talks to vLLM is open, so your engineers can read exactly what passes between the two.
- Flat memory per conversation
- About 4 GB with a 70B model, from 128 thousand tokens to 10 million.
- Right answers far past 128K
- 48 of 48 at 1M and at 10M tokens in our tests on one H200.
- Conversations that survive restarts
- A long conversation reloads from disk and carries on. Ten million tokens came back and answered right in under 17 seconds.
What stays the same
memfree adds to your stack. It does not ask you to replace any of it.
- vLLM
- Its scheduler, its own cache and your settings run as they do today.
- Your models
- The open models you already serve.
- Your API
- Customers keep calling the same OpenAI-compatible endpoint.
- Your data
- Your customers' text stays on your machines.
- Your pricing
- You decide how to sell long context to your customers.
What it runs on
Data-center NVIDIA GPUs, starting with Hopper.
- GPUs
- NVIDIA H100 and H200 now. Blackwell, starting with B200, next.
- Servers
- Linux on x86-64, with NVIDIA's CUDA drivers.
- Serving
- vLLM, as you run it today.
- Models
- Open models served by vLLM. Our published numbers use Llama 3.3 70B Instruct.
What it changes for a GPU cloud
More conversations per card
At 128K tokens, one H200 held 26 memfree conversations against 2 standard ones in our tests.
Long context as a product
Offer a million tokens and beyond on the cards you have, instead of turning those customers away.
No new hardware
The gain comes from software on your existing nodes, not from buying more memory.
Try it on your own nodes
A pilot installs memfree on one of your GPU nodes and measures it against your own workloads.