Results from our GPU day
On 7 November 2026 we ran memfree on one NVIDIA H200 with Llama 3.3 70B Instruct. At each length we asked the same 48 questions, each with one right answer sitting in one place in the text, and counted only exact answers.
Right answers
Each dot is one question. Filled dots were answered exactly right.
128K tokens
48 of 48
Standard serving: 45 of 48
1M tokens
48 of 48
Standard serving
Does not fit on one card: its cache alone would need 344 GB.
10M tokens
48 of 48
Standard serving
Does not fit on one card: its cache alone would need 3.4 TB.
GPU memory per conversation
Standard serving grows with every token. memfree stays at about 4 GB. The scale goes up by ten at each line.
Show as a table
| Length | Standard serving | memfree |
|---|---|---|
| 128K tokens | 44 GB (measured) | 4.1 GB (measured) |
| 1M tokens | 344 GB (worked out) | 4.6 GB (measured) |
| 10M tokens | 3.4 TB (worked out) | 4.1 GB (measured) |
Conversations per card
With the model loaded, the rest of one H200 is room for conversations. At 128K tokens each, standard serving fits 2 and memfree fits 26.
One H200's memory, filled with 128K-token conversations
Conversation blocks drawn to scale from the memory measured on 7 November.
Time to make a text answerable
How long the card works before a long text can be asked about, and what happens after a restart.
of card time for memfree to make a text answerable, at 128K, 1M and 10M tokens alike.
of card time for standard serving to read 128K tokens.
for memfree to take in ten million tokens, start to finish.
for a ten-million-token conversation to reload from disk after a restart and answer right.
How we measured
- Card
- One NVIDIA H200 SXM, rented in the cloud.
- Model
- Llama 3.3 70B Instruct, 4-bit (Q4_K_M), the same file for both setups.
- Questions
- 48 at each length, each with one exact answer sitting in one place in the text. An answer counts only if it gives that exact figure.
- Standard serving
- The same model file served by stock llama.cpp, with the whole 128K text held in GPU memory.
- Beyond 128K
- No single card can hold a standard conversation at 1M or 10M tokens, so those figures are the model's own cache, worked out: about 328 KB a token at 16-bit.
- Memory
- Read from the card itself, sampled through every run. GB means a billion bytes.
- Checking
- Every answer was counted again from the test files by a second process, and every figure matched.
See these numbers on your own cards
A pilot measures the same things on your GPUs, with your models and your workloads.