GPT-OSS-20B benchmark: Comparing response time, tokens/second, and cost efficiency on L4, L40S & H100

More and more companies of all sizes are already using LLMs across many aspects of their operations, but the key concern they raise now is: “Where does our data go?” As a result, many organizations have started considering hosting LLMs locally.

The market already offers high-quality LLM models (OpenAI, DeepSeek, Mistral, Qwen, and many others), but the real questions are: what are the limits, how much throughput can you achieve, and at what cost? In this post, we provide clear answers.

Shadow AI issue

A short quote from the OpenAI Privacy Policy: https://openai.com/policies/row-privacy-policy/

We collect Personal Data that you provide in the input to our Services, including your prompts and other content you upload… We may use Personal Data to provide, maintain, develop, and improve our Services… We may disclose Personal Data if required to do so by law or in response to valid legal requests by public authorities.

So Public ChatGPT (including the web at chatgpt.com or the mobile app) should not be considered a legally privileged or confidential communication system. It is a commercial online service governed by its Terms and Privacy Policy, not a protected legal environment.

This creates a new issue called Shadow AI. Companies take a serious risk when their employees use public AI tools outside approved systems, potentially uploading sensitive data — contracts, source code, financials, customer PII, or strategic plans — without governance or data-processing safeguards. This exposes the company to data leaks, compliance violations, IP uncertainty, and legal discovery risks while weakening security controls and auditability. For founders, the real danger is not just misuse — it’s losing visibility and control over how critical information flows through unapproved third-party AI systems.

Strategy 1: Build your own tooling using LLM APIs

Companies can use APIs (e.g., OpenAI) to build agents or custom chat systems. This model is safer than public SaaS, but not risk-free.

OpenAI states that API data is not used for model training by default and is typically retained for up to 30 days for abuse monitoring.

However, data may be disclosed if required by law, and the 30-day period is not an absolute guarantee against extended retention due to legal or compliance obligations.

Policy changes at arbitrary times are also a common concern.

Strategy 2: Self-hosting with a local provider

For example, a German company hosting its LLM with a German provider under a signed DPA gains clearer legal recourse than relying on a U.S.-based AI provider.

If a dispute arises, litigation within the same jurisdiction is generally more straightforward than cross-border claims.

This does not eliminate disclosure or operational risks — it increases jurisdictional control, not absolute security.

Strategy 3: Self-hosting using on-premise hardware

Running LLMs on fully owned on-premise hardware further reduces third-party exposure. Controlled physical access and isolated infrastructure maximize data locality and remove dependence on external AI or cloud providers.

Absolute zero risk does not exist. On-premise deployment reduces third-party trust assumptions but shifts full responsibility for security, resilience, and compliance to the organization.

Self-hosting - measuring limitations

Self-hosting a model on custom hardware is not technically complex for experienced teams, but the important factor is understanding the limitations, which mainly depend on the hardware. So here we will try to measure them on example of gpt-oss-20b model hosted on different hardware and with different vLLM params.

To understand every term in the future reading I strogly recommend to read my LLM Terminology explained simply post

Max effective sequence length benchmark (input + generated tokens)

On every sequence run, the model takes the whole prompt (including, for example, system messages, history messages, summary, and RAG context) and then generates output from it.

For reasoning models, it first generates some amount of “thoughts” in the output and only then generates the real response.

The sum of the actual input and output tokens is called the effective sequence length.

The actual effective sequence length can’t be larger than context_window. Example context windows:

• Proprietary GPT-5 mini - 400k tokens
• Open gpt-oss-20B - ~131k tokens (128 Ki tokens)
• Open gpt-oss-120B - ~131k tokens (128 Ki tokens)
• Open GPT-J which we self-hosted in 2022 (GPT 3 Davinci analog) - ~2k tokens (2 Ki tokens)

So what we will do in the experiment:

1. In each experiment, we take the first N words from Moby-Dick (Herman Melville, plain text) and ask the model to return a summary in JSON format.
2. We increase the prompt in large steps (e.g., 5k words per point) and measure how fast it generates results and whether the result is correct.
3. We add a secret code at the start of the book to ensure the model is still using the full content.
4. We always add a random number at the beginning to prevent vLLM from using prefix caching (we’re interested in the worst case).
5. At each point we run two checks: whether the response is good and parseable, and whether the secret code is correct. Since availability/parseability issues can generally be retried, we do 3 attempts: if all 3 attempts are OK we mark the point green; if at least one attempt is OK we mark it as “retried” (yellow); if all attempts fail we stop the experiment and mark that point red in the charts (no sense in continuing).
6. If the output is parseable but the secret code is not detected in at least one experiment, we mark it red, because this kind of sense loss can’t be fixed by retries.

In this test we never run parallel requests: we wait until the first one finishes before starting the next (single-user emulation).

We always run all requests with:

medium: { 
  max_tokens: 1536,  // max_tokens
  temperature: 0.15, 
  top_p: 0.9 
},

Here is example JSON which we received:

{
  "status": 200,
  "streamedResponse": {
    "main_theme": "The obsessive quest for the white whale and its destructive consequences",
    "key_characters": [
      "Ishmael",
      "Captain Ahab",
      "Starbuck",
      "Queequeg",
      "Moby Dick"
    ],
    "causal_events": [
      "Ahab's vow to kill Moby Dick",
      "The crew's moral conflict",
      "The final battle with the whale",
      "The ship's destruction"
    ],
    "hidden_implication": "The story critiques the hubris of humanity in confronting nature",
    "secret_code": "314781-21",
    "confidence": 0
}

Structured output mode

We executed all our experiments in Structured output mode. In two words, this is an API-level setting that asks vLLM to respect an output JSON schema at decoding time.

Modern structured output modes (like those in vLLM or OpenAI-compatible APIs) don’t just concatenate “please produce correct JSON” — they actively constrain generation at the token level. Under the hood, the model still produces logits (raw probabilities for every possible next token), but the runtime system masks out any tokens that would break the expected structure (for example, invalid JSON syntax at a given position). Sampling with temperature or top-p then happens only over the allowed tokens. This makes invalid outputs effectively impossible rather than just unlikely.

In production, this is critical: it dramatically reduces malformed responses, eliminates most retry logic, and significantly improves consistency and adherence to the required schema. If you naively prompt a model to “return valid JSON,” it will work most of the time — but occasionally (say 1 in 100 responses) you’ll still get broken JSON and need to retry. Structured decoding avoids that class of failures, which is why it’s essential for reliable, production-grade systems.

And vLLM supports it!

How we started vLLM

We used pure docker container delivered from terraform to server with GPU:

docker run --rm \
    --name gpt-20b \
    --gpus all \
    --ipc=host \
    -e HUGGING_FACE_HUB_TOKEN=${var.hf_token} \
    -p 8000:8000 \
    vllm/vllm-openai:latest \
    --model ${var.model_id} \
    --dtype bfloat16 \
    --kv-cache-dtype fp8

At the time of the research, the latest version of vLLM is 0.15.1.

Besides the explicit parameters, there are many implicit ones under the hood. Several important parameters that may be worth tuning are below. I intentionally left them at default values, since vLLM defaults are a good starting point for many tasks.

gpt-oss-20B on L4 24G

Typical rental price for server with L4 1x24G assuming mid-range stable on-demand: €520–€1175 / mo higher ones closer to Enterprise-grade hosters.

If you want to have it on premises

Here is our benchmark result:

Time to first generation grows non-linearly and reaches ~120 seconds for a ~130k-token prompt. Decoding speed falls from ~120 to ~45 tokens/second.

And on this setup we can run sequences with ~55k input tokens (ESL up to ~55k + 1.5k = ~56.5k).

VRAM at this experiment is stable and reaches 21.725Gi/22.494G.

The two decoding-speed drops are most likely caused by internal attention-kernel switching inside the inference engine. As the input sequence length crosses certain thresholds (around ~30k and ~50k tokens), the runtime switches to different CUDA/FlashAttention kernel variants or GEMM configurations optimized for larger context sizes. These transitions can temporarily reduce SM occupancy or change memory access patterns, resulting in a brief throughput dip before stabilizing again. Since this happens with a single request on an otherwise idle GPU and the performance trend recovers immediately afterward, it strongly suggests a kernel-boundary effect rather than memory pressure or resource contention.

Does the model think more with more content?

In the same experiment we also fetched information about reasoning-token volumes (all experiments were executed with fixed reasoning_afford=medium):

Conclusion: thinking tokens do not depend on input size; on average the model “thinks” for about the same time (with some variance) for both 5k input words and 40k input tokens.

gpt-oss-20B on L40S 48G

Another very popular video card on the market is the L40S. Average rental price is around €1050-€1400; a new one on eBay might cost $9249 USD.

VRAM consumption is 42.112Gi/44.988Gi and stayed stable during the full experiment session.

gpt-oss-20B on H100-1-80G

Across the whole test run we saw VRAM up to 74.208Gi/79.647Gi, which means that with one sequence at a time the VRAM was not fully occupied.

At the 60k point we saw the first failure, which was then successfully retried from one attempt. The issue was next, model started to repeat:

"{"main_theme":"The secret code is 314781-21","key_characters":["Ishmael","Queequeg","Captain Ahab","Captain Peleg","Captain Bildad"],\
"causal_events":[ "Ishmael leaves New Bedford","Queequeg joins him","They sign aboard the Pequod","Captain Ahab is introduced","The Pequod sets sail","The crew faces the first whale encounter","The crew confronts the great whale","The crew is pursued by the whale","The crew is rescued","The crew returns to Nantucket","The crew faces the final whale","The crew is rescued again","The crew returns home","The crew faces the final whale again","The crew is rescued again","The crew returns home","The crew faces the final whale again","The crew is rescued again","The crew returns home","The crew faces the final whale again"... and so on up to max_tokens

See details about this issue at the end of the post.

Managed gpt-oss-20b hosting

Some hosting services nowadays provide inference-as-a-service. For example, they may host gpt-oss-20b for you, so you pay for used tokens instead of paying for a whole server running 24/7. This will be still cheaper as many official vendor LLMs, but again risk of data exposure is high if you don't sign dedicated DPAs.

Generally they don’t disclose what hardware they use under the hood or what vLLM parameters they run.

What we can see from this chart:

• For short sequences the best decoding throughput was better than on L4 (~80 token/s), but worsen then on L40S & H100 with our setup
• Time to first token is nearly same to L40S (e.g., ~5s at ~40k input tokens), but our H100 setup is better.

Ability to handle parallelism benchmark (multiple users wait time)

Another test: we take a fixed point, e.g. 10k input words (pretty typical for an average RAG/agent task), and simulate requests from parallel users. This is like several users using the model at the same time, and we observe how average per-user throughput and response time react. In the previous example we measured decoding throughput (the speed at which tokens are generated after the first token is received). In this test it’s more interesting to look at min/max TTFT and min response time (for the most “lucky” user) versus max response time (for the most “unlucky” user).

gpt-oss-20B on L4 24G

On model load we saw the following in vLLM logs:

Your GPU does not have native support for FP4 computation but FP4 quantization is being used. Weight-only FP4 compression will be used leveraging the Marlin kernel. This may degrade performance for compute-heavy workloads.

Maximum concurrency for 131,072 tokens per request: 2.78x

Let’s run a benchmark:

With 8 parallel users, we still see that all 8 users get the final response at roughly the same time (the spread between max and min response time is ~3 seconds: the most “lucky” user gets a response in ~33 seconds and the most “unlucky” user in ~36 seconds). But the spread between the shortest and longest TTFT still grows. If the LLM is used in a chat-like interface with 8 users asking at the same time and prompts are around 10k words (e.g., include RAG and history), the chart implies that the most “lucky” user receives the first token after ~3 seconds and starts streaming, while the most “unlucky” user waits ~25 seconds for the first token.

You might ask: how, in this 8-user case, do they still receive the full response at almost the same time with such a significant TTFT spread? The large TTFT spread happens because TTFT depends on when each request gets its turn for prefill on the GPU — if a user is later in the queue, their first token simply starts later. However, once decoding begins, vLLM time-slices the GPU fairly across all active requests, so they generate tokens at roughly the same pace. The key insight is that the “unlucky” user doesn’t fall behind by the full TTFT gap, because while they were waiting, the others were not running at full speed either — they were also sharing the same GPU and spending time on prefill and other users’ work. That’s why TTFT spread can be large, but total response-time spread stays relatively small.

To illustrate the difference, let’s set --max-num-seqs to 1, which effectively disables parallelism:

Here we see huge spreads even in total response time: with 8 users, the most “lucky” user receives the full response within ~10s, and the most “unlucky” within ~60s.

Future way to tune the model

If you wish to reduce TTFT for all requests, you can experiment by raising --max-num-batched-tokens, e.g. to 4k or 8k. By doing this, the vLLM scheduler will handle larger prefill chunks, which reduces the total number of forward passes and can reduce overall TTFT. The drawback is that you increase the risk of VRAM OOM. Generally, VRAM OOM is less painful than RAM OOM, and Docker will gracefully auto-restart vLLM (at least it worked on my machine), but any OOM still causes downtime for a minute or so, which is definitely not good.

If you are going to change any parameters like --max-num-batched-tokens from default values, you should stress-test for OOM at the highest context and high concurrency. If you hit OOM after increasing MNBT (which is very probable), you’ll need to sacrifice something, e.g. decrease --max-model-len to 57344 tokens (56 Ki tokens = 56 × 1024).

gpt-oss-20B on L40S 48G

Here in launch log we can see that vLLM says we can run up to 16 requests on 131k sequences:

Your GPU does not have native support for FP4 computation but FP4 quantization is being used. Weight-only FP4 compression will be used leveraging the Marlin kernel. This may degrade performance for compute-heavy workloads.
Maximum concurrency for 131,072 tokens per request: 15.97x

So it is still the “Marlin” kernel, but the promised concurrency at 131k is ~16x; let’s see what it means in reality:

Here is result on our previous setup:

gpt-oss-20B on H100 80G

Using Triton backend
Maximum concurrency for 131,072 tokens per request: 31.52x

This is a first model using Triton with native fp4 support.

How managed LLMs handle parallelism?

I tried to run the same test on one of these services.

We see that 3rd-party hosting keeps delay at a good level and it does not significantly degrade as the number of users increases. When you are using LLM-as-a-Service endpoints, you can generally run as many sequences (API requests) as you need in parallel and can expect them to generate in parallel. This is possible because the external provider has a horizontally scalable (from your perspective, effectively infinite) pool of GPUs and can split API requests from many clients. Obviously they split the payload across multiple inference endpoints to achieve maximum efficiency.

Structured output failure due to degenerate repetition

Since we had one retried case on H100, after running the main experiments, we stress-tested the model by running at peak points for half an hour.

And very rarely we see retries caused by broken JSON:

{
 "main_theme": "Moby-Dick",
 "key_characters":[ "Ishmael","Queequeg","Captain Ahab","Captain Peleg","Captain Bildad","Father Mapple"],
  "causal_events":[ "Ishmael leaves New Bedford for Nantucket","Ishmael meets Queequeg at the Spouter‑Inn","Ishmael and Queequeg sign aboard the Pequod","Captain Ahab’s leg is lost to a whale","The Pequod sets sail","The crew encounters the great white whale","The crew is pursued by the whale","The crew is rescued by the crew of the ship “The Rachel”","The crew is pursued by the whale again","The crew is rescued by the crew of the ship “The Delight”","The crew is pursued by the whale again"....

This issue was always successfully retried on the next attempt.

Its frequency is quite low: we saw from 1 to 5 occurrences during 30 minutes, but this frequency may depend, for example, on the entropy and lengths of incoming prompts.

This issue happened at any point of any experiment, and this may be a common picture for the GPT-20B model.

Moreover, this issue happens with any LLMs. Gemini, for example, quite often goes into this repetition even in the public UI, while OpenAI might hide it via retries (it’s quite easy to detect repetitions).

Attempts to run gpt-oss-120B on H100 80G

Tn theory 80G might be enough to run even a 120B model. It's weights in MXFP4 consume at least 60GB by definition (in fact I’ve seen 64.38 GiB in logs - some CUDA overhead), so what’s left for KV cache and activations is much less than 14G, which is very tight assuming number of layers for this model is higher.

Default parameters for this model on Hugging Face / vLLM are definitely not optimized for 80G VRAM:

| MML (max-model-len) | 131072 | | MNBT (max-num-batched-tokens) | 8192 | | MMU (gpu-memory-utilization) | 0.9 | | MNS (max-num-seqs) | 256 |

With these settings the model does not load at all:

Model loading took 64.38 GiB memory  
Available KV cache memory: -0.62 GiB

Setting MNBT to 2048 still did not help. So with the same config as we used for 20B on 24G VRAM, we can’t run 120B on 80G VRAM.

The reason 120B cannot start on a single H100 80GB with the same 131k context settings that worked for 20B is that KV cache memory scales with the number of layers and the hidden size of the model, not just with available VRAM. While 20B left ~10GB for KV cache on a 24GB GPU, the 120B model has roughly 4–5× larger per-token KV footprint due to significantly more layers and a much larger hidden dimension. So even though ~16GB appears “available” after loading weights on H100, each token now consumes far more KV memory, making a 131k context mathematically infeasible on a single GPU.

Reducing --max-model-len to half (e.g. 57344) didn’t help either. You can probably extract something from this card by reducing values even more, but it will be very limited and needs careful OOM testing. Ideally you should use 2× H100 80G and set --tensor-parallel-size 2.

Conclusion

At a typical “real” chat/RAG workload size (≈10k input tokens/words: history + retrieved context + a short user question like “Given the attached policy and our last 5 messages, what are the top 3 risks and the next steps?”), the single-user latency for gpt-oss-20b differs dramatically by GPU. In our 1-user run at that prompt size, L4 24G delivered ~3.50s TTFT and ~6.93s full response time, L40S 48G ~1.17s TTFT and ~1.92s full response time, and H100 80G ~0.80s TTFT and ~1.62s full response time.

If your system is used frequently by many people (for example, an internal chat tool where several employees ask questions at the same time), the same ~10k context request can be in-flight concurrently. With 5 parallel users at this prompt size, we observed that L4 24G spreads TTFT from ~4.61s (lucky) to ~16.38s (unlucky) and full response time from ~20.99s to ~24.04s; L40S 48G keeps TTFT in ~1.42–4.55s and full response in ~6.78–7.65s; and H100 80G keeps TTFT in ~1.29–2.37s and full response in ~3.48–4.51s. In practice this matters because user-perceived latency is dominated by TTFT in streaming UIs, and concurrency is the normal case for shared tools.

From a cost-efficiency standpoint, L4 is the cheapest entry point (and works well for low concurrency), but it becomes latency-limited quickly as context grows. L40S is often the sweet spot for gpt-oss-20b: it cuts single-user latency by ~3–4× versus L4 while typically costing less than 2×, and it also provides substantially more parallelism headroom at long contexts. H100 wins on raw latency and concurrency, but unless you specifically need the lowest possible tail latencies at higher load, it is usually harder to justify purely on €/latency for a 20B-class model.

Usefull links

• Optimization and Tuning of vLLM clearly describes main tuning ideas.
• A must LLM Terminology to understand this post explains all the terms used in this post.
• gpt-oss-20b on Hugging Face