# LLM throughput benchmark for Windows (version 1.0)

This benchmark measures how fast and how correctly a language model answers on **your own PC** when several requests
arrive at the same time. It is meant for infernix with Qwen3.8-Flash-Next or Qwen3.8-27B, but it measures any server
with an OpenAI-compatible API (llama.cpp, for example). Everyone who uses this package sends exactly the same 16
requests on the same schedule, so results can be compared in the forum.

**Duration:** about 30 minutes (short, default) or 3 hours (full). **What it does not do:** it opens no port, sends
nothing to the internet and downloads nothing. It only talks to the server you enter, and only if that server runs on
this PC (127.0.0.1) or in your Tailscale network.

> **Untested on Windows:** the package was tested on macOS and Linux (also with simulated Windows behaviour), but
> never on a real Windows PC. Please report problems in the forum with the text from the window (remove your key
> first, see below).

This guide comes twice: as `.md` and as `.txt` (the `.txt` opens in Notepad with a double-click). The program's own
messages are in German; the tables below translate the questions and messages.

## What you need

- Windows 10 or 11, an NVIDIA GPU with a current driver (for infernix: RTX 50xx, driver 580 or newer).
- infernix (Windows release) and a model.
- Python 3.11 or newer (step 3).
- About 1 GB of free disk space for the results.

Type the commands below into **PowerShell**: open the Start menu, type `PowerShell`, press Enter. Copy one line,
paste it into the PowerShell window with **Ctrl+V**, press Enter.

Windows note: Explorer often hides file extensions; the files then show as `benchmark-starten` and `infernix-starten`
(without `.cmd`). On Windows 11, "Edit" in the right-click menu is under **Show more options**.

## Step 1: verify the download

The forum post lists a checksum (SHA256) next to the download. This shows the checksum of your file (adjust the path
if it is not in your Downloads folder):

    (Get-FileHash -Algorithm SHA256 "$env:USERPROFILE\Downloads\llm-durchsatz-benchmark-1.0.zip").Hash.ToLower()

The output must match the one in the forum **exactly**. If it does not: delete the file, do not unpack it (someone
may have changed it).

Then tell Windows the file is fine, so it does not ask again for every file later:

    Unblock-File "$env:USERPROFILE\Downloads\llm-durchsatz-benchmark-1.0.zip"

## Step 2: unpack

Right-click the ZIP file, **Extract All**, enter `C:\benchmark` as the target. The files are then in
`C:\benchmark\llm-durchsatz-benchmark-1.0\`, the folder that contains `benchmark-starten.cmd`. Do not
double-click inside the ZIP window; that does not work.

## Step 3: install Python (once)

    winget install -e --id Python.Python.3.13 --source winget

Windows asks whether the installer may make changes: confirm with **Yes**. Then **close and reopen** PowerShell and
check:

    py -3 --version

It must show `Python 3.11` or higher.

Without winget ("winget is not recognized"): download the **Windows 64-bit** installer from
https://www.python.org/downloads/, run it, keep **py launcher** and **Add python.exe to PATH** ticked, install.

## Step 4: create a key (recommended)

A key stops other programs, or web pages in your browser, from using infernix. Create one (48 characters, digits and
a to f):

    $b = New-Object byte[] 24; [Security.Cryptography.RandomNumberGenerator]::Create().GetBytes($b); -join ($b | ForEach-Object { $_.ToString('x2') })

Select the output with the mouse and copy it with **Ctrl+C**; you need it in steps 5 and 6.

**Important:** never post the key in the forum. It is stored in `infernix-starten.cmd` on your PC and, while infernix
runs, visible in the process list (infernix only accepts it that way). Before posting text or screenshots from the
window, remove the key; if in doubt, create a new one.

## Step 5: start infernix

1. Right-click `infernix-starten.cmd`, **Edit** (opens Notepad).
2. Between `HIER ANPASSEN / ADJUST HERE` and `ENDE / END` set the values. **Use paths without umlauts, special
   characters or exclamation marks**, for example `D:\models` (otherwise Windows does not find the files):
   - `INFERNIX`: path to your `infernix-serve.exe`.
   - `MODELLART`: `flash-next` or `27b`.
   - `MODELL` and (Flash Next only) `NGRAM`: paths to the model files.
   - `EXPERTEN_RAM_MIB` (Flash Next only): how much host RAM the experts get.
     - `auto` (default): infernix takes what is free at start. With about **73 GB of free RAM** all experts are in
       RAM (fastest).
     - A number in MiB if you want to give less. This command shows free RAM in GB:

           [math]::Round((Get-CimInstance Win32_OperatingSystem).FreePhysicalMemory/1MB,1)

       Rule of thumb: (free RAM in GB − 8) × 1024, for example 60 GB free: `53248`.
   - `SCHLUESSEL`: the key from step 4 (or leave it empty).
3. Save, close Notepad, **double-click** `infernix-starten.cmd`. Wait until the window shows
   `listening on http://127.0.0.1:8080` (a few minutes for Flash Next). Keep the window open.

**Do not click into the infernix window.** In the classic Windows console a click pauses the output and can stall
the server. If it hangs anyway: click into the window and press **Enter**.

Please **do not change** the other settings (2 slots, context 131072, queue 16), or your result cannot be compared.

## Step 6: run the benchmark

**Double-click** `benchmark-starten.cmd`. The assistant asks one question after another; Enter accepts the suggestion:

| Question (German) | Meaning | Answer |
|---|---|---|
| Adresse des Servers | server address | Enter (`http://127.0.0.1:8080`) |
| Schlüssel | key | paste the key from step 4 with Ctrl+V, Enter; it is not shown |
| Modell | model | Enter (the name comes from the server) |
| Server und Version | server and version | for example `infernix windows-2026.10.11` |
| Arbeitsspeicher des PCs in GB | RAM of the PC in GB | for example `96` |
| Experten-RAM | expert RAM | the same value as `EXPERTEN_RAM_MIB` in step 5 |
| Kontext, Plätze | context, slots | Enter |
| Umfang | scope | Enter (`kurz` = short); `voll` (full) for slow PCs, see below |

**kurz or voll?** `kurz` (short) measures 8, 16, 32 and 64 requests per minute (levels 4 to 7) and is enough for PCs
that write an answer within a few seconds. If your PC needs a minute or more per answer (for example because parts
of the model come from the SSD), choose `voll` (full): the benchmark then also measures 1, 2 and 4 requests per
minute (levels 1 to 3). If the report after `kurz` says "unter Stufe 4" (below level 4), only a run with `voll` shows
what the PC can handle.

It then measures two runs: **thinking off** (`Denken aus`) and **thinking on** (`Denken an`). Meanwhile:

- Do not run anything else on the GPU (no games, no second model).
- The benchmark asks Windows not to go to sleep. On a laptop, plug in the charger and keep the lid open.
- Do not measure across the daylight saving change (night to 25 October 2026, 3 am): the power log cannot handle the
  time jump.
- **Ctrl+C** stops (this can take up to 15 seconds). If Windows then asks "Terminate batch job (Y/N)?" (German:
  "Batchvorgang abbrechen (J/N)?"), answer **N** so the window with the messages stays open.

## Step 7: look at and share the result

At the end the window lists the report paths. Open the folder:

    explorer "$env:USERPROFILE\llm-durchsatz\laeufe"

Each run has a folder with `lauf.md` (the report, one table row per load level), `ergebnis.json` (the same numbers
for programs) and `messpunkte.jsonl` (every single request). For the forum, **`lauf.md`** of both runs is enough
(right-click, Open with, Notepad).

If the window says "Für diesen Lauf gibt es noch keinen Bericht" (no report for this run yet), the next line is a
command that creates the report later: copy it into PowerShell and run it.

**Reading the report:** each row is a load level (requests per minute). The key columns are **Richtig** (share of
correct answers), **Richtige/min** (correct answers per minute), **Tokens/s gesamt** (all answers together, over the
whole level including idle time), **Tokens/s je Anfrage** (how fast a single answer is written), **Erstes Token**
(time to first token) and, below the table, **Kapazität** (capacity: the highest level the PC still handles well).
**Kennzeichen** (flags) such as `ueberlastet` (overloaded) explain why a number does not stand on its own. There is
deliberately no overall score.

Two flags appear in every run and are not errors: `unter_empfehlung` (below recommendation: the benchmark limits the
answer length below the model card's recommendation on purpose, the same for everyone) and, for Flash Next only,
`profil_widerspruch` (profile contradiction: sources on Flash Next contradict each other in two places; this is noted
in the model profile).

## What the files contain (read before sharing)

- `lauf.md` and `ergebnis.json`: numbers, model name, your description of the PC (GPU, RAM, server, expert RAM).
- `messpunkte.jsonl`: also the model's answers to the 16 made-up requests. No personal data.
- The key is in no file of the benchmark (it checks this and redacts it).
- The power log is in `%USERPROFILE%\llm-durchsatz\messlog\` (GPU power, clocks, temperature per second).

## If something goes wrong

| Message (German) | Meaning | What to do |
|---|---|---|
| `Bitte zuerst das ZIP entpacken` | extract the ZIP first | step 2: extract, then double-click in the extracted folder |
| `Python wurde nicht gefunden`, `Python 3 fehlt` or `Bitte Python 3.11 oder neuer installieren` | Python missing or too old | repeat step 3, reopen PowerShell |
| `... nicht gefunden` when starting infernix | file not found | check the path in `infernix-starten.cmd`; umlauts or special characters? |
| `Schluessel: nur Buchstaben ... und Ziffern` | key: letters and digits only | use the key from step 4 (digits and a to f only) |
| `MODELLART: flash-next oder 27b` | wrong model type | enter exactly `flash-next` or `27b` in `infernix-starten.cmd` |
| `EXPERTEN_RAM_MIB: auto oder eine Zahl` | wrong expert RAM | `auto` or a number without dots or spaces, at most 7 digits |
| `Adresse ...` / `Erlaubt sind dieser PC` | address not allowed | the server must run on this PC (`127.0.0.1`) or be a Tailscale name |
| `Schlüssel abgelehnt` / HTTP 401 | key rejected | same key as in `infernix-starten.cmd`? |
| `Modellliste nicht lesbar (nicht erreichbar)` | server not reachable | is infernix running and showing `listening on`? |
| infernix reports port 8080 in use | port busy | another program uses the port; close it or restart the PC |
| `nvidia-smi liefert keine Karte` | no NVIDIA GPU found | install or update the NVIDIA driver; otherwise `j` for a run without power log |
| `Stufe ... ist überlastet` | level overloaded | not an error: the PC cannot keep up from this load on; the ladder stops there |
| `Anfragenpaket oder Fahrplan weichen ab` | package changed | download again and check step 1 |
| Windows warns on double-click (SmartScreen) | file from the internet | step 1 skipped? Otherwise "More info", "Run anyway" |
| `Unerwarteter Fehler (...)` | unexpected error | report this line in the forum (it contains no key) |

## What is not checked

- Windows: see the note above. Only NVIDIA GPUs are measured (not AMD or Intel).
- `Richtig` checks only the form of the answer (number, short text, JSON), not the quality of long texts.
- Each level has only 16 requests: small differences between two PCs are not reliable.
- infernix itself is not part of this package and is not checked here.

License: Apache-2.0 (file `LICENSE`), without warranty.
