How to Benchmark Your Local LLM Playground

So, you’ve hoarded a half-dozen local models across different machines on your network. Your main workstation is sweating, your flashy new mini-PC is screaming, and your gadget-buying habits have officially transitioned from buying physical lenses to hoarding VRAM. Congratulations.

But now you have a real problem: you have an army of LLMs, and you have absolutely no idea which one is actually intelligent and which one is just confidently stupid. You're routing requests blindly and hoping the 7B parameter model isn't hallucinating its way through your code refactoring.

You want a magical, one-click Web UI that auto-discovers your models and gives you a neat little chart. Spoiler alert: it doesn't exist. The open-source community still treats benchmarking like a developer hazing ritual.

So, we are doing this the right way—in the CLI, with Docker, and a proper proxy architecture. Here is how to build a unified LLM benchmarking stack that actually tells you the truth about your hardware and models.

Step 1: The Traffic Cop (LiteLLM)

Before we benchmark anything, we need to talk about infrastructure. Pointing a benchmarking tool at three different machines running three different APIs (Ollama, vLLM, llama.cpp) is a nightmare.

Enter LiteLLM.

LiteLLM is a proxy server. It sits in front of all your random local AI servers and translates everything into one perfect, unified OpenAI-compatible API.

  • Why it helps: You point your apps (and our benchmark tool) at one IP address. LiteLLM handles the routing.
  • The hidden superpower: It exports Prometheus metrics. It tracks tokens-per-second, latency, and exact request counts.

The LiteLLM Docker Setup: Spinning it up is trivial. Drop this in a docker-compose.yml:

version: '3.8'
services:
  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    container_name: litellm-proxy
    ports:
      - "4000:4000"
    volumes:
      - ./litellm_data:/app/data
    command: --config /app/data/config.yaml

Once it's up, you open http://localhost:4000, add your backend server IPs, and it spits out a single unified API key (e.g., sk-my-local-key). Save that key. We are going to hammer it.

Step 2: The Judge (EleutherAI Evaluation Harness)

Now that our endpoints are unified, we need the testing engine. We are using the EleutherAI LM Evaluation Harness. If you've ever looked at the Hugging Face Open LLM Leaderboard, this is the exact engine that generates those scores.

It has hundreds of academic tests baked in. You don't write prompts. You don't write assertions. You just tell it to run, and it interrogates your model with thousands of questions.

Because we want this clean and portable, we are going to build a micro-container for it. Create a new folder on your server, and drop this Dockerfile inside:

FROM python:3.11-slim
WORKDIR /app
RUN apt-get update && apt-get install -y git gcc && rm -rf /var/lib/apt/lists/*
RUN pip install --no-cache-dir "lm-eval[api]"

Next to it, create a docker-compose.yml. Notice the network_mode: "host"—this is critical so the container can cleanly hit your LiteLLM proxy running on the same machine without Docker networking headaches.

services:
  lm-eval:
    build: .
    container_name: lm-eval-runner
    network_mode: "host" 
    volumes:
      - ./results:/results
    environment:
      # The key you got from LiteLLM
      - OPENAI_API_KEY=sk-my-local-key
    entrypoint:
      - /bin/bash
      - -c
      - 'python -m lm_eval --model local-chat-completions --model_args base_url=http://localhost:4000/v1/chat/completions,model="$$0",num_concurrent=10 --apply_chat_template --tasks gsm8k,mmlu --batch_size auto --output_path /results'

Step 3: Running the Gauntlet

To benchmark a model, open your terminal, build the container once (docker compose build), and then pass the model's LiteLLM name as an argument.

Let's test that shiny new Qwen model: docker compose run --rm lm-eval qwen2.5-7b-instruct

Sit back and watch the progress bar. When it finishes, it dumps a massive JSON file and prints a clean Markdown table directly into your terminal showing the exact percentage accuracy.

What are we actually testing?

In the compose file above, I set the --tasks to gsm8k and mmlu.

GSM8K (Grade School Math): This doesn't test if the AI knows quantum physics. It tests sequential, multi-step logic. It feeds the model a word problem, forces it to think step-by-step, and strictly grades the final number. If the model hallucinates 7 x 8 = 45 in step four, it fails. Zero partial credit. It’s brutally deterministic.

MMLU: The massive 57-subject multiple-choice test. It tests raw knowledge breadth across law, anatomy, astronomy, and more.

A Pro-Tip for the Hardware Pushers

When you run these tests, the Harness sends massive "few-shot" prompts (huge walls of text showing the model how to answer before asking the real question).

If your backend llama.cpp server suddenly crashes or stalls out, it means your server is panicking while trying to fit that massive prompt into its active KV cache.

Don't neuter the test. Fix your server. Give it a massive RAM parking lot by appending --swa-checkpoints 0 --cache-ram 16384 to your llama-server launch command. This forces the backend to offload that fat context window into your system RAM instead of choking your limited GPU VRAM. (asuming you are not really poor)

Happy benchmarking. May your TTFT be low and your exact-match accuracy be high.