Run 7B model locally — Part 2

Let’s do it. We’ll grab Q8_0, sanity-check at 4k, then (carefully) try 16k.

1) Download Q8

cd ~/ougway_env/llama.cpp
mkdir -p models/qwen2.5-7b-instruct

hf download bartowski/Qwen2.5-7B-Instruct-GGUF \
  --repo-type model \
  --include "Qwen2.5-7B-Instruct-Q8_0.gguf" \
  --local-dir models/qwen2.5-7b-instruct

Expect ~12–13 GB on disk.

2) 4k sanity run (should fit easily)

./build/bin/llama-cli \
  -m models/qwen2.5-7b-instruct/Qwen2.5-7B-Instruct-Q8_0.gguf \
  -ngl 999 -c 4096 -b 128 -t $(nproc)

Tip: watch -n 1 nvidia-smi in another tab. If VRAM is comfy, you can try -b 256.

3) One-shot benchmark (prints tokens/sec)

./build/bin/llama-cli \
  -m models/qwen2.5-7b-instruct/Qwen2.5-7B-Instruct-Q8_0.gguf \
  -ngl 999 -c 4096 -b 128 -t $(nproc) \
  -n 192 --no-interactive \
  -p "In ≤60 words, explain how you reason and list 5 limitations."

Look for the eval time … ( … tokens per second) line.

4) Try 16k (go slow & safe on a 12 GB card)

Start conservative; increase only if stable.

# Step A: 16k with small batch
./build/bin/llama-cli -m models/qwen2.5-7b-instruct/Qwen2.5-7B-Instruct-Q8_0.gguf \
  -ngl 999 -c 16384 -b 64 -t $(nproc)

# If VRAM is tight or OOM:
# 1) drop batch further
./build/bin/llama-cli -m ...Q8_0.gguf -ngl 999 -c 16384 -b 32 -t $(nproc)

# 2) keep KV cache in system RAM (slower but big VRAM saver)
./build/bin/llama-cli -m ...Q8_0.gguf -ngl 999 -c 16384 -b 32 --no-kv-offload -t $(nproc)

# 3) as a last resort, offload fewer layers
./build/bin/llama-cli -m ...Q8_0.gguf -ngl 64 -c 16384 -b 32 --no-kv-offload -t $(nproc)

What to expect

• Quality: Q8 is closest to full precision; you’ll see slightly cleaner logic/wording vs Q5.
• Speed: slower than Q5 (often ~25–40% hit). If Q5 was ~53 tok/s for you, Q8 might land ~30–40 tok/s at similar settings.
• VRAM: 4k should be fine; 16k is doable only with a small batch and/or KV offload on a 12 GB GPU.

If you post back your Q8 tokens/sec at 4k and whether 16k fits (plus peak VRAM), I’ll suggest a “Q8 stretch” preset and confirm whether it’s worth keeping Q8 around or sticking with Q5 as your daily driver.