Database Formalization Progress — Part 2

The CPU result settles the main question:

• Two threads: 54 ms median for a live query
• Four threads: 38 ms median
• Full 1,500-character chunk: 212 ms with four threads
• Cold model load: 6.4 seconds

That is comfortably fast enough to keep live Nomic embedding entirely on the CPU. We do not need concurrent Nomic/Qwen GPU inference or a GPU coordinator in v1.

My starting policy would be:

• Keep one CPU Nomic worker resident in RAM.
• Give it two threads for live queries; 54 ms is already negligible.
• Permit four threads for deferred conversation write-back.
• Reserve the GPU for Qwen.
• Use the GPU for bulk embedding only during explicit ingestion sessions.

Now we measure Qwen at 8K, 16K, and 32K contexts. This is read-only and will unload Qwen after each measurement. It may take several minutes.

LOG="$HOME/Desktop/qwen-vram-benchmark-$(date +%Y%m%d-%H%M%S).txt"

python3 - <<'PY' 2>&1 | tee "$LOG"
import json
import subprocess
import threading
import time
import urllib.request
import urllib.error

MODEL = "qwen3:8b"
CONTEXTS = (8192, 16384, 32768)
OLLAMA_URL = "http://127.0.0.1:11434/api/generate"

def gpu_used_mib():
    result = subprocess.check_output(
        [
            "nvidia-smi",
            "--query-gpu=memory.used",
            "--format=csv,noheader,nounits",
        ],
        text=True,
    )
    return int(result.strip().splitlines()[0])

def ollama_request(payload, timeout=1800):
    request = urllib.request.Request(
        OLLAMA_URL,
        data=json.dumps(payload).encode("utf-8"),
        headers={"Content-Type": "application/json"},
    )
    with urllib.request.urlopen(request, timeout=timeout) as response:
        return json.loads(response.read().decode("utf-8"))

def unload():
    try:
        ollama_request(
            {
                "model": MODEL,
                "prompt": "",
                "stream": False,
                "keep_alive": 0,
            },
            timeout=120,
        )
    except Exception:
        pass
    time.sleep(2)

def ollama_ps():
    return subprocess.run(
        ["ollama", "ps"],
        text=True,
        capture_output=True,
    ).stdout.strip()

print("Model information:")
subprocess.run(["ollama", "show", MODEL], text=True)
print()

for context in CONTEXTS:
    unload()
    baseline = gpu_used_mib()

    # Raw mode avoids chat-template uncertainty. This aims near, but safely
    # below, the requested context length.
    prompt_words = context - 512
    prompt = (" memory" * prompt_words).lstrip()

    samples = []
    stop_monitor = threading.Event()

    def monitor():
        while not stop_monitor.is_set():
            try:
                samples.append(gpu_used_mib())
            except Exception:
                pass
            stop_monitor.wait(0.10)

    thread = threading.Thread(target=monitor, daemon=True)
    thread.start()

    started = time.perf_counter()
    error = None
    response = None

    try:
        response = ollama_request(
            {
                "model": MODEL,
                "prompt": prompt,
                "raw": True,
                "stream": False,
                "keep_alive": "5m",
                "options": {
                    "num_ctx": context,
                    "num_predict": 1,
                    "temperature": 0,
                    "seed": 1,
                },
            }
        )
    except Exception as exc:
        error = repr(exc)

    elapsed = time.perf_counter() - started
    stop_monitor.set()
    thread.join()

    peak = max(samples) if samples else gpu_used_mib()

    print("=" * 60)
    print("Requested context:", context)
    print("Baseline GPU MiB:", baseline)
    print("Peak GPU MiB:", peak)
    print("Increment above baseline MiB:", peak - baseline)
    print("Elapsed seconds:", round(elapsed, 2))

    if response:
        print("Prompt tokens evaluated:", response.get("prompt_eval_count"))
        prompt_duration = response.get("prompt_eval_duration", 0)
        if prompt_duration:
            print(
                "Prompt evaluation seconds:",
                round(prompt_duration / 1_000_000_000, 2),
            )

    if error:
        print("ERROR:", error)

    print("Ollama process state:")
    print(ollama_ps() or "(none)")
    print()

    unload()

print("=" * 60)
print("Final GPU MiB after unloading:", gpu_used_mib())
print("Benchmark complete.")
PY

echo
echo "Saved benchmark: $LOG"

Please let it finish all three context sizes, then paste the resulting output. If 32K exceeds the available memory, the script should record the error and continue to its cleanup rather than harming anything.

I’m using the writing-block format here because you need a complete relay that can be copied to Claude intact. I’ll include the design decision, the exact machine findings, what the first benchmark actually measured, and what the currently running Qwen test is intended to decide.

Copy this whole relay to Claude:

:::writing{variant="chat_message" id="68417"}
CHATGPT → CLAUDE

Darren asked how Oogway should prevent Nomic Embed and Qwen from competing for the RTX 3060’s 12 GB of VRAM. We have moved from design discussion into direct measurement on Oogway.

DECISION BEING TESTED

The simplest proposed v1 execution path is:

• Nomic query embeddings run on CPU.
• Qwen retains exclusive ordinary use of the GPU.
• Completed-conversation embeddings can run on CPU or be deferred.
• Bulk document embedding may use the GPU only during explicit ingestion sessions when Qwen is idle.
• No GPU lock/coordinator is built unless measurement demonstrates a need.

This would avoid simultaneous CUDA contexts, allocator partitioning, and dynamic VRAM coordination entirely.

OOGWAY INVENTORY

Measured before loading either model:

• CPU: 8 logical processors
• RAM: 31 GiB total, approximately 27 GiB available
• GPU: RTX 3060, 12,288 MiB
• Baseline GPU use: 564 MiB
• Ollama: 0.32.5
• Qwen model: qwen3:8b, 5.2 GB on disk
• No Ollama model was loaded
• PostgreSQL was active and accepting connections on port 5432
• The default shell Python did not contain PyTorch or SentenceTransformers
• Nomic was not installed as an Ollama model

We then located the original ingestion environment and cached model:

• Python environment:
  /home/darren/Documents/ougway_venv
• Cached model:
  /mnt/storage/hf-cache/models--nomic-ai--nomic-embed-text-v1.5
• Original ingestion scripts:
  /home/darren/Downloads/ingest_pile_v2.py
/home/darren/Documents/tokenspace/ingest_pile_v2.py

BENCHMARK ENVIRONMENT

The actual Nomic CPU benchmark used:

• Python 3.12.3
• PyTorch 2.6.0+cu124
• SentenceTransformers 5.1.1
• CUDA_VISIBLE_DEVICES=""
• HF_HUB_OFFLINE=1
• TRANSFORMERS_OFFLINE=1
• Model: nomic-ai/nomic-embed-text-v1.5
• Device explicitly set to CPU
• max_seq_length explicitly set to 8192
• normalize_embeddings=True
• One interop thread
• Two and four intra-op thread tests
• Correct task prefixes:
• search_query: for the short query
• search_document: for the full chunk
• A read-only PostgreSQL vector-retrieval loop was started concurrently against tokenspace

One qualification: the script established that the PostgreSQL loop started, but did not count completed database queries. The CPU timings have enough margin that this does not affect the immediate decision, but we should not describe the concurrent load as quantitatively measured.

The short query was 23 tokens. The test chunk was 1,517 characters and 250 tokens.

RESULTS

Cold model load:

• 6.401 seconds

Two CPU threads:

• Short-query median: 54.30 ms
• Short-query p95: 58.45 ms
• Full-chunk median: 310.03 ms
• Full-chunk p95: 312.62 ms

Four CPU threads:

• Short-query median: 38.15 ms
• Short-query p95: 45.32 ms
• Full-chunk median: 211.95 ms
• Full-chunk p95: 223.68 ms

The benchmark was saved as:

/home/darren/Desktop/nomic-cpu-benchmark-20260806-152318.txt

INTERPRETATION

CPU Nomic is decisively fast enough for live retrieval. Even the two-thread result adds only about 54 ms median to a query while leaving most CPU capacity available to PostgreSQL and the rest of the system.

The provisional v1 policy is therefore:

• Keep one Nomic CPU worker resident so the 6.4-second cold load is not paid per request.
• Use two threads for live-query embeddings.
• Optionally use four threads for deferred conversation write-back.
• Give Qwen ordinary exclusive use of the GPU.
• Use GPU Nomic only for explicit bulk-ingestion sessions while Qwen is idle.
• Do not build a GPU coordinator in v1.

PROFILE PRECISION NOTE

The benchmark used the cached SentenceTransformers/PyTorch path and is expected to be FP32, but the script did not print a parameter dtype. Because precision is profile identity, we will verify the actual loaded dtype before freezing the first embedding profile rather than infer it from the runtime.

QWEN TEST CURRENTLY RUNNING

A second benchmark is running now against qwen3:8b. It tests requested context allocations of:

• 8,192
• 16,384
• 32,768

For each context it:

• Unloads Qwen first
• Records baseline VRAM
• Sends a raw prompt near the requested context length
• Requests only one generated token
• Samples nvidia-smi every 100 ms
• Records peak VRAM and increase above baseline
• Records actual prompt tokens evaluated and prompt-evaluation time
• Captures ollama ps, including actual context and processor placement
• Unloads Qwen before proceeding to the next size

The test makes no database changes. Its purpose is to establish Qwen’s real long-context VRAM curve and determine the largest context that leaves an acceptable operational margin on the 12 GB card. It is no longer needed to decide Nomic scheduling—the CPU result already did that—but it is still needed to define Oogway’s safe Qwen context ceiling.

Darren will relay the Qwen output when the run completes.
:::

The execution path is now settled. Qwen remains entirely on the GPU through 32K context, while CPU-based Nomic is fast enough that the two never need to compete for VRAM.

All three remained 100% GPU-resident. Qwen also unloaded cleanly, returning the GPU to 552 MiB.

My v1 recommendation is:

• Normal/default Qwen context: 16K
• Permitted maximum: 32K
• Do not use the model’s full 40,960-token capability routinely; extrapolation leaves too little safety margin.
• Live Nomic embeddings: CPU, two threads
• Deferred CPU embedding: up to four threads
• Bulk GPU embedding: only after explicitly unloading Qwen
• GPU coordinator: unnecessary

That also explains your earlier observation of Qwen occupying approximately 5–6 GB: it was probably operating at a smaller context allocation. Context length accounts for the additional measured VRAM.

Restore the terminal environment now so ordinary Python programs can see the GPU again:

unset CUDA_VISIBLE_DEVICES HF_HUB_OFFLINE TRANSFORMERS_OFFLINE HF_HOME
unset PY LOG DB_LOAD_PID

Here is the update for Claude:

:::writing{variant="chat_message" id="31749"}
CHATGPT → CLAUDE

The Qwen benchmark has completed, and the execution-path question is resolved.

MODEL

• qwen3:8b
• 8.2B parameters
• Q4_K_M
• Native context limit: 40,960
• Embedding length: 4,096
• RTX 3060: 12,288 MiB

RESULTS

8,192 context:

• Prompt tokens actually evaluated: 7,680
• Peak GPU use: 6,785 MiB
• Remaining physical VRAM: 5,503 MiB
• Prompt evaluation: 4.31 seconds
• Entire model remained on GPU

16,384 context:

• Prompt tokens actually evaluated: 15,872
• Peak GPU use: 7,949 MiB
• Remaining physical VRAM: 4,339 MiB
• Prompt evaluation: 10.03 seconds
• Entire model remained on GPU

32,768 context:

• Prompt tokens actually evaluated: 32,256
• Peak GPU use: 10,086 MiB
• Remaining physical VRAM: 2,202 MiB
• Prompt evaluation: 27.73 seconds
• Entire model remained on GPU

After explicit unloading, GPU use returned to 552 MiB.

The benchmark log is:

/home/darren/Desktop/qwen-vram-benchmark-20260806-152555.txt

CONCLUSION

The measured v1 execution policy is:

• Nomic live-query embedding runs on CPU using two threads.
• Its measured median query latency is 54 ms, with p95 58 ms.
• Four threads may be used for deferred write-back; full 1,500-character chunks measured 212 ms median and 224 ms p95.
• Qwen receives exclusive ordinary GPU use.
• Normal Qwen context should be 16K.
• The permitted long-session ceiling should be 32K.
• The model’s full 40,960 context is not operationally attractive on this card; extrapolation would leave approximately 1 GB or less of margin.
• GPU Nomic is reserved for explicit bulk-ingestion sessions after Qwen is unloaded.
• No GPU allocation cap, lock, queue, or coordinator is required in v1.

The only unfinished measurement before freezing the first embedding profile is direct confirmation of the cached SentenceTransformers model’s loaded parameter dtype. The CPU benchmark used that reference path and it is expected to be FP32, but the benchmark did not print the dtype, so we will measure rather than infer it.
:::

Claude’s latency interpretation is useful, but the KV-cache claim is load-bearing enough that I don’t want to accept it from general llama.cpp behavior alone. Ollama and OpenWebUI must actually preserve and reuse the matching prefix across separate API requests; I’m verifying that exact path before we freeze prompt ordering around it.

Claude’s main conclusion is right: at 32K, latency is the practical constraint before VRAM is. The important distinction is that our 28-second measurement was a cold, nearly complete prompt evaluation. Normal cached turns should be much faster if Ollama receives a genuinely stable prefix.

Two qualifications:

• More VRAM alone would not solve the 32K latency. A substantially faster GPU could; merely adding memory would not.
• llama.cpp supports longest-common-prefix reuse, and recent Ollama releases explicitly include prompt-cache improvements. However, actual reuse still depends on how OpenWebUI serializes and injects retrieved material. We should treat reuse as something to verify after integration, not as guaranteed merely because the backend supports it. Ollama releases, llama.cpp prefix-cache explanation

The prompt-assembly invariant is sound regardless:

stable system/persona
stable boot memory
chronological conversation history
dynamic retrieved memory
current user message

Dynamic retrieval belongs near the end. If it is placed near the beginning, its turn-to-turn changes destroy most of the common prefix. We should also keep retrieved text transient rather than permanently copying every retrieved passage into conversation history; provenance references can be retained separately.

Claude’s bulk-ingestion caveat is also accepted. “No coordinator” means explicit exclusivity:

1. Prevent new OpenWebUI requests.
2. Unload Qwen.
3. Verify ollama ps is empty.
4. Run GPU bulk embedding.
5. Unload the embedder.
6. Reopen conversational use.

Now run the final dtype check:

CUDA_VISIBLE_DEVICES="" \
HF_HOME=/mnt/storage/hf-cache \
HF_HUB_OFFLINE=1 \
TRANSFORMERS_OFFLINE=1 \
/home/darren/Documents/ougway_venv/bin/python - <<'PY'
from collections import Counter
import torch
from sentence_transformers import SentenceTransformer

model = SentenceTransformer(
    "nomic-ai/nomic-embed-text-v1.5",
    device="cpu",
    cache_folder="/mnt/storage/hf-cache",
    trust_remote_code=True,
)

parameter_dtypes = Counter(str(p.dtype) for p in model.parameters())
buffer_dtypes = Counter(
    str(b.dtype)
    for b in model.buffers()
    if b.is_floating_point()
)

print("Parameter dtypes:", dict(parameter_dtypes))
print("Floating buffer dtypes:", dict(buffer_dtypes))
print("First parameter dtype:", next(model.parameters()).dtype)
print("SentenceTransformers max_seq_length:", model.max_seq_length)
print("Tokenizer model_max_length:", model.tokenizer.model_max_length)
print("Default device:", model.device)
print("PyTorch version:", torch.__version__)
PY

That completes the last measurement needed before defining the first embedding profile.

That confirms the cached SentenceTransformers model is genuinely FP32:

• All 112 parameter tensors: torch.float32
• All 24 floating-point buffers: torch.float32
• SentenceTransformers limit: 8,192 tokens
• Tokenizer limit: 8,192 tokens
• Device: CPU
• PyTorch: 2.6.0+cu124

So the execution decision is final: live retrieval uses resident, two-thread CPU Nomic FP32; Qwen retains the GPU.

Relay for Claude:

CHATGPT → CLAUDE

The final dtype measurement is complete.

Cached reference runtime:

- Model: nomic-ai/nomic-embed-text-v1.5
- Device: CPU
- Parameter tensors: 112, all torch.float32
- Floating-point buffers: 24, all torch.float32
- First parameter dtype: torch.float32
- SentenceTransformers max_seq_length: 8192
- Tokenizer model_max_length: 8192
- PyTorch: 2.6.0+cu124

Therefore the first embedding profile’s precision is definitively FP32 rather than inferred.

The performance and scheduling decision is now closed:

- Resident Nomic CPU worker
- Two threads for live-query embedding
- Four threads permitted for deferred write-back
- Qwen receives ordinary exclusive GPU use
- 16K normal Qwen context
- 32K permitted ceiling
- GPU bulk embedding only during explicitly exclusive sessions
- No GPU coordinator in v1

Prompt assembly preserves the longest stable prefix by placing dynamic retrieved memory immediately before the current user turn. Actual Ollama/OpenWebUI prefix reuse remains an integration acceptance test rather than an assumed guarantee.

The measurement phase is complete. When we create the profile record, we still need to capture the cached model artifact revision/digest and explicitly select the overlength-input policy—most likely rejection instead of silent truncation. Those are implementation records, not further performance experiments.