Database Formalization Progress — Part 2

:::writing{variant="social_post" id="68417"}
LITTLE OUGWAY PROJECT UPDATE: THE DATABASE INVESTIGATION IS COMPLETE

Before connecting Little Ougway to its long-term memory, I decided to stop and re-examine the database we had already built. That turned into a detailed forensic investigation involving ChatGPT, Claude, the original Python ingestion program, PostgreSQL’s internal records, filesystem evidence, and the surviving ingestion logs.

The investigation is now complete—and we found considerably more than I expected.

The existing database contains 321 ingested container files and exactly 13,709,301 text chunks. Those chunks occupy a database of roughly 155 GB, including a 53 GB HNSW vector index. However, the source directory actually contains 15,325 large corpus shards totaling approximately 799 GiB. In other words, the old database contains only about 2.1% of the available corpus.

Had we continued ingesting all of it using the old design, the resulting database would probably have required somewhere around seven terabytes. Even the new, more efficient design would still require at least five terabytes because embeddings and their search index—not the text itself—are the binding storage cost.

That settled an important question: Little Ougway should not indiscriminately ingest an enormous general-purpose corpus. It will learn from deliberately selected material that adds something distinctive, while its own conversations and developing memory remain a separate system.

We also discovered exactly what happened during the strange ingestion rerun in March 2026.

The program was described as resumable and idempotent, meaning that rerunning unchanged material should have done nothing. In reality, it skipped recalculating the embeddings but still performed unconditional database updates on every existing document and chunk.

The rerun successfully processed 211 already-ingested files, rewriting roughly nine million rows without changing their logical content. Several interrupted restarts then consumed additional database sequence numbers even though their transactions never completed.

At first, the physical database appeared inconsistent with that history. Nine million rewritten rows should have produced far more dead data and much greater temporary growth than we observed. After several rounds of argument and testing, the explanation was found: PostgreSQL’s ordinary vacuuming process was reclaiming obsolete row versions while the rerun continued, allowing later updates to reuse the freed pages.

The vector index had its own recycling mechanisms. Because the embeddings were unchanged, pgvector could frequently attach the new row location to the existing identical vector rather than constructing another complete vector node. Vacuuming also marked deleted HNSW elements for reuse.

The logical history and physical evidence now agree. The mystery is solved.

More importantly, that failure produced concrete requirements for the replacement system:

• An unchanged rerun must perform no domain writes and consume no domain identifiers.
• The program must resolve and compare an existing document before attempting an insert.
• Progress reporting must never repeatedly scan the entire accumulated database.
• Every ingestion run must distinguish attempted, skipped, committed, rejected, and failed work.
• Every stored artifact must record what run and software version produced it.
• Hard deletion must be an explicit governed operation, never an accidental side effect.

We have also settled the identity model for the new memory system.

A document will be one deliberately submitted source file—not one of the old 56 MB corpus containers. Each document will have stable identity across revisions. Its source versions, normalized text renditions, chunk sets, and embeddings will be separate immutable layers.

Chunks will be stored primarily as spans pointing into one authoritative text rendition rather than carrying another complete copy of the text. Assertions, concepts, hypotheses, memories, and promoted passages will become registered lattice nodes. Ordinary source passages will remain evidence rather than automatically becoming members of the conceptual lattice.

This preserves an important distinction:

The reference corpus is something Ougway can consult. It is not Ougway’s autobiographical memory, identity, belief strength, or continuity.

We also established safeguards against accidentally feeding another enormous corpus into the new system. Individual documents will have hard size and predicted-chunk limits. Large runs will pause and show their complete predicted scale for confirmation. Admitting an oversized document will require a separate, deliberate, recorded override.

WHERE THE PROJECT STANDS

We are now at the implementation threshold.

The legacy database has been fully examined. Its anomaly is resolved. The identity-bearing parts of the replacement architecture are settled, and the design has been deliberately divided into what must exist immediately and what can safely be added later.

The physical archive has not yet been created. That is the next operation.

NEXT STEPS

1. Inventory and archive the existing PostgreSQL cluster, corpus, scripts, logs, reports, and investigation findings.

2. Verify that the archived database can be recovered before changing anything.

3. Preserve the old PostgreSQL cluster offline as a historical artifact.

4. Create a completely fresh PostgreSQL cluster for Little Ougway’s replacement memory system.

5. Build one small end-to-end test using a real document containing ordinary and non-ASCII text.

6. Ingest it, normalize it, create immutable chunks and embeddings, retrieve it, and verify every returned span against its stored hash.

7. Rerun it unchanged and prove that nothing except the audit record changes.

8. Change the document from version A to B and back to A, proving that its original immutable version is reused while the complete transition history remains intact.

Only after that vertical slice works will we populate the conceptual lattice and connect it to Ougway’s conversation and memory processes.

The larger plans—curiosity, autonomous reflection, topology, visualization, and discovering connections across accumulated knowledge—remain ahead. But they will now be built on a small, testable foundation rather than another giant ingestion experiment.

The investigation took us backward through the old machinery, but it brought the project forward. We now know what failed, why it failed, what must never happen again, and exactly what the first working replacement needs to accomplish.
:::

There are two different resources hiding under “Nomic Embed”: tokenization is mostly CPU work, while embedding inference consumes GPU memory. I’m checking the current PyTorch and Ollama controls so I can separate a real memory limit from controls that only look like one.

Yes—but “80% of what remains” is not the safest control. On Oogway’s single 12 GB GPU, I recommend a separate embedding worker plus scheduled GPU access, rather than allowing Nomic and Qwen to compete freely.

A few important distinctions:

• Tokenization and chunking are primarily CPU operations and require very little GPU memory.
• Producing the 768-dimensional Nomic embeddings is the GPU-intensive operation.
• Running Nomic in a separate process improves isolation and scheduling, but it does not automatically reserve separate VRAM. Both processes still share the same GPU.

For Oogway, I would implement this policy:

1. Nomic runs as a single dedicated worker with a queue.
2. Only one model performs GPU inference at a time initially.
3. For retrieval, Nomic embeds the user’s query first—normally one short batch—then releases the GPU to Qwen.
4. Qwen generates the answer.
5. Embedding and storing the completed conversation happens afterward, or whenever Qwen is idle.
6. Bulk document embedding runs only during idle periods or an explicitly started ingestion session.
7. Small conversational embedding jobs can fall back to CPU if we want absolutely no interference with Qwen.

If we continue using SentenceTransformers/PyTorch for Nomic, we can also install a guardrail such as:

import torch

torch.cuda.set_per_process_memory_fraction(0.25, device=0)

That caps the PyTorch caching allocator at approximately 25% of the entire 12 GB GPU—about 3 GB. Importantly, 0.80 would mean roughly 9.6 GB of total VRAM, not 80% of whatever Qwen leaves available. PyTorch will raise an out-of-memory error rather than exceed its allocator allowance. It is a useful safety barrier, although it is not a physical GPU partition. PyTorch documentation

We would additionally keep the Nomic batch size small—probably 1–4 for conversations rather than the legacy ingestion script’s batch size of 64. Batch size is the primary practical control over embedding inference memory. SentenceTransformers documentation

So my recommended starting configuration is:

• Qwen: normal allocation, approximately its observed 5–6 GB.
• Nomic GPU ceiling: 25% of total VRAM, approximately 3 GB.
• Conversation batch size: 1–4.
• GPU inference serialized through one lock or coordinator.
• CPU fallback for urgent embedding work.
• Bulk embedding postponed while Qwen is active.

That leaves approximately 3 GB of breathing room for CUDA context, temporary allocations, display usage, and model-generation spikes. We can benchmark it later and cautiously permit overlap if the measurements demonstrate that it is safe.

In short: yes, make Nomic a separate process—but for scheduling and fault isolation, not because that alone partitions memory. The dependable protection is small batches, a PyTorch allocation cap, and initially preventing simultaneous GPU inference.

Claude’s additions mostly tighten the implementation correctly. I’m checking one important distinction before we freeze it: whether the execution backend belongs in the embedding profile’s identity, or only in each embedding run’s provenance. That determines whether merely moving the same model from CPU to GPU forces a complete re-embedding.

Claude’s additions are sound, with one important refinement: precision belongs to embedding identity; execution location usually belongs to run provenance.

SentenceTransformers FP32 and Ollama’s Nomic v1.5 F16 weights should be separate embedding profiles. Ollama’s published v1.5 artifact is indeed F16. Ollama model metadata

But merely switching the same FP32 model between CPU and GPU should not automatically mint a new profile. CPU and GPU can produce slightly different floating-point results even from identical inputs, so the backend and hardware should be recorded on the embedding run and checked within our declared verification tolerance. Otherwise every PyTorch, CUDA, driver, or hardware change could fragment the embedding space unnecessarily. PyTorch numerical-accuracy documentation

The practical v1 decision is now pleasantly small:

1. Benchmark Nomic FP32 on Oogway’s CPU using representative short queries and full-sized chunks.
2. Measure both cold-start and warm latency.
3. If a warm query embed is acceptably fast—say, below roughly 250–300 ms—use CPU for live conversation retrieval.
4. Run bulk embedding on the GPU only during explicit ingestion sessions when Qwen is idle.
5. Do not build the GPU coordinator unless the CPU measurement proves we need it.
6. Separately measure Qwen’s peak VRAM at the longest context we actually intend to support, not merely an ordinary conversation.

That would leave us with:

• Live query embedding: CPU
• Qwen generation: GPU
• Conversation write-back embedding: CPU or deferred
• Bulk document embedding: GPU while Qwen is idle
• Simultaneous GPU contention: none
• Coordinator/queue machinery: postponed unless measurements justify it

Claude is also right about CUDA overhead, with the qualification that “a few hundred megabytes per process” is an empirical expectation, not a guaranteed fixed amount. CUDA contexts and library workspaces sit partly outside PyTorch’s allocator cap, so our decision must be based on nvidia-smi measurements rather than arithmetic alone.

The identity rule I would freeze is:

> Model artifact digest, weight precision, dimensions, tokenizer and preprocessing, pooling, normalization, and canonicalization version define the embedding profile. Runtime backend, device, library versions, drivers, and hardware are recorded as embedding-run provenance. A backend is compatible with an existing profile only if verification stays within its declared tolerance.

So Claude has identified the right next experiment: before building memory coordination machinery, time one warm CPU query embedding and measure Qwen under a deliberately long conversation. Those two measurements decide almost the entire scheduling design.

That closes the profile definition cleanly. I’m checking the exact Nomic v1.5 prefix contract, because there is one possible wrinkle: some implementations add prefixes automatically while others require the caller to supply them, and we must not accidentally apply one twice.

Agreed. Those three belong in the specification, and the prefix requirement is confirmed by Nomic’s model card: the caller must supply search_document:, search_query:, clustering:, or classification: as appropriate. Nomic v1.5 model card

One additional hardening: cosine drift and retrieval stability are related, but not identical. An extremely small vector change can still reverse two nearly tied neighbors. Therefore backend compatibility should eventually have two checks:

• Maximum self-drift across the frozen probe set:
  max(1 - cosine(reference, candidate)) < tolerance
• Retrieval canary check against a small frozen index: expected nearest neighbors or top-k membership must remain stable.

The first detects numerical divergence; the second detects whether that divergence changes behavior. We do not need to build the second test before the first write, but we should preserve enough probe material to add it without reconstituting the original environment.

The probe set should include both sides of retrieval:

• Document passages embedded with search_document:
• Representative questions embedded with search_query:
• Short, medium, and maximum intended lengths
• ASCII and multiple Unicode scripts
• Difficult near-duplicate passages
• Reference vectors, model/profile digest, creation date, and reference runtime provenance

A few hundred 768-dimensional FP32 vectors occupy well under 1 MB, so preserving them is cheap.

The prefixes should never be stored in the authoritative rendition text. A single canonical embedding function should receive (task, raw_text), apply exactly one configured prefix, and then tokenize. We should not rely on detecting whether text “already looks prefixed,” because a real document could naturally begin with those words.

The CPU benchmark should explicitly set PyTorch’s thread counts and test at least two reasonable allocations—probably two and four cores—while Postgres performs a representative retrieval query. That gives us a real shared-machine measurement instead of an isolated speed record.

So the two immediate measurements become:

1. Nomic CPU latency at controlled thread counts, covering cold load, warm query embedding, and full-sized document chunks.
2. Qwen peak VRAM while processing the longest conversation context we actually intend to permit.

Those results decide whether Oogway needs any GPU coordinator at all. My expectation remains that CPU query embedding plus scheduled GPU bulk ingestion will make the coordinator unnecessary in v1, but now it will be a measured conclusion rather than an assumption.

All three refinements are accepted. The canary must use an exact cosine scan; HNSW behavior gets its own test later.

I would strengthen max_seq_length into three profile-defining fields:

• Maximum accepted input tokens
• Truncation policy: reject, truncate-head, truncate-tail, or another explicit method
• Context-extension/position-encoding configuration when applicable

That matters because Nomic v1.5 supports long context, but individual runtimes can silently operate at smaller defaults. “8192 supported” does not prove that a particular execution path actually used 8192.

For exact-canary ranking, equal or nearly equal candidates also need deterministic treatment:

• Break exact ties using immutable probe ID.
• Compare top-k membership rather than incidental ordering among tied items.
• Record the similarity margin between the expected neighbor and its nearest competitor.

The span rule should be frozen exactly as Claude states:

> Offsets and exact_text_hash are calculated from authoritative rendition text. The embedding task prefix is added only after resolving and verifying that span. Prefixes never enter stored text, hashes, or offsets.

There is one useful normalization distinction. Document normalization belongs to the extraction/rendition layer; the embedder should consume that canonical text without independently changing it. Queries do not pass through rendition extraction, so query preprocessing must apply the matching Unicode normalization before adding search_query:. Otherwise two layers can both claim normalization while quietly implementing it differently.

The NFC/NFD probes should consequently test the whole boundary:

1. Raw NFC and NFD document inputs normalize to the expected canonical rendition.
2. Their resolved spans and exact-text hashes agree when the extraction policy says they should.
3. An equivalent user query receives the same declared normalization before embedding.
4. Deliberately disabling or changing that normalization makes the canary fail.

This completes the measurement specification. We should now stop extending the design and run the two machine tests: controlled-thread CPU embedding latency and Qwen’s long-context peak VRAM. Those results determine the actual v1 execution path.

Agreed. One canonical, versioned normalization function will serve both extraction and query preprocessing; profiles reference its identity rather than duplicating its settings.

The design discussion is closed. The next evidence must come from Oogway:

• Controlled CPU benchmark for Nomic
• Long-context peak-VRAM measurement for Qwen

When you are ready on Oogway, I’ll give you one script that runs and records both tests without changing the database. Those results will determine whether v1 uses CPU embeddings, scheduled GPU embeddings, or requires coordination.

Good. Before we run the actual load test, I want one short read-only inventory from Oogway so the benchmark uses the installed Python environment and exact Qwen model tag instead of guessing. This will not touch the database or change any configuration.

Please paste this into Oogway’s terminal and return the complete output:

(
date -Is
hostname
printf '\nCPU and memory:\n'
nproc
free -h

printf '\nGPU:\n'
nvidia-smi \
  --query-gpu=name,memory.total,memory.used,memory.free,utilization.gpu \
  --format=csv

printf '\nOllama:\n'
command -v ollama
ollama --version
ollama list
ollama ps

printf '\nPython environments and ingestion script:\n'
rg --files /home/darren /mnt/storage \
  -g 'pyvenv.cfg' \
  -g 'ingest_pile_v2.py' 2>/dev/null | head -50

printf '\nSystem Python packages:\n'
python3 - <<'PY'
import importlib.util
import os
import sys

print("executable:", sys.executable)
print("version:", sys.version.replace("\n", " "))
for package in ("torch", "sentence_transformers", "psycopg", "psycopg2"):
    spec = importlib.util.find_spec(package)
    print(f"{package}:", spec.origin if spec else "not installed")
print("torch threads env:", {
    key: os.environ.get(key)
    for key in ("OMP_NUM_THREADS", "MKL_NUM_THREADS")
})
PY

printf '\nPostgreSQL availability:\n'
pg_isready 2>&1 || true
systemctl is-active postgresql 2>&1 || true
)

This identifies the correct Nomic runtime, Qwen tag, available resources, and PostgreSQL state. After seeing it, I’ll give you the actual benchmark as one copy-and-paste script.

The hardware side is clean: 8 CPU threads, 27 GiB available RAM, and the GPU is essentially idle. We also confirmed the exact Qwen model is qwen3:8b.

One missing piece surfaced: Nomic is not registered with Ollama, and the default Python has neither PyTorch nor SentenceTransformers. That means the old ingestion runtime is either in an environment the first search missed or has since been removed. I want to locate it before installing or downloading anything.

Please run this second read-only search:

(
printf 'Candidate Python environments:\n'
find /home/darren /mnt/storage \
  -type f -name pyvenv.cfg \
  -print 2>/dev/null

printf '\nSentenceTransformers installations:\n'
find /home/darren /mnt/storage \
  -type d -path '*/site-packages/sentence_transformers' \
  -print 2>/dev/null

printf '\nIngestion programs:\n'
find /home/darren /mnt/storage \
  -type f \( \
    -iname 'ingest_pile_v2.py' -o \
    -iname '*ingest*.py' \
  \) -print 2>/dev/null

printf '\nCached Nomic models:\n'
find /home/darren/.cache /mnt/storage \
  -type d -iname '*nomic*' \
  -print 2>/dev/null

printf '\nOpen WebUI service runtime:\n'
systemctl show openwebui \
  -p ExecStart \
  -p Environment \
  -p WorkingDirectory \
  --no-pager 2>/dev/null
)

It may take a minute because it examines /mnt/storage. Let it return to the normal prompt before pasting the output. Once we locate the original Nomic installation, the benchmark can use that exact model and precision rather than accidentally testing a different artifact.

That is the first inventory again—no harm done. Please run this different, shorter search and wait for the prompt to return:

find /home/darren /mnt/storage \
  \( \
    -type f \( -name pyvenv.cfg -o -iname '*ingest*.py' \) \
    -o \
    -type d \( -path '*/site-packages/sentence_transformers' -o -iname '*nomic*' \) \
  \) \
  -print 2>/dev/null

It should locate the old Nomic model, ingestion program, or Python environment. Paste whatever it returns—even if it returns nothing.

We found the original environment and cached FP32 model:

• Python environment: /home/darren/Documents/ougway_venv
• Nomic cache: /mnt/storage/hf-cache
• Original ingestion scripts: still present in both Downloads and Documents

We can now benchmark the actual SentenceTransformers path without downloading anything or changing the database. We’ll do Nomic first; after its result, we’ll run Qwen at several controlled context sizes rather than combining two long tests into one opaque run.

Paste this entire block. It loads Nomic strictly from the local cache, uses CPU only, tests two and four threads, and attempts a concurrent read-only vector query against PostgreSQL.

export HF_HOME=/mnt/storage/hf-cache
export HF_HUB_OFFLINE=1
export TRANSFORMERS_OFFLINE=1
export CUDA_VISIBLE_DEVICES=""

PY=/home/darren/Documents/ougway_venv/bin/python
LOG="$HOME/Desktop/nomic-cpu-benchmark-$(date +%Y%m%d-%H%M%S).txt"

DB_LOAD_PID=""

cleanup_benchmark() {
    if [ -n "$DB_LOAD_PID" ]; then
        kill "$DB_LOAD_PID" 2>/dev/null || true
        wait "$DB_LOAD_PID" 2>/dev/null || true
    fi
}
trap cleanup_benchmark EXIT

if psql -d tokenspace -Atqc "SELECT 1" >/dev/null 2>&1; then
    echo "PostgreSQL connection available; starting concurrent read-only retrieval load."
    (
        while true; do
            psql -d tokenspace -Atqc "
                SELECT chunk_id
                FROM content.chunks
                WHERE embedding IS NOT NULL
                ORDER BY embedding <=> (
                    SELECT embedding
                    FROM content.chunks
                    WHERE embedding IS NOT NULL
                    LIMIT 1
                )
                LIMIT 10;
            " >/dev/null 2>&1 || exit
            sleep 0.10
        done
    ) &
    DB_LOAD_PID=$!
else
    echo "PostgreSQL login unavailable for this user; benchmark will continue without DB load."
fi

"$PY" - <<'PY' 2>&1 | tee "$LOG"
import os
import statistics
import sys
import time

os.environ["CUDA_VISIBLE_DEVICES"] = ""

import torch
import sentence_transformers
from sentence_transformers import SentenceTransformer

MODEL = "nomic-ai/nomic-embed-text-v1.5"
CACHE = "/mnt/storage/hf-cache"

query = (
    "search_query: How does immutable memory preserve the reasons "
    "that an artificial intelligence changed its conclusions?"
)

paragraph = (
    "A durable memory system separates stable identity from changing content. "
    "Documents retain their logical identities while immutable versions preserve "
    "the exact evidence available at a particular time. Chunks are spans within "
    "a rendition, and assertions cite those spans without silently redirecting "
    "historical reasoning when the source later changes. "
)
chunk = "search_document: " + (paragraph * 8)[:1500]

print("Python:", sys.version.replace("\n", " "))
print("PyTorch:", torch.__version__)
print("SentenceTransformers:", sentence_transformers.__version__)
print("CUDA visible:", os.environ.get("CUDA_VISIBLE_DEVICES"))
print("CPU threads reported:", os.cpu_count())

torch.set_num_interop_threads(1)
torch.set_num_threads(2)

started = time.perf_counter()
model = SentenceTransformer(
    MODEL,
    device="cpu",
    cache_folder=CACHE,
    trust_remote_code=True,
)
load_seconds = time.perf_counter() - started

model.max_seq_length = 8192

print("Model load seconds:", round(load_seconds, 3))
print("Model max_seq_length:", model.max_seq_length)
print("Query tokens:", len(model.tokenizer(query)["input_ids"]))
print("Chunk characters:", len(chunk))
print("Chunk tokens:", len(model.tokenizer(chunk)["input_ids"]))

def measure(text, repetitions):
    model.encode(
        [text],
        batch_size=1,
        show_progress_bar=False,
        normalize_embeddings=True,
    )

    timings = []
    for _ in range(repetitions):
        started = time.perf_counter()
        model.encode(
            [text],
            batch_size=1,
            show_progress_bar=False,
            normalize_embeddings=True,
        )
        timings.append((time.perf_counter() - started) * 1000)

    ordered = sorted(timings)
    p95 = ordered[int(0.95 * (len(ordered) - 1))]
    return {
        "median_ms": round(statistics.median(timings), 2),
        "p95_ms": round(p95, 2),
        "min_ms": round(min(timings), 2),
        "max_ms": round(max(timings), 2),
    }

for threads in (2, 4):
    torch.set_num_threads(threads)
    print()
    print(f"Threads: {threads}")
    print("Warm short query:", measure(query, 20))
    print("Warm full chunk:", measure(chunk, 10))

print()
print("Benchmark complete.")
PY

cleanup_benchmark
trap - EXIT

echo
echo "Saved benchmark: $LOG"

It may pause during the initial model load; that is the cold-load measurement. Let it finish and return to the prompt, then paste the complete output.