Unix socket database setup — Part 1

Beautiful — your venv sees the GPU:

• cuda_available: True
• GPU: RTX 3060
• CUDA: 12.4 (driver runtime)

Let’s flip the ingest to run on CUDA explicitly and watch it move.

Run ingest on GPU (one-liner)

export ROOT_DIR=/media/darren/storage/pile_uncopyrighted
export PGHOST=/var/run/postgresql
export PGPORT=5433
SENTENCE_TRANSFORMERS_DEFAULT_DEVICE=cuda SENTENCE_TRANSFORMERS_TRUST_REMOTE_CODE=true \
python3 ~/Documents/tokenspace/ingest_pile.py

Optional (make it default every time)

Add these to ~/.bashrc:

export PGHOST=/var/run/postgresql
export PGPORT=5433
export ROOT_DIR=/media/darren/storage/pile_uncopyrighted
export SENTENCE_TRANSFORMERS_DEFAULT_DEVICE=cuda
export SENTENCE_TRANSFORMERS_TRUST_REMOTE_CODE=true

source ~/.bashrc afterwards.

Live monitors (use in another terminal)

GPU:

watch -n 1 nvidia-smi

DB counters (proves rows are climbing):

watch -n 2 'psql -h /var/run/postgresql -p 5433 -d tokenspace -c "select count(*) docs from content.documents; select count(*) chunks from content.chunks;"'

If it still feels slow

You can try these quick tweaks (safe):
– Smaller chunks (faster per-doc commits):
bash
export CHUNK_SIZE_CHARS=800 CHUNK_OVERLAP_CHARS=120
– Slightly smaller embedding batch (helps stability on 12 GB GPUs if needed):
bash
sed -i 's/BATCH = 64/BATCH = 32/' ~/Documents/tokenspace/ingest_pile.py

You’re set—CUDA’s lit, so embeddings should accelerate and you’ll start seeing the [ok] … -> N chunk(s) lines again.

That’s a good readout — GPU is clearly chewing, but DB counters staying at 0 docs / 0 chunks means nothing’s getting committed yet. A couple of things can explain that:

---

1. First file might be huge

The script only prints and commits after it finishes embedding and updating an entire document.
If the first file in /media/darren/storage/pile_uncopyrighted is very large (e.g. multi-MB text, thousands of chunks), you’ll see long GPU churn but no rows until the doc is done. Once that doc commits, you’ll suddenly see docs=1 and a big jump in chunks.

👉 Test: run on just a few tiny files to see if the pipeline works:

mkdir -p /tmp/ingest_test
find /media/darren/storage/pile_uncopyrighted -type f -name '*.txt' | head -n 5 | xargs -I{} cp "{}" /tmp/ingest_test/
ROOT_DIR=/tmp/ingest_test \
PGHOST=/var/run/postgresql PGPORT=5433 \
python3 ~/Documents/tokenspace/ingest_pile.py

If this inserts rows, then the big corpus is simply front-loading a whale of a file.

---

2. Transaction rollback on error

If something in that first insert fails (permissions, schema mismatch, unexpected NULL, etc.), the script rolls back and you’ll never see rows increase. It should also print [error] … if that happens, but if it’s still spinning, it’s likely just processing.

---

3. Confirm DB writes are possible

Try a manual insert:

psql -h /var/run/postgresql -p 5433 -d tokenspace -U darren -c \
"INSERT INTO content.documents (title, external_id, meta) VALUES ('testdoc','/tmp/test','{}') RETURNING doc_id;"

If that works, permissions are fine and the block is in the script’s work loop.

---

4. Watch logs for per-doc prints

The script prints:

[ok] /path/to/file -> N chunk(s)

after each document commit. If you’re not seeing that, you’re still on doc 1.

---

⚡ Next move: try the 5-file test in /tmp/ingest_test. If that produces docs/chunks, then the system is healthy — it’s just grinding a monster file first in the big set.

Do you want me to patch the ingest script to log progress per batch (e.g. every 100 chunks) instead of waiting until the whole document is done? That way you’d see activity immediately even on huge files.

Got it. The GPU is loaded (model sitting in ~10 GB VRAM), disk is twitching, but you’re seeing no DB rows and no more console output after “found 15,325 files.” That means the script is likely chewing a big first file and doesn’t commit or print until the entire document finishes. Let’s make it show progress and write rows immediately, not at the end of the doc.

Below are surgical patches you can apply in-place—no guessing, no refactor—so you’ll see movement within seconds:

---

1) Commit chunk rows before embeddings (so DB counters move fast)

What this does: inserts all chunk metadata for a file and commits right away. Embeddings are updated afterward. You’ll see content.chunks grow quickly.

# backup first
cp ~/Documents/tokenspace/ingest_pile.py ~/Documents/tokenspace/ingest_pile.py.bak

# add a commit right after inserting chunks (BEFORE embeddings)
awk '
  {print}
  /# Upsert chunk rows \(metadata only first\), collect ids/ {flag=1}
  flag && /for seq, chunk_text_i, tokc in zip/ {seen=1}
  flag && seen && /chunk_ids.append\(cid\)/ {print; print "            # COMMIT after inserting chunk rows so DB shows progress"; print "            conn.commit()"; next}
' ~/Documents/tokenspace/ingest_pile.py > /tmp/ingest_pile.tmp && mv /tmp/ingest_pile.tmp ~/Documents/tokenspace/ingest_pile.py

> If the awk doesn’t find the exact anchor, no worries—see step 4 for a minimal alternative.

---

2) Print progress for each document and each embedding batch

What this does: logs how many chunks a file has and prints every time a batch of embeddings finishes.

# print chunks count per doc (right after chunking)
sed -i '/chunks = chunk_text/s/$/\n            print(f"[doc] {external_id} -> {len(chunks)} chunks")/' \
  ~/Documents/tokenspace/ingest_pile.py

# print per-embed batch progress and commit each batch (so you see work even on giant docs)
sed -i '/for i in range(0, len(chunks), BATCH):/a \\
                print(f" {i+len(batch_texts)}/{len(chunks)} for {external_id}")\\
                conn.commit()' \
  ~/Documents/tokenspace/ingest_pile.py

---

3) (Optional) Smaller batches + smaller chunks to speed early commits

# shrink embedding batch to reduce latency per batch
sed -i 's/BATCH = 64/BATCH = 16/' ~/Documents/tokenspace/ingest_pile.py

# for this run, use smaller chunking (quicker per-file turnaround)
export CHUNK_SIZE_CHARS=800
export CHUNK_OVERLAP_CHARS=120

---

4) If the awk/sed anchors missed (quick minimal patch)

As a fallback, add two prints and an early commit by hand:

Open the file:

nano ~/Documents/tokenspace/ingest_pile.py

Insert these three lines in the indicated spots:

• After:

chunks = chunk_text(text, CHUNK_SIZE_CHARS, CHUNK_OVERLAP_CHARS)

add:

print(f"[doc] {external_id} -> {len(chunks)} chunks")

• After the loop that inserts chunk rows (right after chunk_ids.append(cid) loop ends), add:

# commit chunk rows so DB shows progress immediately
conn.commit()
print(f"[commit] wrote {len(chunk_ids)} chunk rows for {external_id} (no embeddings yet)")

• Inside the embeddings batch loop, after for i in range(0, len(chunks), BATCH): and after the inner for cid, vec in zip(...): finishes (i.e., at end of the batch), add:

print(f" {min(i+BATCH, len(chunks))}/{len(chunks)} for {external_id}")
conn.commit()

Save and exit.

---

5) Run with GPU on and watch both GPU + DB counters

export ROOT_DIR=/media/darren/storage/pile_uncopyrighted
export PGHOST=/var/run/postgresql
export PGPORT=5433
export SENTENCE_TRANSFORMERS_DEFAULT_DEVICE=cuda
export SENTENCE_TRANSFORMERS_TRUST_REMOTE_CODE=true

python3 ~/Documents/tokenspace/ingest_pile.py

In another terminal:

watch -n 1 nvidia-smi

and:

watch -n 2 'psql -h /var/run/postgresql -p 5433 -d tokenspace -c "select count(*) as docs from content.documents; select count(*) as chunks from content.chunks;"'

You should start seeing:

• [doc] /path/to/file -> N chunks almost immediately per file,
• [commit] wrote N chunk rows ... so chunks counter rises fast,
• X/Y for /path ... every batch,
• and eventually [ok] ... -> N chunk(s) (your original per-file summary).

---

Why this fixes your “stuck after scan” symptom

• Previously, you only saw output and rows after the whole doc was done (all chunks embedded). If doc 1 is massive, that can look like a freeze.
• Now, chunk metadata is committed before embeddings, and progress prints every batch—so the DB counters move and the terminal talks even while the GPU is still embedding.

If anything still doesn’t print within ~10–20 seconds after starting, tell me and I’ll give you an even more aggressive “commit per 100 chunks” version, but the above is usually enough to make progress visible quickly.

Yeah—50 MB plain-text files are enormous for an embedding pipeline. Quick back-of-the-napkin:

• 50 MB ≈ ~50,000,000 chars.
• With our chunking (1500 chars, 200 overlap → effective step ~1300), that’s ~38,000 chunks per file.
• Each chunk → 768-dim vector, so per file that’s ~38k forward passes + DB updates. Even on a 3060 this can take many minutes per file. With 15,325 files, it’ll look “stuck” because the script only commits/prints after finishing the first monster.

Here’s how to get visible progress right now without touching any code:

1) Run unbuffered so you actually see prints immediately

export ROOT_DIR=/media/darren/storage/pile_uncopyrighted
export PGHOST=/var/run/postgresql
export PGPORT=5433
export SENTENCE_TRANSFORMERS_DEFAULT_DEVICE=cuda
export SENTENCE_TRANSFORMERS_TRUST_REMOTE_CODE=true

python3 -u ~/Documents/tokenspace/ingest_pile.py

-u forces Python to flush output promptly.

2) Prove the pipeline with a tiny subset (no code edits)

mkdir -p /tmp/ingest_subset
find /media/darren/storage/pile_uncopyrighted -type f -name '*.txt' | head -n 3 | xargs -I{} cp "{}" /tmp/ingest_subset/

ROOT_DIR=/tmp/ingest_subset \
PGHOST=/var/run/postgresql PGPORT=5433 \
SENTENCE_TRANSFORMERS_DEFAULT_DEVICE=cuda SENTENCE_TRANSFORMERS_TRUST_REMOTE_CODE=true \
python3 -u ~/Documents/tokenspace/ingest_pile.py

You should see [doc] … -> N chunks, then [ok] … -> N chunk(s) within a couple minutes.

3) If you want progress while processing giant files (still no code edits)

Lower chunk size actually creates more chunks (slower). Instead, keep chunk size, but reduce GPU batch size via an env the library respects:

export TOKENIZERS_PARALLELISM=false  # avoids tokenizer thread thrash

And watch DB counters in another window:

watch -n 2 'psql -h /var/run/postgresql -p 5433 -d tokenspace -c "select count(*) docs from content.documents; select count(*) chunks from content.chunks;"'

If these remain 0 during the full first file, that’s just because our current script commits once per document.

4) Practical operating pattern for 50 MB docs

• Stage by stage: run 3–10 files at a time (like the subset above), verify rows increase, then move the next batch in.
• Optional later optimization (I can patch for you if you want it baked-in):
• Commit chunk metadata first (so chunks counter climbs immediately), then update embeddings in batches with periodic commits and progress logs. This removes the “no movement until doc ends” effect on huge files.

---

TL;DR

50 MB text → ~38k chunks per file = long first commit. You’re not broken; it’s just huge input. Use python3 -u for live logs and process a tiny subset to confirm flow. If you want me to hard-patch the script to commit chunks first and print per-batch progress so the DB counters climb during the first file, say the word and I’ll drop in the updated code.