Good news first: the ingest is clearly running (you’re getting all those [ok] … -> N chunk(s) lines). The reason your counts still show 30 docs / 1,277,343 chunks is almost certainly that your watch/psql window is looking at a different session/target than the script is writing to, or you’re reprocessing the same 30 files (idempotent upserts won’t raise the totals).

Let’s prove exactly where the script is writing and line it up with what you’re watching — no guesswork.

1) Tag the running ingest connection so we can see it

Run your ingest with an app-name tag (no code changes):

PGAPPNAME=ingest_pile \
SENTENCE_TRANSFORMERS_DEFAULT_DEVICE=cuda SENTENCE_TRANSFORMERS_TRUST_REMOTE_CODE=true \
PGHOST=/var/run/postgresql PGPORT=5433 \
python3 -u ~/Documents/tokenspace/ingest_pile.py

2) In another terminal: find that exact DB session

psql -h /var/run/postgresql -p 5433 -d tokenspace -c \
"SELECT pid, usename, application_name, backend_start, client_addr,
        current_setting('port') AS port,
        current_database() AS db,
        (SELECT setting FROM pg_settings WHERE name='data_directory') AS data_dir
   FROM pg_stat_activity
  WHERE application_name='ingest_pile';"

• You should see a row for ingest_pile. Note the port, db, and data_dir.

3) Make your counter query hit that same place (bullet-proof)

Use the same socket+port and show a timestamp so you know it’s live:

watch -n 2 'echo -n "$(date +"%H:%M:%S")  "; psql -h /var/run/postgresql -p 5433 -d tokenspace -Atc "select (select count(*) from content.documents)::text || E\"\t\" || (select count(*) from content.chunks)"'

Also confirm you’re on the same data directory:

psql -h /var/run/postgresql -p 5433 -d tokenspace -Atc "show data_directory"

Compare this data_directory to the one you saw for ingest_pile in step 2 — they must match.

4) Quick sanity on duplicates vs new files

If the DB totals aren’t moving, you may be re-ingesting the same 30 files (by design it upserts and won’t increase counts):

# how many distinct documents by path?
psql -h /var/run/postgresql -p 5433 -d tokenspace -Atc \
"select count(distinct external_id) from content.documents"

# show the last few docs actually written
psql -h /var/run/postgresql -p 5433 -d tokenspace -c \
"select doc_id, title, created_at from content.documents order by created_at desc limit 10;"

If the last created_at is old while the script is printing [ok] for more files, that means your counter window wasn’t watching the same place as the script (fix with steps 1–3). If created_at is advancing but totals stay flat, you’re likely replacing the same docs (same external_id) and the chunk counts per doc are steady due to upserts — that’s expected behavior.

5) (Optional) Eliminate any ambiguity forever

Export the exact DSN the script uses before running, so both psql and the script are guaranteed to hit the same cluster/db:

export DB_DSN='postgresql:///tokenspace?host=/var/run/postgresql&port=5433'
# run psql against it once:
psql "$DB_DSN" -Atc "select current_user, current_database(), show_config_by_name('data_directory');"
# then run the ingest with the same env:
PGAPPNAME=ingest_pile python3 -u ~/Documents/tokenspace/ingest_pile.py

If you paste the output of step 2 (the pg_stat_activity row for ingest_pile) plus the show data_directory from your watch window, I’ll sanity-check and tell you in one line if they’re the same target. Once those match, your live counter will rise in lockstep with the [ok] lines.

The traceback is just saying: your ingest script needs NumPy in the venv, and it isn’t there. That’s a simple fix.

Inside your active venv ((ougway_venv) should be in your prompt):

pip install numpy

Verify it worked:

python3 - <<'PY'
import numpy as np
print("numpy ok:", np.__version__)
PY

Once that succeeds, rerun your ingest with the PGAPPNAME=ingest_pile so we can tag the session:

PGAPPNAME=ingest_pile \
SENTENCE_TRANSFORMERS_DEFAULT_DEVICE=cuda SENTENCE_TRANSFORMERS_TRUST_REMOTE_CODE=true \
PGHOST=/var/run/postgresql PGPORT=5433 \
python3 -u ~/Documents/tokenspace/ingest_pile.py

Then in another terminal, confirm the session shows up:

psql -h /var/run/postgresql -p 5433 -d tokenspace -c \
"SELECT pid, usename, application_name, backend_start, client_addr
   FROM pg_stat_activity
  WHERE application_name='ingest_pile';"

That way we’ll know for sure which DB session your ingest is writing to, and you can watch its document/chunk counts in real time.

Do you want me to also give you the schema and queries for the “election program” you mentioned earlier (like a Postgres schema for races/candidates/votes), or were you asking about actual voter info for a specific election?

You’re right: the watch window doesn’t care about your virtualenv. psql talks to Postgres directly; the venv only matters for Python packages. So if the counts still show 30 / 1,277,343, there are only a few possibilities:

1) You’re counting a different target than the script is writing to (different DB/cluster/port/socket).
2) The restart didn’t actually run (e.g., crashed early—like the NumPy missing error).
3) You’re reprocessing the same 30 files (our upserts overwrite, so totals don’t rise).
4) You’re between commits on a giant doc (no new rows until that doc finishes).

Let’s nail it down with zero ambiguity and force both psql and the script to hit the same DSN, then tag the session so we can see it writing.

Do this exact sequence

# 1) Canonical DSN for your socket-only cluster
export DB_DSN='postgresql:///tokenspace?host=/var/run/postgresql&port=5433'

# 2) Prove psql is hitting that DSN (and show the data dir)
psql "$DB_DSN" -Atc "select current_database(), current_user, (select setting from pg_settings where name='data_directory')"

# 3) Start the ingest with a session tag so we can see it in pg_stat_activity
PGAPPNAME=ingest_pile \
SENTENCE_TRANSFORMERS_DEFAULT_DEVICE=cuda SENTENCE_TRANSFORMERS_TRUST_REMOTE_CODE=true \
DB_DSN="$DB_DSN" \
python3 -u ~/Documents/tokenspace/ingest_pile.py

Open a second terminal (any shell; venv not required):

# 4) Confirm the ingest session exists and is on the SAME data directory
psql "$DB_DSN" -c "
SELECT pid, usename, application_name, state, backend_start,
       (SELECT setting FROM pg_settings WHERE name='data_directory') AS data_dir
FROM pg_stat_activity
WHERE application_name='ingest_pile';"

# 5) Live counters against the SAME DSN
watch -n 2 'echo -n "$(date +\"%H:%M:%S\")  "; psql "'"$DB_DSN"'" -Atc "select (select count(*) from content.documents)::text || E\"\t\" || (select count(*) from content.chunks)"'

Quick sanity checks

• Did the ingest print any [ok] /path -> N chunk(s) after you installed NumPy?
  If the run died early (e.g., before NumPy was installed), totals would stay at 30.

• Are those [ok] file paths new or the same 30?
  We guard with UNIQUE (external_id) and UNIQUE (doc_id, seq), so reprocessing the same files won’t raise totals.

• If you want to verify specific files actually landed, check one you saw in the log:

# replace the path with one of the [ok] file paths you saw
psql "$DB_DSN" -c "SELECT doc_id, title, external_id, created_at FROM content.documents WHERE external_id = '/media/darren/storage/pile_uncopyrighted/chunk_10568.txt';"

psql "$DB_DSN" -c "
SELECT count(*) FROM content.chunks c
JOIN content.documents d USING (doc_id)
WHERE d.external_id = '/media/darren/storage/pile_uncopyrighted/chunk_10568.txt';"

If those queries return a row and a large chunk count, you’re definitely looking at the right DB and the numbers are moving—maybe just slowly due to big docs and commit timing. If they return nothing, we were looking at the wrong target before steps 1–5.

If anything in the steps above doesn’t match or prints something odd (e.g., different data_directory values), paste that output and I’ll tell you exactly what to change.

You’re right to be suspicious of that “stuck at 30 / 1.27M” readout. Here’s the clean way to settle it—with proof about what’s being written, and a quick fix for the NUL crash so it doesn’t interrupt commits again.

What’s likely going on

• When the job hit the NUL‐byte error, your script exited. If those 30 docs were the only ones committed before the crash, totals won’t rise until a clean run gets past the bad file.
• If the restart crashed early (e.g., missing numpy earlier) or hit the same bad file again quickly, you’d see a few [ok] lines from earlier output, but no new commits in the DB.
• We need to verify that a specific [ok] file actually exists in content.documents right now. If it doesn’t, those lines aren’t making it to the DB before the crash.

Step 1 — Confirm the sanitizer is actually in your script

We added a clean_text() to strip \x00, but the first patch had an indentation hiccup. Let’s make sure the fixed version is really in place and being called.

# Show the two lines we need to see
grep -n "def clean_text" -n ~/Documents/tokenspace/ingest_pile.py
grep -n "text = clean_text(read_text_file(fpath))" ~/Documents/tokenspace/ingest_pile.py

You should see:
– one line defining def clean_text(...):
– one line where we call clean_text(read_text_file(fpath))

If either grep prints nothing, tell me and I’ll give you a one-liner to re-insert it cleanly.

Step 2 — Tag the DB session and run (so we can see it in Postgres)

export DB_DSN='postgresql:///tokenspace?host=/var/run/postgresql&port=5433'
PGAPPNAME=ingest_pile \
SENTENCE_TRANSFORMERS_DEFAULT_DEVICE=cuda SENTENCE_TRANSFORMERS_TRUST_REMOTE_CODE=true \
DB_DSN="$DB_DSN" \
python3 -u ~/Documents/tokenspace/ingest_pile.py

Step 3 — In another terminal, prove we’re looking at the same DB and that the session exists

# Show the app session and the data dir
psql "$DB_DSN" -c "SELECT pid, usename, application_name, state,
  (SELECT setting FROM pg_settings WHERE name='data_directory') AS data_dir
  FROM pg_stat_activity
  WHERE application_name='ingest_pile';"

# Show the data_dir you are querying in this terminal too:
psql "$DB_DSN" -Atc "show data_directory"

The data_dir values should match.

Step 4 — Check one concrete file that the log says was [ok]

Pick any one from your paste, e.g. chunk_10568.txt:

psql "$DB_DSN" -c "
SELECT doc_id, title, external_id, created_at
FROM content.documents
WHERE external_id = '/media/darren/storage/pile_uncopyrighted/chunk_10568.txt';"

psql "$DB_DSN" -c "
SELECT count(*)
FROM content.chunks c
JOIN content.documents d USING(doc_id)
WHERE d.external_id = '/media/darren/storage/pile_uncopyrighted/chunk_10568.txt';"

• If this returns a row and a big count, then we are updating the same 30 docs repeatedly (idempotent upsert), or your counter is looking at the wrong cluster.
• If this returns nothing, then those [ok] lines never committed (job is crashing before conn.commit()), so we need to stop the crash (NULs) and/or add an earlier commit.

Step 5 — Make the run robust against NULs (so it won’t crash mid-doc)

If Step 1 showed both greps OK, you already have the sanitizer. If not, here’s the clean re-insert:

# backup
cp ~/Documents/tokenspace/ingest_pile.py ~/Documents/tokenspace/ingest_pile.py.bak

python3 - <<'PY'
from pathlib import Path
p = Path.home()/ "Documents/tokenspace/ingest_pile.py"
s = p.read_text()
if "def clean_text(" not in s:
    anchor = "import numpy as np"
    inject = (
        "\n\ndef clean_text(s: str) -> str:\n"
        "    \"\"\"Drop NULs and control chars (keep \\n, \\t).\"\"\"\n"
        "    if '\\x00' in s:\n"
        "        s = s.replace('\\x00', ' ')\n"
        "    return ''.join(ch if (ch in ('\\n','\\t') or ord(ch) >= 32) else ' ' for ch in s)\n"
    )
    s = s.replace(anchor, anchor + inject, 1) if anchor in s else inject + s
s = s.replace(\"text = read_text_file(fpath)\", \"text = clean_text(read_text_file(fpath))\")
p.write_text(s)
print(\"sanitizer inserted & wired\")
PY

Step 6 — (Optional) Commit chunk metadata before embeddings

This guarantees your counters rise even if a later batch errors:

# add an early commit right after inserting all chunk rows, before embedding
# (backup first)
cp ~/Documents/tokenspace/ingest_pile.py ~/Documents/tokenspace/ingest_pile.py.precommit.bak

python3 - <<'PY'
from pathlib import Path
p = Path.home()/ "Documents/tokenspace/ingest_pile.py"
s = p.read_text()
needle = "            # Embed in batches to keep GPU/CPU happy"
if needle in s and "commit chunk rows" not in s:
    s = s.replace(needle,
        "            # commit chunk rows so DB shows progress immediately\\n"
        "            conn.commit()\\n"
        "            print(f\"[commit] wrote {len(chunk_ids)} chunk rows for {external_id} (no embeddings yet)\")\\n\\n"
        + needle, 1)
p.write_text(s)
print("early commit inserted")
PY

Now even if something later fails, you’ll still see chunk rows land and the counters go up.

Step 7 — Bullet-proof watcher (timestamped, same DSN)

watch -n 2 'echo -n "$(date +\"%H:%M:%S\")  "; psql "'"$DB_DSN"'" -Atc "select (select count(*) from content.documents)::text || E\"\t\" || (select count(*) from content.chunks)"'

Why I think your totals stayed at 30

Because the process crashed on the NUL error before getting past the next big file, and your earlier [ok] lines came from the run before that crash that actually committed the first 30. With the sanitizer + (optionally) early commit, you’ll see numbers rise again and keep rising even if another dirty file shows up.

If any command above returns something surprising (e.g., the single-file check returns no row), paste that output and I’ll pinpoint the exact reason in one line.

Totally fair. Fresh reboot + clean verification is the right move. Here’s a tight, no-changes checklist you can run end-to-end when you’re back. It doesn’t modify code or configs—just proves where data is going and that your counters reflect it.

Before reboot (optional clean stop)

# stop any running ingest
pgrep -a -f ingest_pile.py
# then in each ingest terminal: Ctrl-C
# or force-stop if needed:
pkill -f ingest_pile.py || true

Reboot

sudo reboot

After reboot — minimal verification (read-only)

1) Confirm Postgres is the socket-only cluster on 5433

sudo -u postgres psql -p 5433 -c "SHOW port; SHOW unix_socket_directories; SHOW data_directory;"
sudo ss -ltnp | grep postgres || echo "No TCP listeners — OK"

2) Baseline counts (what the DB has right now)

psql -h /var/run/postgresql -p 5433 -d tokenspace -Atc "select count(*) from content.documents"
psql -h /var/run/postgresql -p 5433 -d tokenspace -Atc "select count(*) from content.chunks"

3) Start a solid watcher (timestamped)

watch -n 2 'echo -n "$(date +\"%H:%M:%S\")  "; psql -h /var/run/postgresql -p 5433 -d tokenspace -Atc "select (select count(*) from content.documents)::text || E\"\t\" || (select count(*) from content.chunks)"'

4) Launch ingest (no code edits), tag the session so we can see it

Open a new terminal for the ingest:

# venv if needed
source ~/Documents/ougway_venv/bin/activate

export ROOT_DIR=/media/darren/storage/pile_uncopyrighted
export PGHOST=/var/run/postgresql
export PGPORT=5433
export SENTENCE_TRANSFORMERS_DEFAULT_DEVICE=cuda
export SENTENCE_TRANSFORMERS_TRUST_REMOTE_CODE=true

# tag the DB session so we can find it; capture logs to a file too
PGAPPNAME=ingest_pile python3 -u ~/Documents/tokenspace/ingest_pile.py | tee -a ~/ingest_run.log

5) In a third terminal, prove the ingest session you just started is connected where you’re watching

psql -h /var/run/postgresql -p 5433 -d tokenspace -c "
SELECT pid, usename, application_name, state, query_start,
       (SELECT setting FROM pg_settings WHERE name='data_directory') AS data_dir
FROM pg_stat_activity
WHERE application_name='ingest_pile';"

• You should see application_name = ingest_pile.
• The data_dir here should match the SHOW data_directory; from step 1.

6) Spot-check a file the script prints as [ok]

Pick one path the script prints (e.g., /media/darren/storage/pile_uncopyrighted/chunk_10568.txt) and check it landed:

psql -h /var/run/postgresql -p 5433 -d tokenspace -c "
SELECT doc_id, title, external_id, created_at
FROM content.documents
WHERE external_id = '/media/darren/storage/pile_uncopyrighted/chunk_10568.txt';"

psql -h /var/run/postgresql -p 5433 -d tokenspace -c "
SELECT count(*)
FROM content.chunks c
JOIN content.documents d USING(doc_id)
WHERE d.external_id = '/media/darren/storage/pile_uncopyrighted/chunk_10568.txt';"

If that returns a row and a big count, you’re watching the right DB and the numbers will climb as commits happen. If it returns nothing while the ingest shows [ok] for that same file, paste that result and we’ll zero in immediately (that would indicate a crash/rollback before commit).

7) If a NUL-byte crash reappears (same error string)

Don’t change code yet—just note the exact file it died on (last [ok] line and the next file it started), and we’ll handle that specific path after you’re back.

Sleep is a feature, not a bug. When you return, run steps 1–6 exactly; they’ll tell us, with receipts, whether the counters and the writer are in sync. If anything’s off, share the outputs and I’ll give you the precise next command.

Got it — that error is on me. The “double pipe” operator || in Postgres concatenates text, but I wrapped it in a way that tripped up watch because of quoting. Let’s simplify it so you don’t have to fight with the shell parser.

Here are three safer versions you can pick from:

Option A — two separate counts (super clear)

watch -n 2 'psql -h /var/run/postgresql -p 5433 -d tokenspace -c "select count(*) as docs from content.documents; select count(*) as chunks from content.chunks;"'

This just prints two lines every 2 seconds: one for docs, one for chunks.

Option B — one row with both numbers

watch -n 2 'psql -h /var/run/postgresql -p 5433 -d tokenspace -c "select (select count(*) from content.documents) as docs, (select count(*) from content.chunks) as chunks;"'

Option C — add a timestamp for context

watch -n 2 'echo -n "$(date +%H:%M:%S)  "; psql -h /var/run/postgresql -p 5433 -d tokenspace -Atc "select (select count(*) from content.documents), (select count(*) from content.chunks)"'

💡 Tip: Option B is probably the cleanest, since you get both counts in one line. Option C is nice if you want to see when the counts were sampled.

Do you want me to also add a quick query you can run to show the last few docs (title + timestamp), so you can confirm new rows are landing without relying only on the totals?

Exactly right 👍 — start clean, venv active, no code edits, and tag the session so we can see what’s really happening. That way we’ll know whether it’s committing new docs or just spinning.

Here’s the step-by-step for the rerun:

1. Activate your venv

source ~/Documents/ougway_venv/bin/activate

2. Set environment

export ROOT_DIR=/media/darren/storage/pile_uncopyrighted
export PGHOST=/var/run/postgresql
export PGPORT=5433
export SENTENCE_TRANSFORMERS_DEFAULT_DEVICE=cuda
export SENTENCE_TRANSFORMERS_TRUST_REMOTE_CODE=true

3. Launch ingest with session tag + log to file

PGAPPNAME=ingest_pile \
python3 -u ~/Documents/tokenspace/ingest_pile.py | tee -a ~/ingest_run.log

(-u = unbuffered, so you see logs live. tee saves them too.)

4. In another terminal, confirm the session exists

psql -h /var/run/postgresql -p 5433 -d tokenspace -c "
SELECT pid, usename, application_name, state, query_start,
       (SELECT setting FROM pg_settings WHERE name='data_directory') AS data_dir
FROM pg_stat_activity
WHERE application_name='ingest_pile';"

You should see one row with application_name = ingest_pile.

5. Watch counters again

watch -n 2 'psql -h /var/run/postgresql -p 5433 -d tokenspace -c "select count(*) as docs from content.documents; select count(*) as chunks from content.chunks;"'

6. Spot-check a file the log says [ok]

Pick one of the new [ok] lines, e.g. chunk_9238.txt, and check it:

psql -h /var/run/postgresql -p 5433 -d tokenspace -c "
SELECT doc_id, external_id, created_at
FROM content.documents
WHERE external_id = '/media/darren/storage/pile_uncopyrighted/chunk_9238.txt';"

🔍 If that query comes back empty while the ingest printed [ok], then we know the process is still dying/rolling back before commit(). If it returns a row + timestamp, then the doc is landing and your counters should rise past 30.

Do you want me to also give you a one-liner to check how many distinct external_id paths are in the DB? That way we can confirm it isn’t just overwriting the same 30 docs.

You’re right to double-check paths. From earlier, your 3.6 TB drive sdb was mounted at:

/media/darren/storage

…and that’s the path that worked when you saw “found 15,325 file(s).”

Right now your run shows 0 files, which usually means the drive isn’t mounted after the reboot (or we pointed to the wrong dir).

Quick path + mount checks

Run these exactly:

# See where the disks are mounted
lsblk -o NAME,MOUNTPOINT,SIZE,FSTYPE | sed -n '1,200p'

# Confirm the expected directory has files
ls -la /media/darren/storage | head

# Does the target folder exist and contain your chunks?
ls -la /media/darren/storage/pile_uncopyrighted | head

If /media/darren/storage is empty or not mounted:

# Try to (re)mount the 3.6T drive
udisksctl mount -b /dev/sdb

# Verify again
ls -la /media/darren/storage | head
ls -la /media/darren/storage/pile_uncopyrighted | head

(If udisksctl isn’t available or the block device is different, use lsblk’s output to pick the right /dev/sdX.)

Then set ROOT_DIR to the confirmed path and rerun

export ROOT_DIR=/media/darren/storage/pile_uncopyrighted
PGHOST=/var/run/postgresql PGPORT=5433 \
SENTENCE_TRANSFORMERS_DEFAULT_DEVICE=cuda SENTENCE_TRANSFORMERS_TRUST_REMOTE_CODE=true \
PGAPPNAME=ingest_pile \
python3 -u ~/Documents/tokenspace/ingest_pile.py | tee -a ~/ingest_run.log

If you truly have a different mount like /media/storage/... on this boot, just export ROOT_DIR=/media/storage/pile_uncopyrighted instead—use whichever ls shows with files.

Want me to add a tiny auto-detect in the script so it tries both /media/darren/storage/... and /media/storage/... (and /mnt/storage/...) and picks the first that exists? That would prevent this after reboots.

You’re good—the mount and path are exactly what we used before:

• Drive sdb is mounted at /media/darren/storage
• Your files are in /media/darren/storage/pile_uncopyrighted/
• Lots of chunk_0000.txt, chunk_0001.txt, … are present

So when the ingest showed “found 0 file(s)”, that wasn’t because the path didn’t exist. Let’s sanity-check the environment the script actually sees, then kick it off again.

Quick sanity (no changes)

# 1) What does the shell think ROOT_DIR is?
echo "$ROOT_DIR"

# 2) Do we see .txt files from the shell?
find "$ROOT_DIR" -type f -name '*.txt' | head
find "$ROOT_DIR" -type f -name '*.txt' | wc -l

# 3) Do we see .txt files using the same Python globbing as the script?
python3 - <<'PY'
import os, pathlib, sys
root = pathlib.Path(os.getenv('ROOT_DIR', '/mnt/storage/pile_uncopyrighted')).resolve()
files = [p for p in root.rglob('*') if p.is_file() and p.suffix.lower() in {'.txt','.md','.markdown','.log'}]
print("ROOT_DIR =", root)
print("count =", len(files))
print("sample =", [str(p) for p in files[:5]])
PY

If those counts are >0, the path/env is fine.

Run ingest again (same mount, clean env, tagged)

# venv on
source ~/Documents/ougway_venv/bin/activate

# env
export ROOT_DIR=/media/darren/storage/pile_uncopyrighted
export PGHOST=/var/run/postgresql
export PGPORT=5433
export SENTENCE_TRANSFORMERS_DEFAULT_DEVICE=cuda
export SENTENCE_TRANSFORMERS_TRUST_REMOTE_CODE=true

# run, tag session & log
PGAPPNAME=ingest_pile \
python3 -u ~/Documents/tokenspace/ingest_pile.py | tee -a ~/ingest_run.log

In another terminal, watch counts:

watch -n 2 'psql -h /var/run/postgresql -p 5433 -d tokenspace -c "select count(*) as docs from content.documents; select count(*) as chunks from content.chunks;"'

And confirm the session:

psql -h /var/run/postgresql -p 5433 -d tokenspace -c \
"SELECT pid, application_name, state,
        (SELECT setting FROM pg_settings WHERE name='data_directory') AS data_dir
   FROM pg_stat_activity
  WHERE application_name='ingest_pile';"

Why you see “darren” in the path

That’s normal for desktop Linux: the automounter places removable/internal volumes under /media/<username>/…. You’re not doing anything wrong; /media/darren/storage/... is the canonical mountpoint for that 3.6 TB drive (and we verified it’s mounted and full of your files).

If the script ever shows “found 0 file(s)” again, run the 3 checks at the top—especially the small Python snippet. It mimics the script’s file discovery exactly and will tell us immediately if it’s an env/path issue vs. something else.

Nice—this confirms the ingest process is connected and active:

pid | application_name | state
6616| ingest_pile      | active

A few quick notes + checks while it runs:

1) That blank data_dir

Some builds hide data_directory in a subselect. Just run a plain SHOW:

psql -h /var/run/postgresql -p 5433 -d tokenspace -c "SHOW data_directory;"

(You should see a full path; it’s just a sanity check.)

2) Counters won’t move until the first doc commits

Given your ~50 MB files (≈40k chunks each), the totals stay at 30/1.27M until doc #31 finishes and commits. That can take a while.

Keep the watcher running:

watch -n 2 'psql -h /var/run/postgresql -p 5433 -d tokenspace -c "select count(*) as docs from content.documents; select count(*) as chunks from content.chunks;"'

3) Verify a specific file landed (when you see the next [ok] … line)

As soon as the ingest prints an [ok] /media/darren/storage/pile_uncopyrighted/chunk_NNNN.txt -> … for a new file, run:

# replace the path with the one you just saw in the [ok] line
DOC="/media/darren/storage/pile_uncopyrighted/chunk_10568.txt"

psql -h /var/run/postgresql -p 5433 -d tokenspace -c "
SELECT doc_id, external_id, created_at
FROM content.documents
WHERE external_id = '${DOC}';"

psql -h /var/run/postgresql -p 5433 -d tokenspace -c "
SELECT count(*) AS chunks_for_doc
FROM content.chunks c
JOIN content.documents d USING (doc_id)
WHERE d.external_id = '${DOC}';"

If those return a row and a large chunk count, you’ve proved the commit happened—even if the global counters haven’t refreshed in your other terminal yet.

4) Optional: show last few docs while it runs

psql -h /var/run/postgresql -p 5433 -d tokenspace -c "
SELECT doc_id, title, created_at
FROM content.documents
ORDER BY created_at DESC
LIMIT 5;"

If you want, we can also make the ingest print per-batch progress or commit chunk metadata before embeddings so you’ll see DB movement sooner during each huge file—but you don’t need that to be correct. For now, letting it churn is fine.

Yep—that error is expected for non-superusers. data_directory is considered sensitive, so Postgres hides it unless your role has the pg_read_all_settings privilege (or you’re postgres).

You’ve got three easy options:

Option A — run the check as postgres (one-off)

sudo -u postgres psql -p 5433 -d tokenspace -c "SHOW data_directory;"

Option B — grant your user read-settings (persistent; safe)

Run once as postgres:

sudo -u postgres psql -p 5433 -d tokenspace -c "GRANT pg_read_all_settings TO darren;"

Then your original command will work:

psql -h /var/run/postgresql -p 5433 -d tokenspace -c "SHOW data_directory;"

Option C — skip data_directory and use non-privileged signals

You can still prove you’re on the right cluster without special roles:

• Current DB/user/socket/port:

psql -h /var/run/postgresql -p 5433 -d tokenspace -c "SELECT current_database(), current_user;"
psql -h /var/run/postgresql -p 5433 -d tokenspace -c "SHOW port; SHOW unix_socket_directories;"

• See your tagged ingest session:

psql -h /var/run/postgresql -p 5433 -d tokenspace -c \
"SELECT pid, usename, application_name, state, backend_start
 FROM pg_stat_activity
 WHERE application_name='ingest_pile';"

• Confirm new rows are landing (no special perms needed):

psql -h /var/run/postgresql -p 5433 -d tokenspace -c \
"SELECT doc_id, title, created_at
   FROM content.documents
  ORDER BY created_at DESC
  LIMIT 5;"

psql -h /var/run/postgresql -p 5433 -d tokenspace -c \
"SELECT (SELECT count(*) FROM content.documents) AS docs,
        (SELECT count(*) FROM content.chunks)    AS chunks;"

If you want to keep your role minimal, stick with Option C. If you’d like the full “SHOW anything” convenience, do Option B once.

Either way, the important part is that your ingest_pile session shows as active, and the documents/chunks queries will reveal when the next commit lands.

Nice — that confirms you’re hitting the intended cluster:

data_directory = /mnt/storage/tokenspace/pgdata

So we’re watching the right Postgres. Given your ~50 MB docs, it’s normal for the global counters to sit still until the next huge document finishes and commits.

Here are two tiny checks that’ll tell you, without doubt, when the next commit lands:

1) Live counters (simple)

watch -n 5 'psql -h /var/run/postgresql -p 5433 -d tokenspace -c "select count(*) as docs from content.documents; select count(*) as chunks from content.chunks;"'

2) “Recent docs” view (shows the latest row as soon as it’s committed)

psql -h /var/run/postgresql -p 5433 -d tokenspace -c \
"select doc_id, title, created_at
   from content.documents
  order by created_at desc
  limit 5;"

If you want to be extra sure your current run is still busy, you can also peek at the session:

psql -h /var/run/postgresql -p 5433 -d tokenspace -c \
"select pid, application_name, state, query_start
   from pg_stat_activity
  where application_name='ingest_pile';"

• state = active for a long stretch during a giant file is expected.
• As soon as the file completes, you’ll see the doc count jump to 31 and a fresh row in the “recent docs” query.

If you ever want the ingest to show intermediate progress (e.g., per 1k chunks) or to commit chunk metadata before embeddings so the chunk count rises during a giant file, say the word and I’ll give you a tiny, safe patch.