Ugoe Machine Ingestion Script — Part 2

It’s probably not hung. It’s likely chewing through a very large first file with almost no intermediate logging.

What your script does is:

1. look up the doc
2. count existing chunks
3. read and re-chunk the whole file anyway
4. upsert every chunk row anyway
5. only then decide whether to skip embedding
6. commit
7. print the next [ok ...] line

That behavior is right in the script: it logs the existing chunk count, then reads/chunks the file, then runs CHUNK_UPSERT_SQL for every chunk, and only after that does it decide whether to skip re-embedding. fileciteturn1file0L1-L18 fileciteturn1file1L59-L73

So this line:

[file 1] existing chunks for doc_id=2: 40974

means the first document already has 40,974 chunk rows associated with it. Since the script upserts chunk metadata one row at a time before it prints the next progress line, a long silent pause is expected. fileciteturn1file0L12-L18

Also, the script is explicitly designed as an “idempotent resume,” but in its current form that only skips embedding, not the metadata upsert work. fileciteturn1file3L3-L8

So your instinct is close: it is probably “seeing it already did some,” but then it still spends a long time reprocessing that file’s chunk rows.

Fast checks

In another terminal, watch whether Postgres is alive and the log is moving:

tail -f ~/Documents/tokenspace/ingest.log

and:

ps -fp $(pgrep -f ingest_pile_v2.py)

and optionally:

top

If Python is still burning CPU, it’s working, just slowly.

The real bottleneck

This loop is the expensive part:

• for every chunk in the file, run an INSERT ... ON CONFLICT ... DO UPDATE RETURNING chunk_id one row at a time. fileciteturn1file2L23-L30

With tens of thousands of chunks on file 1, that can look frozen.

Best fix

You probably want the script changed so that when:

• pre_existing_chunks > 0
• and FORCE_REEMBED=0

it skips the entire file immediately, instead of re-reading, re-chunking, and re-upserting everything.

In other words, add a fast path right after this line:

pre_existing_chunks = cur.fetchone()[0]

like this:

if pre_existing_chunks > 0 and not FORCE_REEMBED:
    log(f"[file {idx}] skipping file entirely (already ingested)")
    continue

That would make resume behavior actually fast.

What I’d do right now

Stop the run and patch the script, because otherwise it may spend ages re-upserting already-ingested content.

Use this edit in ingest_pile_v2.py immediately after:

pre_existing_chunks = cur.fetchone()[0]
log(f"[file {idx}] existing chunks for doc_id={doc_id}: {pre_existing_chunks}")

Add:

if pre_existing_chunks > 0 and not FORCE_REEMBED:
    log(f"[file {idx}] skipping file entirely (already ingested)")
    continue

That change is consistent with how the script already intends to resume safely, but avoids the expensive metadata rewrite. fileciteturn1file0L20-L24

One caution: this assumes the source files have not changed. If they did change and you need metadata/text refreshed, then skipping whole files would miss that.

If you want, I can give you the exact sed command to patch the file without opening an editor.