Let’s calculate it carefully:

• Files: 15 325
• Average time per file: 55 seconds

Now multiply:

15,325 × 55 = 842,875 seconds

Convert:

• Seconds → minutes:
  842,875 ÷ 60 = 14,048 minutes
• Minutes → hours:
  14,048 ÷ 60 = 234 hours
• Hours → days:
  234 ÷ 24 = 9.75 days

✅ So at ~55 s per file, the full run would take about 9 ¾ days (call it ~10 days).

Of course, that’s assuming every file really is ~55 s and there’s no slowdown. If the later files are smaller or faster, it could be shorter.

Do you want me to also show you a simple SQL query you can run during the ingest to project remaining time dynamically? That way you won’t have to guess.

Perfect 👍 Glad that helps. Rule of thumb:

• If times hold steady, think ~10 days.
• If it averages a bit faster (say 45 s/file), you’d shave it down closer to 8 days.
• And because commits are every 2 000 chunks, you won’t lose progress if you stop it mid-way — you’ll just resume with what’s left.

So, no harm letting it run, and you’ll come out the other side with a very well-seeded datastore.

Anytime 🙂 Good luck with the ingest run — sounds like you’ve got it dialed in now. If you hit anything odd while it’s chugging away, just holler.

You’re 100% right to lock this down. Here’s a clean, non-negotiable plan:

• Always use SentenceTransformers.
• Always use Nomic v1.5 (768-dim).
• No hash fallback. If ST isn’t available, the script exits.
• We’ll redo from 0315 using ST and keep going—without leaving any hash vectors behind.
• We’ll add a --redo switch so anything at/after your start point is reinserted with ST (deletes the old row for that filename before insert).

A. Replace the ingester with an ST-only, Nomic-only build

> This script refuses to run unless it can load Nomic v1.5 (768-dim).
> It also supports --redo to delete+reinsert each file starting at your chosen point.

Create/overwrite the file:

cat > ~/Desktop/ingestpile.py <<'PY'
#!/usr/bin/env python3
"""
ST-only, NOMIC-only ingester with safe resume and *redo* support.
- Model: nomic-ai/nomic-embed-text-v1.5 (768-d). No fallback allowed.
- Each file => one chunk, stored with meta->>'filename' for dedupe/resume.
- Resume chooses the doc for the folder with **most** chunks.
- --start-index / --start-name pick where to begin.
- --redo: for each file at/after start, DELETE existing row with that filename (if any) before inserting ST embedding.
- Per-file SAVEPOINT; commits periodically; strips NULs.
"""
import os, sys, time, re, argparse, pathlib, traceback
from typing import List, Tuple, Set, Dict
import numpy as np
import psycopg2

# Make tokenizers quiet / predictable
os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")

# ===== Config =====
DSN = os.environ.get("TS_DSN", "dbname=tokenspace user=darren password=LittleOugway host=localhost port=5432")
EMBED_MODEL = "nomic-ai/nomic-embed-text-v1.5"  # hard-locked to Nomic
COMMIT_EVERY = int(os.environ.get("COMMIT_EVERY", "2000"))

# ===== SentenceTransformers REQUIRED =====
try:
    from sentence_transformers import SentenceTransformer
    ST = SentenceTransformer(EMBED_MODEL, trust_remote_code=True)
    dim = getattr(ST, "get_sentence_embedding_dimension", lambda: None)()
    if dim != 768:
        print(f"ERROR: {EMBED_MODEL} reports dim={dim}; need 768.", file=sys.stderr); sys.exit(3)
except Exception as e:
    print(f"ERROR: SentenceTransformers required and not available: {e}", file=sys.stderr); sys.exit(3)

def embed_texts(texts: List[str]) -> np.ndarray:
    arr = ST.encode(texts, normalize_embeddings=True)
    return np.asarray(arr, dtype=np.float32)

def vec_literal(v: np.ndarray) -> str:
    return "[" + ",".join(f"{x:.6f}" for x in v.tolist()) + "]"

def is_chunk_name(name: str) -> Tuple[bool, int]:
    s = name.lower()
    m = re.fullmatch(r"(?:chunk_)?(\d{1,8})\.txt", s)
    if m: return True, int(m.group(1))
    m = re.fullmatch(r"chunk_(\d{1,8})", s)
    if m: return True, int(m.group(1))
    return False, -1

def load_text(p: pathlib.Path) -> str:
    return p.read_text(encoding="utf-8", errors="ignore")

def ingest_directory(dir_path: pathlib.Path, start_index: int | None, start_name: str | None, limit: int | None, force_doc_id: int | None, redo: bool):
    # Collect files, sorted numerically by chunk index if possible
    all_files = [p for p in dir_path.iterdir() if p.is_file()]
    filt = []
    for p in all_files:
        ok, idx = is_chunk_name(p.name)
        if ok: filt.append((idx, p))
    if not filt:
        filt = [(i, p) for i, p in enumerate(sorted([p for p in all_files if p.suffix.lower()==".txt"], key=lambda x: x.name))]
    filt.sort(key=lambda t: (t[0], t[1].name))
    files = [p for _, p in filt]
    name_to_index: Dict[str,int] = {p.name:i for i,p in enumerate(files)}
    total = len(files)
    print(f"[dir] {dir_path} | files detected: {total}")
    if total == 0: return

    # Determine start
    user_start_idx = None
    if start_index is not None:
        user_start_idx = max(0, int(start_index))
    elif start_name is not None:
        if start_name in name_to_index:
            user_start_idx = name_to_index[start_name]
        else:
            print(f"[warn] start-name '{start_name}' not found; ignoring manual start.")

    conn = psycopg2.connect(DSN); conn.autocommit = False
    cur = conn.cursor()
    try:
        title = dir_path.name or str(dir_path)
        if force_doc_id is not None:
            doc_id = int(force_doc_id); print(f"Forcing document: doc_id={doc_id}")
        else:
            cur.execute("""
                SELECT d.doc_id
                FROM content.documents d
                JOIN content.sources s ON s.source_id = d.source_id
                LEFT JOIN content.chunks c ON c.doc_id = d.doc_id
                WHERE s.uri = %s AND d.title = %s
                GROUP BY d.doc_id
                ORDER BY COUNT(c.*) DESC, d.authored_at DESC
                LIMIT 1;
            """, (str(dir_path), title))
            row = cur.fetchone()
            if row:
                doc_id = row[0]; print(f"Reusing existing document: doc_id={doc_id}")
            else:
                cur.execute("INSERT INTO content.sources(kind, uri, meta) VALUES('file', %s, '{}'::jsonb) RETURNING source_id;", (str(dir_path),))
                source_id = cur.fetchone()[0]
                cur.execute("INSERT INTO content.documents(source_id, external_id, title, authored_at, meta) VALUES(%s, %s, %s, now(), '{}'::jsonb) RETURNING doc_id;", (source_id, None, title))
                doc_id = cur.fetchone()[0]
        conn.commit()

        # Resume info
        cur.execute("SELECT COALESCE(MAX(seq)+1, 0) FROM content.chunks WHERE doc_id=%s;", (doc_id,))
        next_seq = cur.fetchone()[0] or 0
        cur.execute("SELECT meta->>'filename' FROM content.chunks WHERE doc_id=%s AND meta ? 'filename';", (doc_id,))
        done_names: Set[str] = set(r[0] for r in cur.fetchall() if r and r[0])

        # Start point: if redo is True and user specified a start, respect user's start
        if user_start_idx is not None and redo:
            start_from = user_start_idx
        else:
            start_from = max(next_seq, user_start_idx or 0)

        print(f"[lock] model={EMBED_MODEL}, ST-only=ON")
        print(f"[resume] db_next_seq={next_seq}, already_by_name={len(done_names)}, start_from_index={start_from}, redo={redo}, limit={limit}", flush=True)

        started = time.time()
        inserted_now = 0
        seq = next_seq

        for i, fp in enumerate(files):
            if i < start_from:
                continue
            if limit is not None and inserted_now >= limit:
                print(f"[limit] reached {limit} files; stopping.")
                break

            fname = fp.name
            cur.execute("SAVEPOINT sp_file")
            t0 = time.time()
            try:
                # If redo requested, delete any existing row for this filename first
                if redo:
                    cur.execute("DELETE FROM content.chunks WHERE doc_id=%s AND meta->>'filename' = %s;", (doc_id, fname))
                    if fname in done_names:
                        done_names.remove(fname)

                # If not redo and we already have it, skip
                if not redo and fname in done_names:
                    print(f"  skip (seen): {i}/{total}: {fname}")
                    cur.execute("RELEASE SAVEPOINT sp_file")
                    continue

                # Load + sanitize text
                txt = load_text(fp)
                if "\x00" in txt: txt = txt.replace("\x00","")
                if not txt.strip():
                    print(f"  skip empty: {fname}")
                    cur.execute("RELEASE SAVEPOINT sp_file")
                    continue

                # Embed (ST only) and insert
                vec = embed_texts([txt])[0]
                cur.execute("""
                    INSERT INTO content.chunks (doc_id, seq, text, token_count, embedding, lang, tags, meta)
                    VALUES (%s, %s, %s, %s, %s::vector, 'en', ARRAY['pile'], jsonb_build_object('filename', %s));
                """, (doc_id, seq, txt, len(txt.split()), vec_literal(vec), fname))
                seq += 1; inserted_now += 1

                if inserted_now % COMMIT_EVERY == 0:
                    conn.commit(); print(f"  committed {inserted_now} chunks in {time.time()-started:.1f}s")

                cur.execute("RELEASE SAVEPOINT sp_file")
                print(f"  processed {i}/{total}: {fname} in {time.time()-t0:.2f}s", flush=True)
            except Exception as e:
                cur.execute("ROLLBACK TO SAVEPOINT sp_file")
                print(f"  ERROR on {fname}: {e}")
                traceback.print_exc(limit=1)

        conn.commit()
        print(f"Done: doc_id={doc_id}, newly inserted={inserted_now}, elapsed={time.time()-started:.1f}s")
    finally:
        cur.close(); conn.close()

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("path", nargs="?", default="/mnt/storage/pile_uncopyrighted")
    ap.add_argument("--start-index", type=int)
    ap.add_argument("--start-name", type=str)
    ap.add_argument("--limit", type=int)
    ap.add_argument("--doc-id", type=int)
    ap.add_argument("--redo", action="store_true", help="delete+reinsert files at/after start")
    args = ap.parse_args()

    path = pathlib.Path(args.path).expanduser().resolve()
    if not path.exists():
        print(f"Path not found: {path}"); sys.exit(2)
    if path.is_file():
        print("Single-file mode disabled in ST-only build."); sys.exit(2)

    ingest_directory(path, start_index=args.start_index, start_name=args.start_name, limit=args.limit, force_doc_id=args.doc_id, redo=args.redo)

if __name__ == "__main__":
    main()
PY
chmod +x ~/Desktop/ingestpile.py

B. Run it from 0315 with Nomic ST only, and redo anything from there

> This will delete & reinsert each file at/after 0315, guaranteeing no hash leftovers.

# (Optional but recommended for stability on desktop boxes)
export OMP_NUM_THREADS=4 MKL_NUM_THREADS=4 TOKENIZERS_PARALLELISM=false
# If your GPU driver is flaky, force CPU for stability:
# export CUDA_VISIBLE_DEVICES=""

# Start from 0315, redo everything from there with ST, append to your doc_id (change if not 5)
python ~/Desktop/ingestpile.py --doc-id 5 --start-name chunk_0315.txt --redo
# Or by index:
# python ~/Desktop/ingestpile.py --doc-id 5 --start-index 315 --redo

• --redo ensures each file at/after 0315 is deleted and reinserted with Nomic ST before moving on.
• No hash path exists in this script. If ST can’t load Nomic v1.5 (768-dim), it exits with an error.

C. Optional hard guard against dupes (do later, when calm)

After it’s running smoothly, consider a unique index so we never get duplicate filenames per document:

-- run in psql; 'CONCURRENTLY' keeps the table readable while building
CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS uniq_doc_filename
ON content.chunks (doc_id, (meta->>'filename'));

I hear you on the database pain—this locks the process into one path so we don’t drift again. If anything hangs the box (GPU drivers can be… “fun”), set CUDA_VISIBLE_DEVICES="" to stay on CPU; it’s slower but rock-solid.

Totally—on Ubuntu (and most Debian-based distros), use these:

See what you have now

nvidia-smi           # shows current driver (e.g., 535.xx)

List available drivers (recommended way)

# if this isn’t installed:
sudo apt-get update
sudo apt-get install -y ubuntu-drivers-common

# list/scan GPU and show all driver candidates + the recommended one
ubuntu-drivers devices

Also handy

# list every nvidia-driver package in your repos (550, 535, etc.)
apt-cache search '^nvidia-driver-[0-9]+$'

# check the exact candidate version for a specific series
apt-cache policy nvidia-driver-550
apt-cache policy nvidia-driver-535

If nvidia-driver-550 shows up and looks good, the upgrade flow is:

sudo apt-get update
sudo apt-get install -y nvidia-driver-550
sudo reboot

After reboot:

nvidia-smi   # should report 550.xx

> Notes:
> • If nvidia-driver-550 doesn’t appear, you can add the official graphics PPA, then retry:
> bash
> sudo add-apt-repository -y ppa:graphics-drivers/ppa
> sudo apt-get update
> ubuntu-drivers devices
>
> • If Secure Boot is ON, Ubuntu may prompt you to enroll a MOK during install—follow the on-screen steps and complete it on reboot.
> • If you’ve pinned CUDA packages, double-check compatibility before jumping driver major versions.

That error means the kernel module and the userspace NVML library are different versions (half-upgraded driver or an old module still loaded). Here’s a tight, safe path to see options and then upgrade to 550 cleanly.

1) List available drivers

# make sure helper is installed
sudo apt-get update
sudo apt-get install -y ubuntu-drivers-common

# show all candidates and the recommended one
ubuntu-drivers devices

# (optional) see all driver packages available in your repos
apt-cache search '^nvidia-driver-[0-9]\+$'
apt-cache policy nvidia-driver-550

2) Quick fix sometimes works: reboot to sync module + libs

sudo reboot

After reboot:

nvidia-smi

If it still says “Driver/library version mismatch”, continue.

3) Upgrade to 550 (in-place)

sudo apt-get update
sudo apt-get install -y nvidia-driver-550
sudo reboot

4) If the mismatch still appears after that

This means the old 535 bits are lingering or the kernel module didn’t load (Secure Boot signing, DKMS failure, etc.). Check and fix:

Check what’s installed / loaded:

# What nvidia packages are installed?
dpkg -l | grep -E 'nvidia-driver-|nvidia-dkms|libnvidia|cuda' | grep ^ii

# DKMS status – make sure nvidia/550 is built for your kernel
dkms status

# Kernel module loaded?
lsmod | grep nvidia || echo "nvidia module not loaded"
dmesg | grep -i nvidia | tail -n +1

If Secure Boot blocks the module (you’ll see messages about signature/verification in dmesg): complete the MOK enrollment on reboot, or temporarily disable Secure Boot, then reinstall the driver.

If things are messy, do a clean reinstall (WARNING: may remove CUDA packages):

# stop display stack to avoid file-in-use issues (optional on servers)
# sudo systemctl isolate multi-user.target

# purge old driver bits (be cautious if you rely on CUDA packages)
sudo apt-get purge -y '^nvidia-.*' 'libnvidia*'

# re-add the official drivers PPA if 550 isn’t offered
sudo add-apt-repository -y ppa:graphics-drivers/ppa
sudo apt-get update

# install driver 550 fresh
sudo apt-get install -y nvidia-driver-550
sudo reboot

5) Verify

nvidia-smi          # should show 550.xx
cat /proc/driver/nvidia/version

Notes for your setup

• Don’t run llama-server or any CUDA jobs during the upgrade.
• If you’re on a recent kernel and DKMS shows errors, reinstall the driver after updating your kernel headers:
  bash
sudo apt-get install -y linux-headers-$(uname -r)
sudo apt-get install -y --reinstall nvidia-driver-550
sudo reboot
• If you must keep CUDA toolkits, avoid the full purge and just install nvidia-driver-550 (plus nvidia-dkms-550 if offered). If CUDA breaks, we can re-pin compatible CUDA packages after the driver is stable.

If you hit any snags (e.g., Secure Boot prompt, DKMS build failure output), paste the last ~20 lines of the command output and I’ll zero in on it.

Got it. Let’s do a clean, boring, predictable upgrade to 580 that fixes the “Driver/library version mismatch”.

Below is a safe, copy-paste sequence for Ubuntu/Debian systems. It:

• cleanly removes old NVIDIA bits,
• reboots,
• installs nvidia-driver-580,
• handles Secure Boot/DKMS,
• verifies success.

Step 0 — Prep & sanity

# See current state (just info)
nvidia-smi || true
uname -r
lsb_release -a || cat /etc/os-release

Step 1 — Make sure tools & headers exist

sudo apt-get update
sudo apt-get install -y ubuntu-drivers-common linux-headers-$(uname -r)

Step 2 — Stop the GUI (avoids files-in-use)

> You’ll drop to a text console. If you’re SSH’d in, you’re fine.

sudo systemctl isolate multi-user.target

Step 3 — Purge old NVIDIA userspace + modules

> ⚠️ This removes driver packages. It does not necessarily remove CUDA toolkits,
> but if you installed CUDA via apt, it might remove some CUDA libs too.
> If you care, we can reinstall CUDA after the driver is stable.

sudo apt-get purge -y '^nvidia-.*' 'libnvidia*'
sudo apt-get autoremove -y

(If you ever created a manual X config, remove it:)

sudo rm -f /etc/X11/xorg.conf

Rebuild initramfs so modules are consistent:

sudo update-initramfs -u

Step 4 — Reboot to a clean slate

sudo reboot

Step 5 — After reboot, confirm no old module is loaded

lsmod | grep nvidia || echo "no nvidia module loaded (good)"
# Nouveau might be active temporarily — that's fine before we install 580.

Step 6 — Ensure 580 is available; add PPA if needed

sudo add-apt-repository -y ppa:graphics-drivers/ppa
sudo apt-get update
ubuntu-drivers devices       # just to see the recommendation (you said it shows 580)
apt-cache policy nvidia-driver-580

Step 7 — Install the 580 driver

sudo apt-get install -y nvidia-driver-580
# If your repo splits DKMS:
# sudo apt-get install -y nvidia-dkms-580

If Secure Boot is enabled, Ubuntu will prompt to enroll a MOK (set a one-time password). On reboot, complete the MOK enrollment screen so the kernel module can load.

Step 8 — Reboot into the new driver

sudo reboot

Step 9 — Verify everything matches

nvidia-smi
cat /proc/driver/nvidia/version
dkms status | sed -n '1,120p'

You should see Driver Version: 580.xx and no “Driver/library version mismatch”.

Troubleshooting (only if needed)

• Still seeing “version mismatch”
  Check what’s actually installed and loaded:
  bash
dpkg -l | grep -E '^ii\s+nvidia-|^ii\s+libnvidia' | sed -n '1,200p'
lsmod | grep nvidia || echo "module not loaded"
dmesg | grep -i -e nvidia -e nouveau | tail -n 80
  If Secure Boot blocked the module, complete MOK enrollment or disable Secure Boot temporarily and reinstall the driver:
  bash
sudo apt-get install -y --reinstall nvidia-driver-580
sudo reboot

• DKMS build failed (you’ll see it in dkms status):
  bash
sudo apt-get install -y linux-headers-$(uname -r)
sudo apt-get install -y --reinstall nvidia-driver-580
sudo reboot

• Black screen after install
  Switch to TTY: Ctrl+Alt+F3, log in, then either:
  bash
sudo systemctl isolate multi-user.target
sudo apt-get purge -y '^nvidia-.*' 'libnvidia*'
sudo apt-get install -y nvidia-driver-580
sudo reboot
  …or roll back to a known-good driver (e.g., 550) if 580 is problematic on your kernel.

When this is green, we can go back to the ingestion with Nomic ST only, confident your GPU stack isn’t causing stalls.

Totally—no TTY needed. We can do this from your GUI terminal with just one reboot at the end.

Here are two safe approaches that keep your desktop up during the steps:

Option A (easiest): use the GUI

1. Open Software & Updates → Additional Drivers.
2. Select nvidia-driver-580 (proprietary, tested) (or whatever is shown as recommended).
3. Apply / Restart when prompted.

Option B (terminal only, no TTY)

This will upgrade in place and only require a reboot at the end.

# 0) (Optional) stop any CUDA-heavy apps so files aren't in use
pkill -f llama-server || true

# 1) Make sure tooling & headers are present
sudo apt-get update
sudo apt-get install -y ubuntu-drivers-common linux-headers-$(uname -r)

# 2) If you’ve ever pinned/held a driver, unhold it
apt-mark showhold
# If you see any nvidia packages held, unhold them, e.g.:
# sudo apt-mark unhold nvidia-driver-535 nvidia-dkms-535 libnvidia-gl-535

# 3) Add/refresh the graphics PPA (harmless if already added)
sudo add-apt-repository -y ppa:graphics-drivers/ppa
sudo apt-get update

# 4) Install the recommended 580 driver (in-place upgrade)
sudo apt-get install -y nvidia-driver-580

# 5) Reboot to load the new kernel module
sudo reboot

After reboot:

nvidia-smi           # should now show Driver Version: 580.xx

If it still says “Driver/library version mismatch”

That usually means old bits are hanging around. You can still fix this from the GUI session (no TTY) with a fast purge → install flow:

# Pre-download 580 so install is quick
sudo apt-get install -y --download-only nvidia-driver-580

# Purge old userspace/kernel packages (screen may flicker, but your session stays)
sudo apt-get purge -y '^nvidia-.*' 'libnvidia*'
sudo apt-get autoremove -y

# Install 580 immediately
sudo apt-get install -y nvidia-driver-580

# Reboot once
sudo reboot

Notes

• If Secure Boot is enabled, you’ll get a MOK enrollment prompt during install and again on reboot—just follow the on-screen steps so the module can load.
• If dkms status later shows build errors, run:
  bash
sudo apt-get install -y linux-headers-$(uname -r)
sudo apt-get install -y --reinstall nvidia-driver-580
sudo reboot

Once you’re on 580 and nvidia-smi is happy, we’ll resume ingestion with Nomic ST only exactly as we locked in.

Got it—here’s the short, no-TTY, just-purge version you’re used to. This keeps your desktop up; you’ll only reboot at the end.

1) See what’s installed (just for sanity)

dpkg -l | grep -E '^ii\s+(nvidia|libnvidia)' || true

2) Purge NVIDIA driver packages

> Minimal purge (keeps non-driver CUDA stuff if any):

sudo apt-get purge -y '^nvidia-driver-[0-9]+$' 'nvidia-dkms-*' \
  'nvidia-utils-*' 'nvidia-compute-utils-*' 'libnvidia-gl-*' 'libnvidia-common-*'
sudo apt-get autoremove -y

> Full purge (if you don’t care about any NVIDIA/CUDA userspace right now):

sudo apt-get purge -y '^nvidia-.*' 'libnvidia*'
sudo apt-get autoremove -y

3) Clean leftover config so it falls back to the open driver

sudo rm -f /etc/X11/xorg.conf
sudo rm -f /etc/modprobe.d/blacklist-nvidia-nouveau.conf /lib/modprobe.d/blacklist-nvidia-nouveau.conf 2>/dev/null || true
sudo update-initramfs -u

4) Reboot once

sudo reboot

After reboot, you should be on the kernel’s nouveau or modesetting driver and still have your GUI.

When you’re ready to install the new one (e.g., 580):

sudo apt-get update
sudo apt-get install -y nvidia-driver-580
sudo reboot

If anything weird happens (e.g., Secure Boot/MOK prompt, or nvidia-smi still mismatched), tell me what you see and we’ll keep it tight and simple.