Perfect, Darren — now we’ve got the real details:

• Device: /dev/sdb
• Label: storage
• UUID: 158c3431-c479-41fc-8ba9-42b5d88e68ac
• Type: ext4
• Size: 3.64 TiB
• Current mount (auto): /media/darren/storage

So the issue wasn’t a bad drive — it was just fstab pointing at a fake placeholder UUID before. Let’s fix it clean and permanent.

1. Backup & edit fstab

sudo cp /etc/fstab /etc/fstab.bak.$(date +%F_%H%M)
sudo nano /etc/fstab

Add this line (exact UUID from your blkid):

UUID=158c3431-c479-41fc-8ba9-42b5d88e68ac  /mnt/storage  ext4  defaults,uid=1000,gid=1000  0 2

2. Unmount the auto-mount

sudo umount -lf /media/darren/storage

3. Reload systemd & mount

sudo systemctl daemon-reload
sudo mount -a

4. Verify

ls -la /mnt/storage

You should now see:

models
webway_documents_old
pile_uncopyrighted

5. Test ingestion

CUDA_VISIBLE_DEVICES=0 \
python3 ~/Documents/tokenspace/ingest_pile.py \
  --root_dir "/mnt/storage/pile_uncopyrighted"

⚠️ One note: since the drive has a label (LABEL="storage"), you could also mount by label instead of UUID in fstab:

LABEL=storage  /mnt/storage  ext4  defaults,uid=1000,gid=1000  0 2

That can be a little easier to remember.

Do you want me to write the fstab entry using LABEL=storage instead of UUID, so it’s more human-readable?

Got it—let’s nail this down step-by-step and avoid guesswork. We’ll (A) verify what’s mounted, (B) cleanly unmount anything, (C) test-mount using the actual UUID, and (D) fix fstab if needed.

A) Verify current state

Run these and skim the outputs:

findmnt /dev/sdb
findmnt /mnt/storage
df -hT | grep -E 'sdb|/mnt/storage|/media/darren/storage'

B) Unmount anything that’s in the way

(These are safe even if nothing is mounted.)

sudo umount -lf /media/darren/storage 2>/dev/null || true
sudo umount -lf /mnt/storage           2>/dev/null || true
sudo umount -lf /dev/sdb               2>/dev/null || true

C) Test-mount by UUID (bypasses fstab)

You already showed:

/dev/sdb: LABEL="storage" UUID="158c3431-c479-41fc-8ba9-42b5d88e68ac" TYPE="ext4"

Now test-mount directly with that UUID:

sudo mkdir -p /mnt/storage
sudo mount -t ext4 -U 158c3431-c479-41fc-8ba9-42b5d88e68ac /mnt/storage
ls -la /mnt/storage

• If you now see models, webway_documents_old, pile_uncopyrighted, the filesystem is fine and the issue is fstab syntax. Jump to D1.
• If you still get the “wrong fs type / bad superblock” error, run:
  bash
dmesg | tail -n 50
  (But given it was mounting at /media/darren/storage earlier, the superblock is almost certainly fine—this error usually means we were trying to mount the wrong node or a conflicting mount was still active.)

D) Fix /etc/fstab

First, show what’s there to catch typos/duplicates:

nl -ba /etc/fstab

D1) Clean, correct entry (recommended: by UUID)

Replace any placeholder with the exact UUID and keep options simple:

UUID=158c3431-c479-41fc-8ba9-42b5d88e68ac  /mnt/storage  ext4  defaults,uid=1000,gid=1000  0 2

(Or mount by label, equivalent and human-readable:

LABEL=storage  /mnt/storage  ext4  defaults,uid=1000,gid=1000  0 2

)

Save, then reload + apply:

sudo systemctl daemon-reload
sudo mount -a
ls -la /mnt/storage

You should now see your three folders at /mnt/storage.

Quick ingest sanity check (once /mnt/storage shows your data)

CUDA_VISIBLE_DEVICES=0 \
python3 ~/Documents/tokenspace/ingest_pile.py \
  --root_dir "/mnt/storage/pile_uncopyrighted"

If it still errors on mount -a

Common culprits to check:

• Duplicate/misaligned lines in /etc/fstab (two entries for /mnt/storage, or one for /dev/sdb and another for the UUID).
• Wrong device in fstab (e.g., /dev/sdb1 when your FS is on /dev/sdb).
• Hidden characters (smart quotes, stray spaces). Re-type the line manually.

If you paste the output of:

nl -ba /etc/fstab
findmnt -rno SOURCE,TARGET,FSTYPE /mnt/storage

I’ll give you the exact corrected line to use.

You didn’t do anything wrong — you just pasted the output of blkid back into the shell, and the shell tried to run /dev/sdb: as a command. The trailing colon comes from the blkid printout; it’s not part of a path. The label isn’t the problem.

Let’s mount this cleanly and make it stick.

Mount it now (whole-disk ext4 on /dev/sdb)

Your earlier df showed the filesystem lives directly on /dev/sdb (no partitions). Do:

# make sure nothing is mounted there
sudo umount -lf /media/darren/storage 2>/dev/null || true
sudo umount -lf /mnt/storage           2>/dev/null || true

# ensure the target mountpoint exists
sudo mkdir -p /mnt/storage

# mount by device (works immediately)
sudo mount -t ext4 /dev/sdb /mnt/storage

# or equivalently, mount by UUID (also fine)
# sudo mount -t ext4 -U 158c3431-c479-41fc-8ba9-42b5d88e68ac /mnt/storage

# verify you see your folders
ls -la /mnt/storage

If you see models, webway_documents_old, pile_uncopyrighted, you’re good.

Make it persistent (fstab)

Edit /etc/fstab and add one of these lines (choose ONE; both are equivalent):

# by UUID (recommended)
UUID=158c3431-c479-41fc-8ba9-42b5d88e68ac  /mnt/storage  ext4  defaults,uid=1000,gid=1000  0 2

# OR by label (human-readable)
LABEL=storage                             /mnt/storage  ext4  defaults,uid=1000,gid=1000  0 2

Apply and verify:

sudo systemctl daemon-reload
sudo mount -a
ls -la /mnt/storage

Run your ingest with the stable path

CUDA_VISIBLE_DEVICES=0 \
python3 ~/Documents/tokenspace/ingest_pile.py \
  --root_dir "/mnt/storage/pile_uncopyrighted"

If mount errors again

• Capture the reason: dmesg | tail -n 50
• Paste it and the result of:
  findmnt /dev/sdb and findmnt /mnt/storage
  (Empty output just means “not mounted” — that’s okay before the mount.)

Good—now we’re hitting the real issue: the model path. The error says:

FileNotFoundError: Path /mnt/storage/models/noemic-embed-768 not found

Two likely problems:

1) Spelling/typo: noemic-embed-768 vs nomic (no “e” after the “no”).
2) That folder doesn’t exist under /mnt/storage/models (even if spelled right).

Let’s fix it fast.

1) See what you actually have

ls -la /mnt/storage/models
find /mnt/storage -maxdepth 4 -type d -iname "*nomic*"

If you don’t see a nomic model directory under /mnt/storage/models, but you do see the cache under HF hub (you showed this earlier):

/mnt/storage/hf-cache/hub/models--nomic-ai--nomic-bert-2048/snapshots/<SHA>

we can just point your script to that cached snapshot.

2) Create a clean, stable path (symlink) for the model

Pick the snapshot (usually there’s only one):

SNAP=$(ls -1d /mnt/storage/hf-cache/hub/models--nomic-ai--nomic-bert-2048/snapshots/* | head -n1)
sudo mkdir -p /mnt/storage/models
# use a clear name; note **nomic** (no extra 'e')
sudo ln -s "$SNAP" /mnt/storage/models/nomic-embed-768
ls -la /mnt/storage/models/nomic-embed-768

Now you’ve got a stable, human-friendly path:

/mnt/storage/models/nomic-embed-768

3) Re-run ingest with the correct MODEL_DIR

If your script takes a flag (e.g., --model_dir), use it; otherwise it’s probably reading a constant at the top. Two ways:

A. CLI flag (if supported):

CUDA_VISIBLE_DEVICES=0 \
python3 ~/Documents/tokenspace/ingest_pile.py \
  --root_dir "/mnt/storage/pile_uncopyrighted" \
  --model_dir "/mnt/storage/models/nomic-embed-768"

B. Environment variable (if your script checks MODEL_DIR):

export MODEL_DIR="/mnt/storage/models/nomic-embed-768"
CUDA_VISIBLE_DEVICES=0 \
python3 ~/Documents/tokenspace/ingest_pile.py \
  --root_dir "/mnt/storage/pile_uncopyrighted"

C. Hard-coded constant
If ingest_pile.py has a line like:

MODEL_DIR = "/mnt/storage/models/noemic-embed-768"

edit it to:

MODEL_DIR = "/mnt/storage/models/nomic-embed-768"

4) Quick sanity check

Before launching the full ingest, make sure the model path resolves:

python3 - << 'PY'
import os
p="/mnt/storage/models/nomic-embed-768"
print("exists:", os.path.exists(p))
print("files:", len(os.listdir(p)) if os.path.exists(p) else "n/a")
PY

(If exists: True and there are files/dirs listed, SentenceTransformer should be able to load it.)

Why this works

• SentenceTransformer(...) can load from a local directory that contains the model files.
• Hugging Face caches models under .../hub/models--ORG--REPO/snapshots/<SHA>.
• Symlinking that snapshot to a friendly path avoids long, brittle cache paths and typos.

If you prefer a different Nomic model (e.g., nomic-ai/nomic-embed-text-v1.5), do the same process: find its cached snapshot under hf-cache/hub/models--nomic-ai--nomic-embed-text-v1.5/snapshots/<SHA>, symlink it to /mnt/storage/models/nomic-embed-768 (or a name you like), and point the script there.

Want me to also drop a llama.cpp startup script on your Desktop for Qwen3 (Q8_0 / 32k window) so it’s one double-click?

Right now your 4 TB drive is mounted at /mnt/storage, but GNOME/Nautilus doesn’t show it in the sidebar because the desktop environment only lists “removable” drives (auto-mounted under /media/...) by default. When you override it with /etc/fstab → /mnt/storage, it works fine for the terminal, but Nautilus won’t display it unless you tell it to.

You’ve got a few options:

1. Bookmark it manually in Nautilus

• Open Nautilus (Files).
• Hit Ctrl+L and type:
  /mnt/storage
• Once it opens, press Ctrl+D (or right-click “Add Bookmark”).
• Now you’ll have a permanent “Storage” entry in the sidebar.

2. Mount it under /media

If you want Nautilus to treat it like a “normal” external drive, you can change your fstab entry so it mounts under /media/darren/storage instead of /mnt/storage:

UUID=158c3431-c479-41fc-8ba9-42b5d88e68ac  /media/darren/storage  ext4  defaults,uid=1000,gid=1000  0 2

• Create the folder if it doesn’t exist:
  bash
sudo mkdir -p /media/darren/storage
• Then remount:
  bash
sudo mount -a

It will then appear in Nautilus automatically (same as before the fstab change).

3. Use a symlink

Keep your clean /mnt/storage mount for scripts, but add a link in your home so the GUI can see it:

ln -s /mnt/storage ~/Storage

Now “Storage” will show up in your home folder, and you can bookmark it in Nautilus.

👉 Most people go with Option 1 (bookmark) because it’s the simplest and keeps /mnt/storage for scripts.
If you want it exactly like before (auto-showing in the sidebar without extra clicks), then Option 2 (mount under /media/darren/storage) is the way.

Do you want it to behave like a “normal removable disk” (auto-show in sidebar), or are you okay with just bookmarking /mnt/storage once and leaving it there?

Ah — good catch. That happens because Nautilus only lets you bookmark folders inside your home directory by default, and /mnt/storage is outside of it. That’s why Ctrl+D was grayed out.

You’ve got two easy workarounds:

Option 1. Add a symlink inside your home

Create a link that points to /mnt/storage but lives inside your home:

ln -s /mnt/storage ~/Storage

Now open Nautilus → go into your home folder → you’ll see a folder called Storage (with the little arrow icon).
– You can bookmark that (Ctrl+D will work).
– Click it anytime to open /mnt/storage through the GUI.

Option 2. Mount under /media/darren/storage

If you want it to behave like a “normal” external disk (auto-shows up in sidebar with an eject icon):

1. Edit /etc/fstab and change the mount point from /mnt/storage → /media/darren/storage

UUID=158c3431-c479-41fc-8ba9-42b5d88e68ac /media/darren/storage ext4 defaults,uid=1000,gid=1000 0 2

2. Make sure the folder exists:
   bash
sudo mkdir -p /media/darren/storage

3. Remount:
   bash
sudo umount /mnt/storage
sudo mount -a

Now Nautilus should show it automatically in the sidebar, like before.

👉 If you want the simplest solution right now: go with Option 1 (symlink + bookmark) — one command and done.

Would you like me to give you the exact command sequence to switch from /mnt/storage to /media/darren/storage in fstab so it auto-appears every boot?

Right — that’s because Nautilus doesn’t let you directly bookmark paths outside your home unless they’re already inside a “user-visible” location. Since /mnt/storage is a system path, it doesn’t count. That’s why the bookmark option is disabled.

Two clean fixes:

✅ Fix 1: Symlink into your home (fastest + reliable)

This makes a shortcut in your home, which Nautilus will let you bookmark:

ln -s /mnt/storage ~/Storage

Now:
– Open Nautilus → go to your Home folder.
– You’ll see a new entry called Storage (with a little arrow icon).
– Right-click → Add to Bookmarks will now be available.

✅ Fix 2: Mount under /media/darren/storage (acts like a “normal drive”)

If you want it to behave exactly like before (auto-appearing in the sidebar with an eject button):

1. Edit fstab:
   bash
sudo nano /etc/fstab
   Change your line to:
   UUID=158c3431-c479-41fc-8ba9-42b5d88e68ac /media/darren/storage ext4 defaults,uid=1000,gid=1000 0 2

2. Make sure the folder exists:
   bash
sudo mkdir -p /media/darren/storage

3. Remount:
   bash
sudo umount /mnt/storage
sudo mount -a

Now it’ll show up in Nautilus automatically like any removable disk.

👉 If you just want to see it in Nautilus sidebar quickly → use Fix 1 (symlink + bookmark).
👉 If you want it to behave like a “real drive” permanently → use Fix 2 (mount under /media).

Do you want me to walk you through switching your fstab entry to /media/darren/storage so it comes back every boot?

Got it. 👍 Thanks for clarifying.

No — switching from LABEL=storage to UUID=158c3431-... won’t make Nautilus suddenly show it in the sidebar. Both LABEL= and UUID= in /etc/fstab just tell the kernel which filesystem to mount; Nautilus doesn’t care which one you use.

Here’s the distinction:

• UUID=… → the most reliable way to identify a disk (never changes unless you reformat).
• LABEL=… → more human-readable, but labels can be duplicated or renamed.
• Nautilus sidebar → only auto-shows mounts under /media/$USER/..., not /mnt/..., regardless of UUID vs LABEL.

So:
– If your goal is stability: use UUID=… in fstab.
– If your goal is human readability: LABEL=… is fine.
– If your goal is sidebar integration: only mounting under /media/darren/... will do it — the identifier (UUID vs LABEL) doesn’t matter.

Since you’re okay with symlinks and just need it working: stick with what you’ve got. The only reason I’d recommend switching to UUID is if you want to be absolutely sure the wrong disk never mounts at /mnt/storage (UUIDs are unique; labels aren’t).

👉 Do you want me to rewrite your fstab entry using UUID so it’s rock-solid, but keep everything else the same?

Perfect choice 👍 — the symlink in your home directory is the cleanest, least-hassle solution. It’ll behave just like a normal folder for you, and you don’t have to wrestle with Nautilus or fstab every time.

So you’re set with:

• /mnt/storage → stable mount point for scripts and ingestion.
• ~/Storage → symlink you can click in Nautilus whenever you want GUI access.

That covers both worlds.

Do you want me to help tidy up your fstab now (make sure it only has the one clean entry for the drive), or do you prefer to leave that untouched since the symlink works?

👍 Got it — let’s get ingestion running smoothly now that storage is sorted.

Right now the script is choking because it’s trying to load the model from:

/mnt/storage/models/noemic-embed-768

but that path doesn’t exist. You actually have the Nomic model cached under your HuggingFace cache at:

/mnt/storage/hf-cache/hub/models--nomic-ai--nomic-bert-2048/snapshots/<SHA>

Step 1. Locate the exact snapshot

Run this to see the snapshot folder name:

ls -d /mnt/storage/hf-cache/hub/models--nomic-ai--nomic-bert-2048/snapshots/*

It will print something like:

/mnt/storage/hf-cache/hub/models--nomic-ai--nomic-bert-2048/snapshots/21fe6acc1eeed0f9cd9355219cbfeb3b193efc73

Step 2. Create a stable model path

Link that snapshot to a simple, permanent name under /mnt/storage/models:

sudo mkdir -p /mnt/storage/models
sudo ln -s /mnt/storage/hf-cache/hub/models--nomic-ai--nomic-bert-2048/snapshots/<YOUR-SHA> /mnt/storage/models/nomic-embed-768

(replace <YOUR-SHA> with what you saw in Step 1)

Step 3. Verify

ls -la /mnt/storage/models/nomic-embed-768

You should see config.json, pytorch_model.bin, tokenizer.json, etc.

Step 4. Rerun ingestion

CUDA_VISIBLE_DEVICES=0 \
python3 ~/Documents/tokenspace/ingest_pile.py \
  --root_dir "/mnt/storage/pile_uncopyrighted" \
  --model_dir "/mnt/storage/models/nomic-embed-768"

If that still errors, we’ll double-check whether your script hardcodes MODEL_DIR inside ingest_pile.py. If it does, I’ll help you edit that one line to point to the new symlink path.

👉 Do you want me to prep the exact MODEL_DIR = "..." replacement line for ingest_pile.py in case it’s hardcoded there?

Got it, Darren — let’s trace this back to the main drive (your OS disk) instead of the big storage SSD.

From what you’ve said before, your project files live under ~/Documents/. Let’s check systematically:

1. List your Documents folder

ls -la ~/Documents

That should show things like tokenspace, ougway_venv, maybe models if you’ve got a copy there.

2. Check inside Documents/models (if it exists)

ls -la ~/Documents/models

3. Search directly under Documents for anything “nomic”

find ~/Documents -maxdepth 3 -type d -iname "*nomic*"

Once we see where the nomic-embed-768 actually is on the main drive, we can:

• update your MODEL_DIR in ingest_pile.py
• or pass --model_dir ~/Documents/models/nomic-embed-768 when you run ingestion

so it stops looking on /mnt/storage and loads from the right place.

👉 Want me to also prep a clean directory structure (like ~/Documents/models, ~/Documents/data, ~/Documents/logs) so all of Oogway’s pieces are consistent and not scattered? That way ingestion, training, and logs all stay in predictable places.

Perfect—your main drive’s ~/Documents/models only has the Qwen file, so the embedder isn’t there yet. Easiest fix: make a symlink in ~/Documents/models that points to the already-cached Nomic embedder under your HF cache on the storage drive. That keeps your “everything under Documents” layout without copying GBs.

1) Point a clean path in Documents to the cached Nomic model

# find the cached snapshot path
SNAP=$(ls -1d /mnt/storage/hf-cache/hub/models--nomic-ai--nomic-bert-2048/snapshots/* | head -n1)

# create a friendly path in Documents/models
mkdir -p ~/Documents/models
ln -s "$SNAP" ~/Documents/models/nomic-embed-768

# quick sanity check
ls -la ~/Documents/models/nomic-embed-768

2) Run ingestion using that path + your pile folder

CUDA_VISIBLE_DEVICES=0 \
python3 ~/Documents/tokenspace/ingest_pile.py \
  --root_dir "/mnt/storage/pile_uncopyrighted" \
  --model_dir "$HOME/Documents/models/nomic-embed-768"

> If your script doesn’t accept --model_dir, open ~/Documents/tokenspace/ingest_pile.py and change the MODEL_DIR = "..." line near the top to:
> python
> MODEL_DIR = "/home/darren/Documents/models/nomic-embed-768"
>

That should clear the “Path not found” error and let SentenceTransformer load the embedder locally (your offline env vars are fine since we’re pointing to a real folder). If anything still barks, paste the new traceback and we’ll adjust.

Totally—let’s find the embedder for real and stop guessing. We’ll search both your home (main OS drive) and the storage drive with safe, quiet commands.

1) Confirm where HF cache wants to live

echo "HF_HOME=$HF_HOME"

(If blank, Hugging Face may be using ~/.cache/huggingface by default.)

2) Search your home drive (quick + quiet)

# common patterns for Nomic embed models/caches
find "$HOME" -type d -iname "*nomic*embed*" 2>/dev/null
find "$HOME" -type d -iname "models--nomic-ai*" 2>/dev/null
# also look in the default HF cache path just in case
find "$HOME/.cache/huggingface" -maxdepth 4 -type d -iname "models--nomic-ai*" 2>/dev/null

3) Search the storage drive (where we saw hf-cache before)

find /mnt/storage -maxdepth 6 -type d -iname "*nomic*embed*" 2>/dev/null
find /mnt/storage -maxdepth 6 -type d -iname "models--nomic-ai*" 2>/dev/null

4) Sanity search for actual model files (broad)

# look for a tokenizer.json or config.json near any 'nomic' directory
find "$HOME" /mnt/storage -type f \( -iname "tokenizer.json" -o -iname "config.json" \) -path "*nomic*" 2>/dev/null | sed 's#/[^/]*$##' | sort -u

5) (Optional) Faster search if you have fd or ripgrep

# fd (aka fdfind on Ubuntu)
fd -HI --type d 'nomic.*embed' "$HOME" /mnt/storage 2>/dev/null || true
# ripgrep
rg -i --hidden --no-messages -g '!*.pyc' 'nomic.*embed' "$HOME" /mnt/storage

What to do with the result

• If you see a path like:
  /mnt/storage/hf-cache/hub/models--nomic-ai--nomic-bert-2048/snapshots/<SHA>
  create a friendly link and use it:
  bash
mkdir -p ~/Documents/models
ln -s "/mnt/storage/hf-cache/hub/models--nomic-ai--nomic-bert-2048/snapshots/<SHA>" \
~/Documents/models/nomic-embed-768

• If it turns up under your home cache (e.g., ~/.cache/huggingface/hub/models--nomic-ai--.../snapshots/<SHA>), link from there instead:
  bash
ln -s "$HOME/.cache/huggingface/hub/models--nomic-ai--nomic-bert-2048/snapshots/<SHA>" \
~/Documents/models/nomic-embed-768

• Then run ingest pointing to this clean path:
  bash
CUDA_VISIBLE_DEVICES=0 \
python3 ~/Documents/tokenspace/ingest_pile.py \
--root_dir "/mnt/storage/pile_uncopyrighted" \
--model_dir "$HOME/Documents/models/nomic-embed-768"

If you paste the actual path that the search returns, I’ll give you the exact ln -s command and a one-line ingest command tailored to it.

Great — HF_HOME is set to /mnt/storage/hf-cache, so if the Nomic embedder was ever pulled, its files will be under that cache. Let’s find it for real and wire it up so ingestion works.

A) Locate the embedder (search both drives)

Paste this block exactly; it prints any likely paths:

# Search the HF cache on storage (primary)
find /mnt/storage/hf-cache/hub -maxdepth 5 -type d -iname "models--nomic-ai*" 2>/dev/null

# Also search your home (in case an older cache exists there)
find "$HOME/.cache/huggingface/hub" -maxdepth 5 -type d -iname "models--nomic-ai*" 2>/dev/null

# Broad sanity check for model configs near "nomic"
find /mnt/storage "$HOME" -type f \( -iname "config.json" -o -iname "tokenizer.json" \) -path "*nomic*" 2>/dev/null | sed 's#/[^/]*$##' | sort -u

You’ll likely see something like:

/mnt/storage/hf-cache/hub/models--nomic-ai--nomic-bert-2048/snapshots/<SHA>

B) One-liner to create a stable model path and run ingest

This will: (1) pick the first found snapshot, (2) symlink it to ~/Documents/models/nomic-embed-768, (3) run ingestion pointing to it.

# Pick the best snapshot from either cache
SNAP=$(ls -1d /mnt/storage/hf-cache/hub/models--nomic-ai*/snapshots/* 2>/dev/null | head -n1)
if [ -z "$SNAP" ]; then
  SNAP=$(ls -1d "$HOME"/.cache/huggingface/hub/models--nomic-ai*/snapshots/* 2>/dev/null | head -n1)
fi

# Bail out if still not found
if [ -z "$SNAP" ]; then
  echo "No Nomic embedder snapshot found. Try running once online to cache it, or tell me the exact model repo you want."
else
  echo "Using snapshot: $SNAP"
  mkdir -p "$HOME/Documents/models"
  # fresh link
  rm -f "$HOME/Documents/models/nomic-embed-768"
  ln -s "$SNAP" "$HOME/Documents/models/nomic-embed-768"
  ls -la "$HOME/Documents/models/nomic-embed-768"

  # Kick off ingestion with explicit model_dir
  CUDA_VISIBLE_DEVICES=0 \
  python3 "$HOME/Documents/tokenspace/ingest_pile.py" \
    --root_dir "/mnt/storage/pile_uncopyrighted" \
    --model_dir "$HOME/Documents/models/nomic-embed-768"
fi

C) If your script ignores --model_dir

Open the script and set the constant:

nano ~/Documents/tokenspace/ingest_pile.py

Find the line defining MODEL_DIR and make it:

MODEL_DIR = "/home/darren/Documents/models/nomic-embed-768"

Save, then re-run the ingest command (without --model_dir if it doesn’t support it).

If the search prints nothing, that means the embedder was never cached on this install. In that case, tell me the exact model you want (e.g., nomic-ai/nomic-bert-2048 or nomic-ai/nomic-embed-text-v1.5), and I’ll give you a quick offline-friendly placement (manual download path or a minimal online fetch to seed the cache).

Alright — since the searches came back empty on both drives, that means the embedder model was never cached on this install. In other words, there’s no local copy of nomic-embed-768 yet.

Right now your ingestion script is failing because it expects that model directory to exist. We’ve got two clean options to fix this:

Option 1: Redownload the embedder (simplest)

If you allow one online fetch (with your Hugging Face offline flags temporarily disabled), we can pull the right model into your cache.

For example, the common one is:

nomic-ai/nomic-embed-text-v1.5

That’s the actual repo name. To download:

# deactivate offline mode
unset TRANSFORMERS_OFFLINE HF_HUB_OFFLINE

# pull the model once
python3 -m sentence_transformers \
    nomic-ai/nomic-embed-text-v1.5 \
    --cache-dir /mnt/storage/hf-cache

Then re-enable offline mode afterward. This will put the snapshot in:

/mnt/storage/hf-cache/hub/models--nomic-ai--nomic-embed-text-v1.5/snapshots/<SHA>/

and we can symlink that into ~/Documents/models/nomic-embed-768.

Option 2: Manual download

If you don’t want your machine to go online, we can download the model archive separately (from another machine or directly via browser) and place it into ~/Documents/models/nomic-embed-768. But that requires you to move files manually.

👉 Since ingestion depends on this embedder, we’ll need at least one clean copy of the model somewhere. Do you want me to prep the exact command to download nomic-ai/nomic-embed-text-v1.5 into your existing /mnt/storage/hf-cache, then make the symlink so the script finds it?

No problem—the sentence_transformers package isn’t an executable module, so that command won’t work. Use one of these two reliable methods:

Method A (simple, uses SentenceTransformer)

This will download nomic-ai/nomic-embed-text-v1.5 into your existing HF cache (/mnt/storage/hf-cache) and then we’ll symlink it.

# 1) Make sure offline is disabled for this one fetch
unset TRANSFORMERS_OFFLINE HF_HUB_OFFLINE

# 2) Download via a tiny Python one-liner (uses your venv)
python3 - << 'PY'
from sentence_transformers import SentenceTransformer
m = SentenceTransformer("nomic-ai/nomic-embed-text-v1.5", trust_remote_code=True, cache_folder="/mnt/storage/hf-cache")
print("Downloaded to:", m._last_saved_dir if hasattr(m, "_last_saved_dir") else "cache ok")
PY

# 3) Find the snapshot path that was just cached
SNAP=$(ls -1d /mnt/storage/hf-cache/hub/models--nomic-ai--nomic-embed-text-v1.5/snapshots/* | head -n1)
echo "Using snapshot: $SNAP"

# 4) Expose a clean, stable path for your script
mkdir -p "$HOME/Documents/models"
rm -f "$HOME/Documents/models/nomic-embed-768"
ln -s "$SNAP" "$HOME/Documents/models/nomic-embed-768"
ls -la "$HOME/Documents/models/nomic-embed-768"

Then run ingest pointing to it:

CUDA_VISIBLE_DEVICES=0 \
python3 ~/Documents/tokenspace/ingest_pile.py \
  --root_dir "/mnt/storage/pile_uncopyrighted" \
  --model_dir "$HOME/Documents/models/nomic-embed-768"

(If your script ignores --model_dir, open it and set MODEL_DIR = "/home/darren/Documents/models/nomic-embed-768".)

Method B (more control, uses huggingface_hub)

If you’ve got huggingface_hub installed, this downloads the repo snapshot explicitly:

python3 - << 'PY'
from huggingface_hub import snapshot_download
p = snapshot_download(
    repo_id="nomic-ai/nomic-embed-text-v1.5",
    cache_dir="/mnt/storage/hf-cache",
    local_files_only=False
)
print("Snapshot at:", p)
PY

SNAP=$(ls -1d /mnt/storage/hf-cache/hub/models--nomic-ai--nomic-embed-text-v1.5/snapshots/* | head -n1)
mkdir -p "$HOME/Documents/models"
rm -f "$HOME/Documents/models/nomic-embed-768"
ln -s "$SNAP" "$HOME/Documents/models/nomic-embed-768"

Then the same ingest command as above.

If either method complains about missing packages, run:

pip install --upgrade sentence-transformers huggingface_hub

Once the symlink exists at ~/Documents/models/nomic-embed-768, your ingest should stop throwing the “Path not found” error.

Yep — your script is ignoring --model_dir and still using a hard-coded path with a typo (noemic vs nomic) at /mnt/storage/models/noemic-embed-768. Two quick ways to fix it. Pick one.

Fastest fix (one-line edit to the constant)

1) Point MODEL_DIR to your desired path (and fix the spelling):

sed -i 's#^MODEL_DIR *= *".*"#MODEL_DIR = "/home/darren/Documents/models/nomic-embed-768"#' \
  ~/Documents/tokenspace/ingest_pile.py

2) Sanity check the change:

grep -n 'MODEL_DIR' ~/Documents/tokenspace/ingest_pile.py

3) Run ingest (no need to pass --model_dir anymore):

CUDA_VISIBLE_DEVICES=0 \
python3 ~/Documents/tokenspace/ingest_pile.py \
  --root_dir "/mnt/storage/pile_uncopyrighted"

Slightly cleaner fix (teach the script to accept --model_dir)

If you’d rather keep it configurable:

1) Open the script:

nano ~/Documents/tokenspace/ingest_pile.py

2) Near the top where import argparse likely is (or add it), add:

import os, argparse

# default path (safe fallback)
MODEL_DIR_DEFAULT = "/home/darren/Documents/models/nomic-embed-768"

parser = argparse.ArgumentParser()
parser.add_argument("--root_dir", required=True)
parser.add_argument("--model_dir", default=os.environ.get("MODEL_DIR", MODEL_DIR_DEFAULT))
args = parser.parse_args()

ROOT_DIR = args.root_dir
MODEL_DIR = args.model_dir

3) Ensure the rest of the script uses ROOT_DIR and MODEL_DIR (it already uses MODEL_DIR where it loads SentenceTransformer(MODEL_DIR, ...)).

4) Run with the flag:

CUDA_VISIBLE_DEVICES=0 \
python3 ~/Documents/tokenspace/ingest_pile.py \
  --root_dir "/mnt/storage/pile_uncopyrighted" \
  --model_dir "/home/darren/Documents/models/nomic-embed-768"

(You can also set MODEL_DIR=/home/darren/Documents/models/nomic-embed-768 in env and omit the flag.)

If you haven’t actually placed the embedder yet

If ~/Documents/models/nomic-embed-768 doesn’t exist, create it once:

# temporarily allow online fetch (1-time)
unset TRANSFORMERS_OFFLINE HF_HUB_OFFLINE

python3 - << 'PY'
from sentence_transformers import SentenceTransformer
m = SentenceTransformer("nomic-ai/nomic-embed-text-v1.5",
                        trust_remote_code=True,
                        cache_folder="/mnt/storage/hf-cache")
print("cached ok")
PY

SNAP=$(ls -1d /mnt/storage/hf-cache/hub/models--nomic-ai--nomic-embed-text-v1.5/snapshots/* | head -n1)
mkdir -p "$HOME/Documents/models"
ln -sfn "$SNAP" "$HOME/Documents/models/nomic-embed-768"
ls -la "$HOME/Documents/models/nomic-embed-768"

Then re-run ingest (with either fix above).

Why it failed

• The script has a hard-coded MODEL_DIR pointing to a nonexistent and misspelled path.
• Passing --model_dir didn’t help because the script never parses/uses that argument.

Fix the constant or wire up argparse, and you’re good.

Your script is still using a hard-coded, misspelled path (noemic), and it’s ignoring the --model_dir flag. Let’s fix it in 60 seconds.

Fix the hard-coded path (fastest)

1) Open the script and change the MODEL_DIR line:

nano ~/Documents/tokenspace/ingest_pile.py

• Find the line that looks like:
  python
MODEL_DIR = "/mnt/storage/models/noemic-embed-768"
• Replace it with:
  python
MODEL_DIR = "/home/darren/Documents/models/nomic-embed-768"
• Save (Ctrl+O, Enter) and exit (Ctrl+X).

2) Make sure that path actually exists (create a link if needed):

# if you already downloaded nomic-ai/nomic-embed-text-v1.5 earlier, wire it up:
SNAP=$(ls -1d /mnt/storage/hf-cache/hub/models--nomic-ai--nomic-embed-text-v1.5/snapshots/* 2>/dev/null | head -n1)
if [ -n "$SNAP" ]; then
  mkdir -p ~/Documents/models
  ln -sfn "$SNAP" ~/Documents/models/nomic-embed-768
  ls -la ~/Documents/models/nomic-embed-768
else
  echo "No local Nomic snapshot found. Temporarily disable offline and fetch once:"
  echo '  unset TRANSFORMERS_OFFLINE HF_HUB_OFFLINE'
  echo "  python3 - << 'PY'"
  echo 'from sentence_transformers import SentenceTransformer'
  echo 'SentenceTransformer("nomic-ai/nomic-embed-text-v1.5", trust_remote_code=True, cache_folder="/mnt/storage/hf-cache")'
  echo "PY"
fi

3) Run ingest:

CUDA_VISIBLE_DEVICES=0 \
python3 ~/Documents/tokenspace/ingest_pile.py \
  --root_dir "/mnt/storage/pile_uncopyrighted"

(Optional) Make the script accept --model_dir

If you’d rather pass it from the CLI:

nano ~/Documents/tokenspace/ingest_pile.py

Add/replace near the top:

import os, argparse

MODEL_DIR_DEFAULT = "/home/darren/Documents/models/nomic-embed-768"

parser = argparse.ArgumentParser()
parser.add_argument("--root_dir", required=True)
parser.add_argument("--model_dir", default=os.environ.get("MODEL_DIR", MODEL_DIR_DEFAULT))
args = parser.parse_args()

ROOT_DIR = args.root_dir
MODEL_DIR = args.model_dir

Now you can run:

CUDA_VISIBLE_DEVICES=0 \
python3 ~/Documents/tokenspace/ingest_pile.py \
  --root_dir "/mnt/storage/pile_uncopyrighted" \
  --model_dir "/home/darren/Documents/models/nomic-embed-768"

Quick sanity checks

• Confirm the script sees the new value:
  bash
grep -n 'MODEL_DIR' ~/Documents/tokenspace/ingest_pile.py
• Confirm the directory exists:
  bash
test -d ~/Documents/models/nomic-embed-768 && echo "Model path OK" || echo "Model path MISSING"

Do the quick edit + link, and the “Path not found” error will disappear.