Conversation provenance
Source: ChatGPT data export · Conversation ID: 68d5d5fd-fe80-8333-84fb-7bf94df49416
Created: 2025-09-25T16:53:36-07:00 · Part 2 of 3
You fixed the typos, but the script still tries to load from /mnt/storage/models/nomic-embed-768, which doesn’t exist yet. Let’s create that path (as a symlink to the actual model). If the model isn’t downloaded, we’ll fetch it once, then wire everything up.
Paste this whole block (it’s safe & idempotent)
# 0) Paths we’ll use
DOCS_MODEL="$HOME/Documents/models/nomic-embed-768"
STOR_MODELS="/mnt/storage/models"
STOR_MODEL_LINK="$STOR_MODELS/nomic-embed-768"
# 1) Try to find an existing cached Nomic embedder snapshot
SNAP=""
for base in "/mnt/storage/hf-cache/hub" "$HOME/.cache/huggingface/hub"; do
CAND=$(ls -1d "$base"/models--nomic-ai--nomic-embed-text-v1.5/snapshots/* 2>/dev/null | head -n1)
if [ -n "$CAND" ]; then SNAP="$CAND"; break; fi
CAND=$(ls -1d "$base"/models--nomic-ai--nomic-bert-2048/snapshots/* 2>/dev/null | head -n1)
if [ -n "$CAND" ]; then SNAP="$CAND"; break; fi
done
# 2) If not found, fetch once (temporarily disable offline)
if [ -z "$SNAP" ]; then
echo "[info] No local Nomic snapshot found — fetching nomic-ai/nomic-embed-text-v1.5 once..."
unset TRANSFORMERS_OFFLINE HF_HUB_OFFLINE
python3 - << 'PY'
from sentence_transformers import SentenceTransformer
# Download to the storage cache so it lives on the big drive
SentenceTransformer("nomic-ai/nomic-embed-text-v1.5",
trust_remote_code=True,
cache_folder="/mnt/storage/hf-cache")
print("download_ok")
PY
# locate the new snapshot
SNAP=$(ls -1d /mnt/storage/hf-cache/hub/models--nomic-ai--nomic-embed-text-v1.5/snapshots/* 2>/dev/null | head -n1)
fi
# 3) Sanity check
if [ -z "$SNAP" ]; then
echo "[error] Still no Nomic snapshot located. Abort."
exit 1
fi
echo "[info] Using snapshot: $SNAP"
# 4) Create friendly paths in BOTH places your script might reference
mkdir -p "$HOME/Documents/models" "$STOR_MODELS"
ln -sfn "$SNAP" "$DOCS_MODEL"
ln -sfn "$SNAP" "$STOR_MODEL_LINK"
echo "[info] Linked:"
ls -la "$DOCS_MODEL" || true
ls -la "$STOR_MODEL_LINK" || true
# 5) (Re)enable offline if you want
export TRANSFORMERS_OFFLINE=1 HF_HUB_OFFLINE=1 HF_HUB_DISABLE_TELEMETRY=1
echo "[info] Offline mode re-enabled."
# 6) Kick off ingestion
CUDA_VISIBLE_DEVICES=0 \
python3 "$HOME/Documents/tokenspace/ingest_pile.py" \
--root_dir "/mnt/storage/pile_uncopyrighted"
What this does:
– Finds (or downloads) nomic-ai/nomic-embed-text-v1.5 into /mnt/storage/hf-cache.
– Exposes two stable symlinks:
– ~/Documents/models/nomic-embed-768
– /mnt/storage/models/nomic-embed-768
– Your script will succeed whether it reads MODEL_DIR from Documents or Storage.
If it still errors, paste the new traceback (it’ll be different now), and we’ll knock out the next blocker.
That's because it's not under the models directory. And it should have gone to this one, yes?mnt/storage/hf-cache/hub/models–nomic-ai–nomic-embed-text-v1.5/snapshots/e5cf08aadaa33385f5990def41f7a23405aec398
Perfect—that’s the snapshot we need. Two quick notes and we’re done:
1) Your path needs a leading slash:
/mnt/storage/hf-cache/hub/models--nomic-ai--nomic-embed-text-v1.5/snapshots/e5cf08aadaa33385f5990def41f7a23405aec398
2) Your script is already trying to load from /mnt/storage/models/nomic-embed-768, so let’s point that path at the snapshot with a symlink.
Do this exactly
# 1) Make sure the target models dir exists
sudo mkdir -p /mnt/storage/models
# 2) Link the snapshot to the path your script expects
sudo ln -sfn "/mnt/storage/hf-cache/hub/models--nomic-ai--nomic-embed-text-v1.5/snapshots/e5cf08aadaa33385f5990def41f7a23405aec398" \
/mnt/storage/models/nomic-embed-768
# 3) Verify the link resolves
ls -la /mnt/storage/models/nomic-embed-768
You should see nomic-embed-768 -> …/snapshots/e5cf0….
Run ingestion
CUDA_VISIBLE_DEVICES=0 \
python3 ~/Documents/tokenspace/ingest_pile.py \
--root_dir "/mnt/storage/pile_uncopyrighted"
That should clear the “Path not found” error, since /mnt/storage/models/nomic-embed-768 now exists and points to your snapshot. If it throws a different error, paste it and we’ll knock that one down next.
Why don't we just fix the code in the script?
Absolutely. Two ways—pick the one you like:
Option A — Quick one-liner (point to your known snapshot)
Since you already know the exact path, just set MODEL_DIR to it and be done.
# 1) backup
cp ~/Documents/tokenspace/ingest_pile.py ~/Documents/tokenspace/ingest_pile.py.bak
# 2) replace any hard-coded MODEL_DIR line with your snapshot path
sed -i 's#^MODEL_DIR *= *".*"#MODEL_DIR = "/mnt/storage/hf-cache/hub/models--nomic-ai--nomic-embed-text-v1.5/snapshots/e5cf08aadaa33385f5990def41f7a23405aec398"#' \
~/Documents/tokenspace/ingest_pile.py
# 3) run
CUDA_VISIBLE_DEVICES=0 \
python3 ~/Documents/tokenspace/ingest_pile.py \
--root_dir "/mnt/storage/pile_uncopyrighted"
Option B — Make it robust (accept --model_dir and env override)
Paste this small block once near the top of ingest_pile.py (right after the imports), and delete any older lines that hard-code MODEL_DIR/ROOT_DIR.
# --- begin configurable args ---
import os, argparse, pathlib
# Your known-good local default (edit if you ever change snapshots)
DEFAULT_MODEL_DIR = "/mnt/storage/hf-cache/hub/models--nomic-ai--nomic-embed-text-v1.5/snapshots/e5cf08aadaa33385f5990def41f7a23405aec398"
parser = argparse.ArgumentParser()
parser.add_argument("--root_dir", required=True, help="Folder with .txt/.md docs")
parser.add_argument("--model_dir",
default=os.environ.get("MODEL_DIR", DEFAULT_MODEL_DIR),
help="Local path to SentenceTransformer model OR HF repo id")
args = parser.parse_args()
ROOT_DIR = args.root_dir
MODEL_DIR = args.model_dir
print(f"[cfg] ROOT_DIR={ROOT_DIR}")
print(f"[cfg] MODEL_DIR={MODEL_DIR}")
# helpful checks
if not os.path.exists(ROOT_DIR):
raise FileNotFoundError(f"ROOT_DIR not found: {ROOT_DIR}")
if not (os.path.isdir(MODEL_DIR) or "/" not in MODEL_DIR):
# if it looks like a local path and doesn't exist, fail early with a clear message
raise FileNotFoundError(f"MODEL_DIR not found: {MODEL_DIR}")
# --- end configurable args ---
Now you can run with any of these:
# use the built-in default (your snapshot path)
CUDA_VISIBLE_DEVICES=0 \
python3 ~/Documents/tokenspace/ingest_pile.py \
--root_dir "/mnt/storage/pile_uncopyrighted"
# or point somewhere else at runtime
CUDA_VISIBLE_DEVICES=0 \
python3 ~/Documents/tokenspace/ingest_pile.py \
--root_dir "/mnt/storage/pile_uncopyrighted" \
--model_dir "/home/darren/Documents/models/nomic-embed-768"
# or via env var
export MODEL_DIR="/home/darren/Documents/models/nomic-embed-768"
CUDA_VISIBLE_DEVICES=0 \
python3 ~/Documents/tokenspace/ingest_pile.py \
--root_dir "/mnt/storage/pile_uncopyrighted"
If you want, I can also give you a tiny patch to replace all old MODEL_DIR = "..." definitions automatically while inserting the argparse block.
No sir, we've done too many edits and it's just getting worse, and I will screw this up more because none of that code matches stuff that's already in the file in a way that I can easily recognize it. So, here's the original ingest underscore pile dot py. What I'd like you to do is rewrite it with all the correct information. However, I'd like to add some other functionality. I'd like to ingest the chunk files in numeric order so that I can more easily tell where I've left off. Because right now I have no idea. I'm hoping there's a log file somewhere that'll say, oh, this one's already done because I remember we have a log file somewhere. With that being said, I'd rather see that go in numeric order, number one. Number two, I'd like it to report back to the terminal that, hey, I'm currently doing this file. And then when it's done, I'd like it to report that it was successful in posting to the database and then giving me an updated count on documents and chunks at the ends of each insert. As well as the time involved in processing, as well as inserting. Can we do that, please?
Heard. I rewrote your script with the features you asked for (numeric ordering, clear progress, per-file timing, and cumulative doc/chunk counts, plus resume behavior). Grab it here:
Download the rewritten script (ingest_pile_v2.py)
Key upgrades vs. your original fileciteturn0file0:
- Natural numeric ordering for files (so you can tell what’s next).
- Loud progress logs: “START file…”, existing-chunk count, and a final “[ok] … cum: docs=… chunks=…”.
- Timings per phase: read+chunk, metadata insert, embed, total.
- Idempotent resume: if a document already has chunks and
FORCE_REEMBED=0, it skips re-embedding (still upserts text/token counts). SetFORCE_REEMBED=1to recompute. - Console + optional logfile (
$HOME/Documents/tokenspace/ingest.logby default).
How to run (copy/paste)
# (optional) put logs here; empty LOG_FILE disables file logging
export LOG_FILE="$HOME/Documents/tokenspace/ingest.log"
# (optional) only re-embed if you really want to recompute all vectors for existing docs
export FORCE_REEMBED=0
# your usual offline flags can stay on if the model dir is local
export TRANSFORMERS_OFFLINE=1 HF_HUB_OFFLINE=1 HF_HUB_DISABLE_TELEMETRY=1
# paths (adjust if needed)
export ROOT_DIR="/mnt/storage/pile_uncopyrighted"
export MODEL_DIR="/home/darren/Documents/models/nomic-embed-768" # or your snapshot path
# DB via unix socket (adjust if your port/DB differ)
export DB_DSN="postgresql:///tokenspace?host=/var/run/postgresql&port=5433"
# run it
CUDA_VISIBLE_DEVICES=0 python3 /path/to/ingest_pile_v2.py
What you’ll see for each file
START <path>existing chunks for doc_id=…: <N>- Final line like:
[ok 17] /…/file_00123.txt -> 42 chunk(s) |
read+chunk=2.31s meta=0.78s embed=11.42s total=14.57s |
cum: docs=31 chunks=1,277,343
Notes / knobs
- Batch size:
BATCH_SIZEenv (default 64). - Chunking:
CHUNK_SIZE_CHARS(1500) andCHUNK_OVERLAP_CHARS(200) envs. - Skip vs re-embed: toggle with
FORCE_REEMBED=0/1. - Counts: It queries actual
content.documents/content.chunkscounts after each file, so your “where am I?” is always precise. - Model path: make sure
MODEL_DIRpoints at your local Nomic snapshot or symlink; the script prints it at start.
If you’d like, I can also make a matching llama.cpp/Qwen launcher script for your Desktop, same style (echo settings, clear timings).
(ougway_venv) darren@ougway:~/Desktop$ CUDA_VISIBLE_DEVICES=0 python3 /home/darren/Documents/tokenspace/ingest_pile_v2.py
[2025-09-26 16:39:52] [init] MODEL_DIR=/home/darren/Documents/models/nomic-embed-768
[2025-09-26 16:39:52] [init] ROOT_DIR=/mnt/storage/pile_uncopyrighted
[2025-09-26 16:39:52] [init] BATCH_SIZE=64 CHUNK_SIZE=1500 OVERLAP=200 FORCE_REEMBED=False
No sentence-transformers model found with name /home/darren/Documents/models/nomic-embed-768. Creating a new one with mean pooling.
Traceback (most recent call last):
File "/home/darren/Documents/tokenspace/ingest_pile_v2.py", line 296, in <module>
main()
File "/home/darren/Documents/tokenspace/ingest_pile_v2.py", line 192, in main
model = SentenceTransformer(MODEL_DIR, trust_remote_code=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/darren/Documents/ougway_venv/lib/python3.12/site-packages/sentence_transformers/SentenceTransformer.py", line 339, in init
modules = self._load_auto_model(
^^^^^^^^^^^^^^^^^^^^^^
File "/home/darren/Documents/ougway_venv/lib/python3.12/site-packages/sentence_transformers/SentenceTransformer.py", line 2112, in _load_auto_model
transformer_model = Transformer(
^^^^^^^^^^^^
File "/home/darren/Documents/ougway_venv/lib/python3.12/site-packages/sentence_transformers/models/Transformer.py", line 87, in init
config, is_peft_model = self._load_config(model_name_or_path, cache_dir, backend, config_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/darren/Documents/ougway_venv/lib/python3.12/site-packages/sentence_transformers/models/Transformer.py", line 162, in _load_config
return AutoConfig.from_pretrained(model_name_or_path, **config_args, cache_dir=cache_dir), False
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/darren/Documents/ougway_venv/lib/python3.12/site-packages/transformers/models/auto/configuration_auto.py", line 1329, in from_pretrained
raise ValueError(
ValueError: Unrecognized model in /home/darren/Documents/models/nomic-embed-768. Should have a model_type key in its config.json, or contain one of the following strings in its name: aimv2, aimv2_vision_model, albert, align, altclip, apertus, arcee, aria, aria_text, audio-spectrogram-transformer, autoformer, aya_vision, bamba, bark, bart, beit, bert, bert-generation, big_bird, bigbird_pegasus, biogpt, bit, bitnet, blenderbot, blenderbot-small, blip, blip-2, blip_2_qformer, bloom, bridgetower, bros, camembert, canine, chameleon, chinese_clip, chinese_clip_vision_model, clap, clip, clip_text_model, clip_vision_model, clipseg, clvp, code_llama, codegen, cohere, cohere2, cohere2_vision, colpali, colqwen2, conditional_detr, convbert, convnext, convnextv2, cpmant, csm, ctrl, cvt, d_fine, dab-detr, dac, data2vec-audio, data2vec-text, data2vec-vision, dbrx, deberta, deberta-v2, decision_transformer, deepseek_v2, deepseek_v3, deepseek_vl, deepseek_vl_hybrid, deformable_detr, deit, depth_anything, depth_pro, deta, detr, dia, diffllama, dinat, dinov2, dinov2_with_registers, dinov3_convnext, dinov3_vit, distilbert, doge, donut-swin, dots1, dpr, dpt, efficientformer, efficientloftr, efficientnet, electra, emu3, encodec, encoder-decoder, eomt, ernie, ernie4_5, ernie4_5_moe, ernie_m, esm, evolla, exaone4, falcon, falcon_h1, falcon_mamba, fastspeech2_conformer, fastspeech2_conformer_with_hifigan, flaubert, flava, florence2, fnet, focalnet, fsmt, funnel, fuyu, gemma, gemma2, gemma3, gemma3_text, gemma3n, gemma3n_audio, gemma3n_text, gemma3n_vision, git, glm, glm4, glm4_moe, glm4v, glm4v_moe, glm4v_moe_text, glm4v_text, glpn, got_ocr2, gpt-sw3, gpt2, gpt_bigcode, gpt_neo, gpt_neox, gpt_neox_japanese, gpt_oss, gptj, gptsan-japanese, granite, granite_speech, granitemoe, granitemoehybrid, granitemoeshared, granitevision, graphormer, grounding-dino, groupvit, helium, hgnet_v2, hiera, hubert, hunyuan_v1_dense, hunyuan_v1_moe, ibert, idefics, idefics2, idefics3, idefics3_vision, ijepa, imagegpt, informer, instructblip, instructblipvideo, internvl, internvl_vision, jamba, janus, jetmoe, jukebox, kosmos-2, kosmos-2.5, kyutai_speech_to_text, layoutlm, layoutlmv2, layoutlmv3, led, levit, lfm2, lightglue, lilt, llama, llama4, llama4_text, llava, llava_next, llava_next_video, llava_onevision, longformer, longt5, luke, lxmert, m2m_100, mamba, mamba2, marian, markuplm, mask2former, maskformer, maskformer-swin, mbart, mctct, mega, megatron-bert, metaclip_2, mgp-str, mimi, minimax, mistral, mistral3, mixtral, mlcd, mllama, mm-grounding-dino, mobilebert, mobilenet_v1, mobilenet_v2, mobilevit, mobilevitv2, modernbert, modernbert-decoder, moonshine, moshi, mpnet, mpt, mra, mt5, musicgen, musicgen_melody, mvp, nat, nemotron, nezha, nllb-moe, nougat, nystromformer, olmo, olmo2, olmoe, omdet-turbo, oneformer, open-llama, openai-gpt, opt, ovis2, owlv2, owlvit, paligemma, patchtsmixer, patchtst, pegasus, pegasus_x, perceiver, perception_encoder, perception_lm, persimmon, phi, phi3, phi4_multimodal, phimoe, pix2struct, pixtral, plbart, poolformer, pop2piano, prompt_depth_anything, prophetnet, pvt, pvt_v2, qdqbert, qwen2, qwen2_5_omni, qwen2_5_vl, qwen2_5_vl_text, qwen2_audio, qwen2_audio_encoder, qwen2_moe, qwen2_vl, qwen2_vl_text, qwen3, qwen3_moe, rag, realm, recurrent_gemma, reformer, regnet, rembert, resnet, retribert, roberta, roberta-prelayernorm, roc_bert, roformer, rt_detr, rt_detr_resnet, rt_detr_v2, rwkv, sam, sam2, sam2_hiera_det_model, sam2_video, sam2_vision_model, sam_hq, sam_hq_vision_model, sam_vision_model, seamless_m4t, seamless_m4t_v2, seed_oss, segformer, seggpt, sew, sew-d, shieldgemma2, siglip, siglip2, siglip_vision_model, smollm3, smolvlm, smolvlm_vision, speech-encoder-decoder, speech_to_text, speech_to_text_2, speecht5, splinter, squeezebert, stablelm, starcoder2, superglue, superpoint, swiftformer, swin, swin2sr, swinv2, switch_transformers, t5, t5gemma, table-transformer, tapas, textnet, time_series_transformer, timesfm, timesformer, timm_backbone, timm_wrapper, trajectory_transformer, transfo-xl, trocr, tvlt, tvp, udop, umt5, unispeech, unispeech-sat, univnet, upernet, van, video_llava, videomae, vilt, vipllava, vision-encoder-decoder, vision-text-dual-encoder, visual_bert, vit, vit_hybrid, vit_mae, vit_msn, vitdet, vitmatte, vitpose, vitpose_backbone, vits, vivit, vjepa2, voxtral, voxtral_encoder, wav2vec2, wav2vec2-bert, wav2vec2-conformer, wavlm, whisper, xclip, xcodec, xglm, xlm, xlm-prophetnet, xlm-roberta, xlm-roberta-xl, xlnet, xlstm, xmod, yolos, yoso, zamba, zamba2, zoedepth
(ougway_venv) darren@ougway:~/Desktop$
Okay, apparently I'm not capable of typing in the right fucking pathing. So the ingest… PileVersion2.py is in HomeDocuments token space. Can you please adjust the command line for me so it will run?
Got it — the script is at:
/home/darren/Documents/tokenspace/ingest_pile_v2.py
and your local model is really at:
/mnt/storage/hf-cache/hub/models--nomic-ai--nomic-embed-text-v1.5/snapshots/e5cf08aadaa33385f5990def41f7a23405aec398
So let’s point directly there. Run this:
CUDA_VISIBLE_DEVICES=0 \
MODEL_DIR="/mnt/storage/hf-cache/hub/models--nomic-ai--nomic-embed-text-v1.5/snapshots/e5cf08aadaa33385f5990def41f7a23405aec398" \
python3 /home/darren/Documents/tokenspace/ingest_pile_v2.py \
--root_dir "/mnt/storage/pile_uncopyrighted"
That way:
– The script path is correct (your v2 file).
– MODEL_DIR env overrides the default, so it doesn’t look for /home/darren/Documents/models/nomic-embed-768.
👉 If you want to avoid typing the long snapshot path every time, I recommend making a clean symlink once:
mkdir -p /home/darren/Documents/models
ln -sfn "/mnt/storage/hf-cache/hub/models--nomic-ai--nomic-embed-text-v1.5/snapshots/e5cf08aadaa33385f5990def41f7a23405aec398" \
/home/darren/Documents/models/nomic-embed-768
Then you can run the shorter command:
CUDA_VISIBLE_DEVICES=0 \
MODEL_DIR="/home/darren/Documents/models/nomic-embed-768" \
python3 /home/darren/Documents/tokenspace/ingest_pile_v2.py \
--root_dir "/mnt/storage/pile_uncopyrighted"
Do you want me to also make you a little run_ingest.sh script on your Desktop so you just double-click or type ./run_ingest.sh instead of retyping this whole command each time?
(ougway_venv) darren@ougway:~/Desktop$ mkdir -p /home/darren/Documents/models
ln -sfn "/mnt/storage/hf-cache/hub/models–nomic-ai–nomic-embed-text-v1.5/snapshots/e5cf08aadaa33385f5990def41f7a23405aec398" \
/home/darren/Documents/models/nomic-embed-768
(ougway_venv) darren@ougway:~/Desktop$ CUDA_VISIBLE_DEVICES=0 \
MODEL_DIR="/home/darren/Documents/models/nomic-embed-768" \
python3 /home/darren/Documents/tokenspace/ingest_pile_v2.py \
–root_dir "/mnt/storage/pile_uncopyrighted"
[2025-09-26 16:41:32] [init] MODEL_DIR=/home/darren/Documents/models/nomic-embed-768
[2025-09-26 16:41:32] [init] ROOT_DIR=/mnt/storage/pile_uncopyrighted
[2025-09-26 16:41:32] [init] BATCH_SIZE=64 CHUNK_SIZE=1500 OVERLAP=200 FORCE_REEMBED=False
No sentence-transformers model found with name /home/darren/Documents/models/nomic-embed-768. Creating a new one with mean pooling.
Traceback (most recent call last):
File "/home/darren/Documents/tokenspace/ingest_pile_v2.py", line 296, in <module>
main()
File "/home/darren/Documents/tokenspace/ingest_pile_v2.py", line 192, in main
model = SentenceTransformer(MODEL_DIR, trust_remote_code=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/darren/Documents/ougway_venv/lib/python3.12/site-packages/sentence_transformers/SentenceTransformer.py", line 339, in init
modules = self._load_auto_model(
^^^^^^^^^^^^^^^^^^^^^^
File "/home/darren/Documents/ougway_venv/lib/python3.12/site-packages/sentence_transformers/SentenceTransformer.py", line 2112, in _load_auto_model
transformer_model = Transformer(
^^^^^^^^^^^^
File "/home/darren/Documents/ougway_venv/lib/python3.12/site-packages/sentence_transformers/models/Transformer.py", line 87, in init
config, is_peft_model = self._load_config(model_name_or_path, cache_dir, backend, config_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/darren/Documents/ougway_venv/lib/python3.12/site-packages/sentence_transformers/models/Transformer.py", line 162, in _load_config
return AutoConfig.from_pretrained(model_name_or_path, **config_args, cache_dir=cache_dir), False
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/darren/Documents/ougway_venv/lib/python3.12/site-packages/transformers/models/auto/configuration_auto.py", line 1329, in from_pretrained
raise ValueError(
ValueError: Unrecognized model in /home/darren/Documents/models/nomic-embed-768. Should have a model_type key in its config.json, or contain one of the following strings in its name: aimv2, aimv2_vision_model, albert, align, altclip, apertus, arcee, aria, aria_text, audio-spectrogram-transformer, autoformer, aya_vision, bamba, bark, bart, beit, bert, bert-generation, big_bird, bigbird_pegasus, biogpt, bit, bitnet, blenderbot, blenderbot-small, blip, blip-2, blip_2_qformer, bloom, bridgetower, bros, camembert, canine, chameleon, chinese_clip, chinese_clip_vision_model, clap, clip, clip_text_model, clip_vision_model, clipseg, clvp, code_llama, codegen, cohere, cohere2, cohere2_vision, colpali, colqwen2, conditional_detr, convbert, convnext, convnextv2, cpmant, csm, ctrl, cvt, d_fine, dab-detr, dac, data2vec-audio, data2vec-text, data2vec-vision, dbrx, deberta, deberta-v2, decision_transformer, deepseek_v2, deepseek_v3, deepseek_vl, deepseek_vl_hybrid, deformable_detr, deit, depth_anything, depth_pro, deta, detr, dia, diffllama, dinat, dinov2, dinov2_with_registers, dinov3_convnext, dinov3_vit, distilbert, doge, donut-swin, dots1, dpr, dpt, efficientformer, efficientloftr, efficientnet, electra, emu3, encodec, encoder-decoder, eomt, ernie, ernie4_5, ernie4_5_moe, ernie_m, esm, evolla, exaone4, falcon, falcon_h1, falcon_mamba, fastspeech2_conformer, fastspeech2_conformer_with_hifigan, flaubert, flava, florence2, fnet, focalnet, fsmt, funnel, fuyu, gemma, gemma2, gemma3, gemma3_text, gemma3n, gemma3n_audio, gemma3n_text, gemma3n_vision, git, glm, glm4, glm4_moe, glm4v, glm4v_moe, glm4v_moe_text, glm4v_text, glpn, got_ocr2, gpt-sw3, gpt2, gpt_bigcode, gpt_neo, gpt_neox, gpt_neox_japanese, gpt_oss, gptj, gptsan-japanese, granite, granite_speech, granitemoe, granitemoehybrid, granitemoeshared, granitevision, graphormer, grounding-dino, groupvit, helium, hgnet_v2, hiera, hubert, hunyuan_v1_dense, hunyuan_v1_moe, ibert, idefics, idefics2, idefics3, idefics3_vision, ijepa, imagegpt, informer, instructblip, instructblipvideo, internvl, internvl_vision, jamba, janus, jetmoe, jukebox, kosmos-2, kosmos-2.5, kyutai_speech_to_text, layoutlm, layoutlmv2, layoutlmv3, led, levit, lfm2, lightglue, lilt, llama, llama4, llama4_text, llava, llava_next, llava_next_video, llava_onevision, longformer, longt5, luke, lxmert, m2m_100, mamba, mamba2, marian, markuplm, mask2former, maskformer, maskformer-swin, mbart, mctct, mega, megatron-bert, metaclip_2, mgp-str, mimi, minimax, mistral, mistral3, mixtral, mlcd, mllama, mm-grounding-dino, mobilebert, mobilenet_v1, mobilenet_v2, mobilevit, mobilevitv2, modernbert, modernbert-decoder, moonshine, moshi, mpnet, mpt, mra, mt5, musicgen, musicgen_melody, mvp, nat, nemotron, nezha, nllb-moe, nougat, nystromformer, olmo, olmo2, olmoe, omdet-turbo, oneformer, open-llama, openai-gpt, opt, ovis2, owlv2, owlvit, paligemma, patchtsmixer, patchtst, pegasus, pegasus_x, perceiver, perception_encoder, perception_lm, persimmon, phi, phi3, phi4_multimodal, phimoe, pix2struct, pixtral, plbart, poolformer, pop2piano, prompt_depth_anything, prophetnet, pvt, pvt_v2, qdqbert, qwen2, qwen2_5_omni, qwen2_5_vl, qwen2_5_vl_text, qwen2_audio, qwen2_audio_encoder, qwen2_moe, qwen2_vl, qwen2_vl_text, qwen3, qwen3_moe, rag, realm, recurrent_gemma, reformer, regnet, rembert, resnet, retribert, roberta, roberta-prelayernorm, roc_bert, roformer, rt_detr, rt_detr_resnet, rt_detr_v2, rwkv, sam, sam2, sam2_hiera_det_model, sam2_video, sam2_vision_model, sam_hq, sam_hq_vision_model, sam_vision_model, seamless_m4t, seamless_m4t_v2, seed_oss, segformer, seggpt, sew, sew-d, shieldgemma2, siglip, siglip2, siglip_vision_model, smollm3, smolvlm, smolvlm_vision, speech-encoder-decoder, speech_to_text, speech_to_text_2, speecht5, splinter, squeezebert, stablelm, starcoder2, superglue, superpoint, swiftformer, swin, swin2sr, swinv2, switch_transformers, t5, t5gemma, table-transformer, tapas, textnet, time_series_transformer, timesfm, timesformer, timm_backbone, timm_wrapper, trajectory_transformer, transfo-xl, trocr, tvlt, tvp, udop, umt5, unispeech, unispeech-sat, univnet, upernet, van, video_llava, videomae, vilt, vipllava, vision-encoder-decoder, vision-text-dual-encoder, visual_bert, vit, vit_hybrid, vit_mae, vit_msn, vitdet, vitmatte, vitpose, vitpose_backbone, vits, vivit, vjepa2, voxtral, voxtral_encoder, wav2vec2, wav2vec2-bert, wav2vec2-conformer, wavlm, whisper, xclip, xcodec, xglm, xlm, xlm-prophetnet, xlm-roberta, xlm-roberta-xl, xlnet, xlstm, xmod, yolos, yoso, zamba, zamba2, zoedepth
(ougway_venv) darren@ougway:~/Desktop$
You’re super close—the error means the folder at ~/Documents/models/nomic-embed-768 isn’t a complete, recognized model (missing a proper config.json / modules.json, etc.). Easiest fix: point the script to the repo id so sentence-transformers can fetch the right files once, cache them to your big drive, and then you’re fully offline again.
One-time “online” fetch (recommended)
# 1) Allow one fetch and make sure cache goes to the big SSD
unset TRANSFORMERS_OFFLINE HF_HUB_OFFLINE
export HF_HOME=/mnt/storage/hf-cache
# 2) Run the new script with the REPO ID (no slashes in a path = we treat it as a repo)
CUDA_VISIBLE_DEVICES=0 \
MODEL_DIR="nomic-ai/nomic-embed-text-v1.5" \
python3 /home/darren/Documents/tokenspace/ingest_pile_v2.py \
--root_dir "/mnt/storage/pile_uncopyrighted"
- On first run it will download to
/mnt/storage/hf-cache. - After that, you can go fully offline again:
bash
export TRANSFORMERS_OFFLINE=1 HF_HUB_OFFLINE=1 HF_HUB_DISABLE_TELEMETRY=1
If you really want to stick with a local path
Your symlinked snapshot looks partial. Check it:
ls -la /home/darren/Documents/models/nomic-embed-768
ls -la /mnt/storage/hf-cache/hub/models--nomic-ai--nomic-embed-text-v1.5/snapshots/e5cf08aadaa33385f5990def41f7a23405aec398 | head
If you don’t see a valid config.json (with a "model_type" key) and typical model files, just force a complete local download and relink:
unset TRANSFORMERS_OFFLINE HF_HUB_OFFLINE
python3 - << 'PY'
from huggingface_hub import snapshot_download
p = snapshot_download("nomic-ai/nomic-embed-text-v1.5", cache_dir="/mnt/storage/hf-cache", local_files_only=False)
print("Snapshot:", p)
PY
# update the link (now points to a complete snapshot)
SNAP=$(ls -1d /mnt/storage/hf-cache/hub/models--nomic-ai--nomic-embed-text-v1.5/snapshots/* | head -n1)
ln -sfn "$SNAP" /home/darren/Documents/models/nomic-embed-768
# re-enable offline if you want
export TRANSFORMERS_OFFLINE=1 HF_HUB_OFFLINE=1 HF_HUB_DISABLE_TELEMETRY=1
# run using the local path
CUDA_VISIBLE_DEVICES=0 \
MODEL_DIR="/home/darren/Documents/models/nomic-embed-768" \
python3 /home/darren/Documents/tokenspace/ingest_pile_v2.py \
--root_dir "/mnt/storage/pile_uncopyrighted"
Either route will resolve the “Unrecognized model” error. The simplest is the first: use MODEL_DIR="nomic-ai/nomic-embed-text-v1.5" once to seed the cache, then flip offline back on.
darren@ougway:~/Desktop$ ls -la /home/darren/Documents/models/nomic-embed-768
ls -la /mnt/storage/hf-cache/hub/models–nomic-ai–nomic-embed-text-v1.5/snapshots/e5cf08aadaa33385f5990def41f7a23405aec398 | head
lrwxrwxrwx 1 darren darren 116 Sep 26 16:41 /home/darren/Documents/models/nomic-embed-768 -> /mnt/storage/hf-cache/hub/models–nomic-ai–nomic-embed-text-v1.5/snapshots/e5cf08aadaa33385f5990def41f7a23405aec398
total 16
drwxrwxr-x 3 darren darren 4096 Sep 26 16:42 .
drwxrwxr-x 3 darren darren 4096 Sep 26 16:15 ..
drwxrwxr-x 2 darren darren 4096 Sep 26 16:42 1_Pooling
lrwxrwxrwx 1 darren darren 52 Sep 26 16:42 config.json -> ../../blobs/c452fcbc4853b2aec17b6a5865ae4681249c8de1
lrwxrwxrwx 1 darren darren 52 Sep 26 16:42 config_sentence_transformers.json -> ../../blobs/f370ff81cd1305bc3e2449d9d18d468af50b6396
lrwxrwxrwx 1 darren darren 76 Sep 26 16:16 model.safetensors -> ../../blobs/9e7d262b1fe5ea350782829496efa831901b77486bbde1cea54a4c822d010d5c
lrwxrwxrwx 1 darren darren 52 Sep 26 16:42 modules.json -> ../../blobs/45a8beea3dc567c0172999f5eca5c9e181a4ae15
lrwxrwxrwx 1 darren darren 52 Sep 26 16:42 README.md -> ../../blobs/67996f4500673da515aebf5a5da96e15f1835d02
lrwxrwxrwx 1 darren darren 52 Sep 26 16:42 sentence_bert_config.json -> ../../blobs/3f14bcb9592c5eb25953f7710d656431dd6fad5d
darren@ougway:~/Desktop$
Nice—thanks for the ls. That explains the crash:
- Your snapshot only has
1_Pooling/and some symlinked configs. - It’s missing the
0_Transformer/directory (the actual encoder weights + tokenizer). - When
SentenceTransformertries to load the Transformer block, it can’t find a properconfig.jsonwith a"model_type", so it errors.
Let’s fix it by pulling a complete copy of nomic-ai/nomic-embed-text-v1.5 into your HF cache and pointing your script at it.
Do this exactly
# 1) Temporarily allow online fetch and ensure cache is on the 4TB drive
unset TRANSFORMERS_OFFLINE HF_HUB_OFFLINE
export HF_HOME=/mnt/storage/hf-cache
# 2) Force a complete snapshot download into your cache
python3 - << 'PY'
from huggingface_hub import snapshot_download
p = snapshot_download(
"nomic-ai/nomic-embed-text-v1.5",
cache_dir="/mnt/storage/hf-cache",
local_files_only=False,
ignore_patterns=["*.pt"], # keep it lean; safetensors is enough
)
print("Snapshot:", p)
PY
# 3) Update your friendly link to point at the *fresh* snapshot
SNAP=$(ls -1d /mnt/storage/hf-cache/hub/models--nomic-ai--nomic-embed-text-v1.5/snapshots/* | tail -n1)
ln -sfn "$SNAP" /home/darren/Documents/models/nomic-embed-768
# 4) Verify the structure now has BOTH modules
ls -la /home/darren/Documents/models/nomic-embed-768 | sed -n '1,120p'
ls -la /home/darren/Documents/models/nomic-embed-768/0_Transformer | head
ls -la /home/darren/Documents/models/nomic-embed-768/1_Pooling | head
You should see a 0_Transformer/ directory with things like config.json, tokenizer.json, vocab.json/merges.txt (or spiece.model, depending on the base model), and model files.
Run ingest
# (optional) go back to offline if you like
export TRANSFORMERS_OFFLINE=1 HF_HUB_OFFLINE=1 HF_HUB_DISABLE_TELEMETRY=1
# kick off ingestion
CUDA_VISIBLE_DEVICES=0 \
MODEL_DIR="/home/darren/Documents/models/nomic-embed-768" \
python3 /home/darren/Documents/tokenspace/ingest_pile_v2.py \
--root_dir "/mnt/storage/pile_uncopyrighted"
If anything still complains, paste the first 30 lines of:
ls -la /home/darren/Documents/models/nomic-embed-768/0_Transformer
cat /home/darren/Documents/models/nomic-embed-768/0_Transformer/config.json | head -n 30
…but I expect this will resolve it. The missing 0_Transformer was the culprit.
Okay, well the machine just reported a giant error, so I'm going to have to restart before we do anything else.
Got it. Reboot first. When you’re back up, here’s a tight checklist to resume cleanly without re-debugging everything:
After restart — quick sanity checks
# 1) Is the 4TB drive mounted where we expect?
ls -la /mnt/storage | head
# 2) Is your home symlink handy (optional GUI access)?
ls -la ~/Storage
# 3) Model snapshot looks complete? (should have 0_Transformer + 1_Pooling)
ls -la /home/darren/Documents/models/nomic-embed-768
ls -la /home/darren/Documents/models/nomic-embed-768/0_Transformer | head
Environment + venv
# 4) Activate your venv
source ~/Documents/ougway_venv/bin/activate
# 5) Offline/telemetry flags (set them how you like)
export TRANSFORMERS_OFFLINE=1 HF_HUB_OFFLINE=1 HF_HUB_DISABLE_TELEMETRY=1
export HF_HOME=/mnt/storage/hf-cache
# 6) (optional) log file location
export LOG_FILE="$HOME/Documents/tokenspace/ingest.log"
# 7) Paths
export MODEL_DIR="/home/darren/Documents/models/nomic-embed-768"
export ROOT_DIR="/mnt/storage/pile_uncopyrighted"
export DB_DSN="postgresql:///tokenspace?host=/var/run/postgresql&port=5433"
Run ingestion
CUDA_VISIBLE_DEVICES=0 \
python3 /home/darren/Documents/tokenspace/ingest_pile_v2.py \
--root_dir "$ROOT_DIR"
If it throws another “giant error”
Grab these right away so we can pinpoint it fast:
# last ~60 lines of your ingest log (if set)
tail -n 60 "$HOME/Documents/tokenspace/ingest.log"
# terminal traceback already shows Python-side issues
# system/kernel hints (disk/GPU/driver)
dmesg | tail -n 100
# GPU sanity
nvidia-smi
If the model folder still lacks 0_Transformer, we’ll just do the one-time complete fetch again (2 commands), but the checklist above should get you rolling as soon as the machine is back.
And this is downloading a shitload of stuff this time, whereas before it said it downloaded a couple things then went away. How many times are we going to have to download the save tensors and all this other stuff? I mean, doesn't it stay in the cache or does it get wiped?
Good question — here’s the straight answer:
-
Yes, it’s supposed to stay.
When you pull a model likenomic-embed-text-v1.5with Hugging Face, it saves into your cache directory (HF_HOME=/mnt/storage/hf-cache). That includes all the big pieces:
•config.json
•pytorch_model.bin/model.safetensors
• tokenizer files
• pooling module -
Why it seems to re-download:
1. If the cache folder isn’t consistent (wrong symlink, different env vars),transformersthinks the model isn’t there and goes back to Hugging Face.
2. If the previous download got interrupted, you’ll see it fetching blobs again.
3. Sometimessentence-transformersalso creates its own wrapper layer (that1_Poolingfolder you saw). That can trigger an extra fetch the first time. -
Once it’s cached, it stays — unless:
• You clear~/.cache/huggingfaceor/mnt/storage/hf-cache.
• You reinstall the venv and wipe the cache path.
• You use a different path inMODEL_DIRthan the actual cached folder.
🔑 In your case: now that we pinned HF_HOME to /mnt/storage/hf-cache and symlinked nomic-embed-768, all future runs should reuse the same safetensors + config without pulling them again.
👉 Do you want me to show you how to do a quick verification step (ls -lh on the model folder + check file sizes) so you’ll know the cache is complete, and Hugging Face won’t try to redownload next time?
So if it has it in the cache, it doesn't go out looking the hugging face. And if it doesn't find it, if it does find it, it just stays local, yes?
Exactly 👍
- If the full set of model files is in your cache (
HF_HOMEor~/.cache/huggingface), it stays local. - On load,
transformersandsentence-transformersfirst check your cache. - If everything’s there and the checksums match → it never goes out to Hugging Face.
- If something’s missing or corrupted → then it tries to fetch just that piece.
So once the cache is complete and your MODEL_DIR points to it correctly, all runs are offline, local-only.
Do you want me to give you the exact ls -lh command to confirm all the expected files (config, tokenizer, safetensors) are sitting in your cached nomic-embed-768 directory?
So let's not force HuggingFace not to check itself. There's no need.As long as we have everything downloaded locally in the first place. So let's just make sure that's the case, that everything's downloaded locally, so that doesn't need to go out and do anything, and then we'll be all set, I think, yes?
Yes — that’s the right approach. Don’t block Hugging Face; just make sure the cache is complete so it never needs to reach out.
Here’s how to confirm everything is already downloaded:
# Look inside your cached model directory
ls -lh /home/darren/Documents/models/nomic-embed-768
For a complete local model you should see at least these:
– config.json
– sentence_bert_config.json (specific to SentenceTransformers)
– config_sentence_transformers.json
– modules.json
– model.safetensors (the actual weights, often big, multiple GB)
– README.md (not critical, but usually present)
– any subdirs like 1_Pooling/ (for pooling layer info).
If all those are there and the safetensors file has a size in the GBs, then you’re good — the cache is complete.
👉 Next run won’t download again. It will just load from disk.
Do you want me to draft a little “sanity check” one-liner that not only lists but also verifies file sizes and highlights if any of those core files are missing?
Well, I think we should run the download one more time once I reboot. This machine is doing some strange things right now. So, we'll do the reboot and try the download for Hugging Face one more time.
That’s a solid plan 👍.
After your reboot, here’s the cleanest way to re-pull Hugging Face’s embedder and ensure the cache is fully populated:
# Reactivate venv if needed
source ~/Documents/ougway_venv/bin/activate
# Run once to force HF to fill the cache
python3 -c "from sentence_transformers import SentenceTransformer; \
SentenceTransformer('nomic-ai/nomic-embed-text-v1.5', cache_folder='/home/darren/Documents/models/nomic-embed-768')"
That command will:
– Go to Hugging Face if anything is missing,
– Download all required config + weights,
– Save them straight into /home/darren/Documents/models/nomic-embed-768.
After that, your ingest script will always load locally without trying to fetch again.
⚠️ Key detail: make sure you don’t have both /mnt/storage/... and /home/darren/Documents/models/... fighting each other. Let’s keep one canonical path (~/Documents/models/nomic-embed-768) and stick with that everywhere.
Want me to also show you how to check your Hugging Face cache size so you’ll know for sure nothing is missing?
Okay, well, I don't know, it looks like this project might be dead in the water, bucko. I don't think anybody wants me to finish this. So, I went ahead and rebooted, and as soon as I sign in, the entire system freezes. Right after I type in my password and hit enter. How do I recover from this?
Got it — don’t panic yet, Darren. When Ubuntu freezes right after login, it’s usually one of these:
- GPU driver crash (most common, especially with NVIDIA).
- Corrupt config in your home directory (e.g.
.Xauthority,.gnome,.config/…). - Filesystem errors on reboot.
Here’s how you can recover step-by-step:
1. Boot into a root shell (recovery mode)
- Reboot, hold Shift (or Esc) to get the GRUB menu.
- Choose Advanced options for Ubuntu → then pick the kernel with (recovery mode).
- In the recovery menu, pick root – Drop to root shell prompt.
From there you’ll be in a root shell.
2. Check filesystem health
mount -o remount,rw /
fsck -f /dev/sdb # check your storage drive
fsck -f /dev/sda1 # check your root partition (replace with actual)
Fix any reported errors.
3. Reset graphics if needed
Sometimes the NVIDIA driver update causes a hard lock. To back out:
apt purge nvidia-*
ubuntu-drivers autoinstall
That will reinstall a stable NVIDIA driver.
4. Clean up possible bad configs
Corrupted X/Wayland login files can also hang the session. From the root shell (or by pressing Ctrl+Alt+F3 to get a TTY after boot):
mv /home/darren/.Xauthority /home/darren/.Xauthority.bak
mv /home/darren/.config /home/darren/.config.bak
(Don’t worry, .config will regenerate — you won’t lose system files, only desktop/app preferences.)
5. Try logging in again
Reboot with:
reboot
Then see if it lets you into the desktop.
⚡ If it’s still locking up, we can temporarily force Ubuntu to boot into a text console only (no graphics) so you can work on it.
Do you want me to show you how to set that up, so you can at least boot in and repair without the GUI crashing?
Shift or escape, do nothing.
Got it — that happens when GRUB is set to “hidden” on single-boot installs. We can still reach recovery mode, but it takes a slightly different trick. Here’s what you do:
1. Force GRUB to show
- Power on the machine.
- As soon as the BIOS splash screen disappears (before Ubuntu logo), hold down the
Shiftkey (BIOS systems) or tapEscrepeatedly (UEFI systems). - If nothing shows, try pressing
EscorShiftrepeatedly right after powering on, not just once. Timing matters.
If GRUB still won’t show, we’ll temporarily force it:
- Boot the machine. When it freezes at the login screen, press
Ctrl+Alt+F3to switch to a text console (TTY).
– If that works, you can log in as your user and we can fix GRUB so it always shows.
– If it doesn’t work and you’re truly locked out, we’ll need a Live USB.
2. If you can reach a TTY (Ctrl+Alt+F3)
Log in, then edit GRUB:
sudo nano /etc/default/grub
Change:
GRUB_TIMEOUT_STYLE=hidden
to:
GRUB_TIMEOUT_STYLE=menu
and make sure:
GRUB_TIMEOUT=10
Save, then run:
sudo update-grub
Next reboot, you’ll see the GRUB menu and can choose Advanced Options → Recovery Mode.
3. If TTY does not work
You’ll need to boot with a Ubuntu Live USB, mount your root partition, and then chroot in to fix drivers or configs. That’s a bit more involved, but doable.
👉 Question for you: when it freezes after login, can you still press Ctrl+Alt+F3 and get a black screen asking for your username? Or is the whole system truly frozen with no keyboard response?
And none of that works. So this time when I logged in, the mouse is not frozen, but the screen is basically blank.It does not respond to any key commands. CTRL-ALT-F3 gets me nowhere.
