You didn’t do anything wrong. The paste got mangled mid-here-doc (your terminal dropped characters and then you accidentally closed the heredoc early). Let’s bypass that entirely and have Python write the whole file for you in one shot.

Do this (exactly as-is) at your bash prompt

“`
python – <<'PY'
import os, stat

content = r"""#!/usr/bin/env python3

Offline RAG runner with two modes:

– default: single-shot answer (non-interactive)

– –chat : build RAG prompt, then drop into llama.cpp REPL (interactive)

Wide-open GPU defaults (32k ctx, full offload) are controlled by env vars

and set to sane values below. You can override at runtime if needed.

import os, sys, subprocess, hashlib, numpy as np, psycopg2, argparse

———————–

Config / Environment

———————–

DSN = os.environ.get("TS_DSN", "dbname=tokenspace user=darren host=localhost password=LittleOugway")
EMBED_MODEL = os.environ.get("EMBED_MODEL", "nomic-ai/nomic-embed-text-v1.5")
TOP_K = int(os.environ.get("RAG_K", "6"))
MAX_CHARS = int(os.environ.get("RAG_MAX_CHARS", "900")) # per chunk in prompt

Llama defaults — wide-open GPU (override with env if needed)

LLAMA_BIN = os.path.expanduser(os.environ.get("LLAMA_BIN", "~/ougway_env/llama.cpp/build/bin/llama-cli"))
LLAMA_CTX = os.environ.get("LLAMA_CTX", "32768") # 32k
LLAMA_NGL = os.environ.get("LLAMA_NGL", "999") # offload as many layers as fit
LLAMA_BATCH = os.environ.get("LLAMA_BATCH","32") # conservative for 32k on 12 GB
LLAMA_T = os.environ.get("LLAMA_T", str(os.cpu_count() or 4))
LLAMA_NTOK = os.environ.get("LLAMA_N", "512") # max tokens to generate

———————–

Embedding (offline-first)

———————–

_USE_ST = False
try:
# Try local SentenceTransformer; falls back to deterministic 768-d hash
from sentence_transformers import SentenceTransformer
_st = SentenceTransformer(EMBED_MODEL, trust_remote_code=True)
dim = getattr(_st, "get_sentence_embedding_dimension", lambda: None)()
_USE_ST = (dim == 768)
except Exception:
_USE_ST = False

def embed_one(text: str) -> np.ndarray:
if _USE_ST:
v = _st.encode([text], normalize_embeddings=True)
v = np.asarray(v, dtype=np.float32)
return v[0]
# Deterministic 768-d hash embed (no internet needed)
h = hashlib.sha256(text.encode("utf-8")).digest()
seed = int.from_bytes(h[:8], "big") % (2**31 – 1)
rng = np.random.default_rng(seed)
v = rng.normal(0.0, 1.0, 768).astype(np.float32)
n = float(np.linalg.norm(v))
if n > 0:
v /= n
return v

def vec_literal(v: np.ndarray) -> str:
return "[" + ",".join(f"{x:.6f}" for x in v.tolist()) + "]"

———————–

Retrieval

———————–

def retrieve(query_vec_lit: str, k: int):
conn = psycopg2.connect(DSN)
cur = conn.cursor()
cur.execute(f"""
SELECT ch.chunk_id,
left(ch.text, {MAX_CHARS}) AS preview,
d.title,
s.uri,
ch.embedding <=> %s::vector AS dist
FROM content.chunks ch
JOIN content.documents d ON d.doc_id = ch.doc_id
LEFT JOIN content.sources s ON s.source_id = d.source_id
ORDER BY dist ASC
LIMIT %s;
""", (query_vec_lit, k))
rows = cur.fetchall()
cur.close()
conn.close()
return rows

———————–

Prompt build

———————–

def build_prompt(question: str, hits):
ctx_blocks = []
for i, (cid, preview, title, uri, dist) in enumerate(hits, 1):
meta = []
if title: meta.append(f"title: {title}")
if uri: meta.append(f"path: {uri}")
meta.append(f"chunk_id: {cid} dist: {dist:.4f}")
ctx_blocks.append(f"### Context {i} ({' | '.join(meta)})\n{preview}")
context = "\n\n".join(ctx_blocks)
sys_inst = (
"You are a concise assistant. Use the provided context only. "
"If the answer is not in the context, say you don't find it."
)
prompt = (
"[SYSTEM]\n" + sys_inst + "\n\n" +
"[CONTEXT]\n" + context + "\n\n" +
"[USER]\n" + question + "\n\n" +
"[ASSISTANT]\n"
)
return prompt

———————–

Llama call

———————–

def call_llama_cpp(model_path: str, prompt: str, chat_mode: bool) -> str | int:
cmd = [
LLAMA_BIN,
"-m", model_path,
"-p", prompt,
"-n", LLAMA_NTOK,
"–temp", "0.2",
"–top_p", "0.9",
"-c", LLAMA_CTX,
"-ngl", LLAMA_NGL,
"-t", LLAMA_T,
"-b", LLAMA_BATCH,
]
if chat_mode:
return subprocess.call(cmd) # Interactive REPL
else:
out = subprocess.run(cmd, capture_output=True, text=True)
if out.returncode != 0:
raise RuntimeError(out.stderr.strip())
return out.stdout.strip()

———————–

Main

———————–

def main():
ap = argparse.ArgumentParser(description="Offline RAG to llama.cpp (Qwen).")
ap.add_argument("model_path", help="Path to .gguf")
ap.add_argument("question", help="User question")
ap.add_argument("k", nargs="?", type=int, default=TOP_K, help="Top-K chunks (default: %(default)s)")
ap.add_argument("–chat", action="store_true", help="Drop into interactive REPL after building RAG prompt")
args = ap.parse_args()

qv   = embed_one(args.question)
qlit = vec_literal(qv)
hits = retrieve(qlit, args.k)
if not hits:
    print("No results.")
    sys.exit(0)

prompt = build_prompt(args.question, hits)

rc_or_text = call_llama_cpp(args.model_path, prompt, chat_mode=args.chat)
if isinstance(rc_or_text, int):
    sys.exit(rc_or_text)  # REPL exit code
else:
    print("\\n=== ANSWER ===\\n" + rc_or_text)

if name == "main":
main()
"""

path = "/home/darren/Desktop/rag_answer.py"
with open(path, "w", encoding="utf-8") as f:
f.write(content)
os.chmod(path, os.stat(path).st_mode | stat

Yep—let’s reset clean and make this painless.

Your terminal is choking on long pastes. We’ll (1) open a fresh terminal, (2) write a tiny RAG script (no fluff, one-shot paste), and (3) run it with your wide-open GPU defaults.

0) Open a fresh terminal

Close the current one. Open a new terminal window and activate your venv:

source ~/ougway_env/venv/bin/activate

1) Create a minimal RAG script (short paste, hash embed only)

Paste this single, short block at the bash prompt (not in Python):

cat > ~/Desktop/rag_min.py <<'PY'
#!/usr/bin/env python3
import os, sys, subprocess, hashlib, numpy as np, psycopg2, argparse
DSN=os.environ.get("TS_DSN","dbname=tokenspace user=darren host=localhost password=LittleOugway")
TOP_K=int(os.environ.get("RAG_K","6"))
LLAMA_BIN=os.path.expanduser(os.environ.get("LLAMA_BIN","~/ougway_env/llama.cpp/build/bin/llama-cli"))
LLAMA_CTX=os.environ.get("LLAMA_CTX","32768")
LLAMA_NGL=os.environ.get("LLAMA_NGL","999")
LLAMA_BATCH=os.environ.get("LLAMA_BATCH","32")
LLAMA_T=os.environ.get("LLAMA_T",str(os.cpu_count() or 4))
LLAMA_NTOK=os.environ.get("LLAMA_N","512")

def embed_one(t):
    h=hashlib.sha256(t.encode()).digest();seed=int.from_bytes(h[:8],"big")%(2**31-1)
    r=np.random.default_rng(seed);v=r.normal(0.0,1.0,768).astype(np.float32);n=float(np.linalg.norm(v))
    return (v/n) if n>0 else v
def vec_lit(v): return "["+(",".join(f"{x:.6f}" for x in v.tolist()))+"]"

def retrieve(q,k):
    c=psycopg2.connect(DSN);cur=c.cursor()
    cur.execute("""
        SELECT ch.chunk_id, left(ch.text, 900), d.title, s.uri, ch.embedding <=> %s::vector AS dist
        FROM content.chunks ch
        JOIN content.documents d ON d.doc_id=ch.doc_id
        LEFT JOIN content.sources s ON s.source_id=d.source_id
        ORDER BY dist ASC LIMIT %s;
    """,(q,k))
    rows=cur.fetchall();cur.close();c.close();return rows

def build_prompt(q,h):
    parts=[]
    for i,(cid,prev,title,uri,dist) in enumerate(h,1):
        meta=[f"title: {title}" if title else None, f"path: {uri}" if uri else None, f"chunk_id: {cid}  dist: {dist:.4f}"]
        meta=[m for m in meta if m];parts.append(f"### Context {i} ({' | '.join(meta)})\n{prev}")
    ctx="\n\n".join(parts)
    sys_inst="You are a concise assistant. Use the provided context only. If the answer is not in the context, say you don't find it."
    return f"[SYSTEM]\n{sys_inst}\n\n[CONTEXT]\n{ctx}\n\n[USER]\n{q}\n\n[ASSISTANT]\n"

def run_llama(model,prompt,chat):
    cmd=[LLAMA_BIN,"-m",model,"-p",prompt,"-n",LLAMA_NTOK,"--temp","0.2","--top_p","0.9",
         "-c",LLAMA_CTX,"-ngl",LLAMA_NGL,"-t",LLAMA_T,"-b",LLAMA_BATCH]
    if chat: return subprocess.call(cmd)
    out=subprocess.run(cmd,capture_output=True,text=True)
    if out.returncode!=0: raise RuntimeError(out.stderr.strip())
    return out.stdout.strip()

def main():
    ap=argparse.ArgumentParser(description="Minimal offline RAG -> llama.cpp")
    ap.add_argument("model"); ap.add_argument("question"); ap.add_argument("k",nargs="?",type=int,default=TOP_K)
    ap.add_argument("--chat",action="store_true")
    a=ap.parse_args()
    qv=embed_one(a.question);hits=retrieve(vec_lit(qv),a.k)
    if not hits: print("No results."); sys.exit(0)
    prompt=build_prompt(a.question,hits)
    r=run_llama(a.model,prompt,a.chat)
    if isinstance(r,int): sys.exit(r)
    print("\n=== ANSWER ===\n"+r)
if __name__=="__main__": main()
PY
chmod +x ~/Desktop/rag_min.py

2) Set your wide-open GPU env (same as you like)

export GGML_CUDA=1
export LLAMA_BIN="$HOME/ougway_env/llama.cpp/build/bin/llama-cli"
export LLAMA_CTX=32768
export LLAMA_NGL=999
export LLAMA_BATCH=32
export TS_DSN="dbname=tokenspace user=darren host=localhost password=LittleOugway"

3) Run it (non-interactive or chat)

Single RAG answer:

python ~/Desktop/rag_min.py \
  "$HOME/ougway_env/llama.cpp/models/qwen3/qwen3-8b-q6_k.gguf" \
  "Summarize the Flower Lattice activation rule."

RAG then keep chatting (REPL):

python ~/Desktop/rag_min.py \
  "$HOME/ougway_env/llama.cpp/models/qwen3/qwen3-8b-q6_k.gguf" \
  "Summarize the Flower Lattice activation rule." \
  --chat

---

If the terminal still chops pastes

• Try Shift+Ctrl+V (some terminals handle multi-line better).
• Or paste in two smaller chunks: the cat > … <<'PY' line, then the body, then a line with only PY.
• Or use this alternative writer (very short, less likely to choke):
  bash
printf '%s\n' '#!/usr/bin/env python3' 'import os,sys,subprocess,hashlib,numpy as np,psycopg2,argparse' \
'DSN=os.environ.get("TS_DSN","dbname=tokenspace user=darren host=localhost password=LittleOugway")' \
'TOP_K=int(os.environ.get("RAG_K","6"))' \
'LLAMA_BIN=os.path.expanduser(os.environ.get("LLAMA_BIN","~/ougway_env/llama.cpp/build/bin/llama-cli"))' \
'LLAMA_CTX=os.environ.get("LLAMA_CTX","32768");LLAMA_NGL=os.environ.get("LLAMA_NGL","999");LLAMA_BATCH=os.environ.get("LLAMA_BATCH","32");LLAMA_T=os.environ.get("LLAMA_T",str(os.cpu_count() or 4));LLAMA_NTOK=os.environ.get("LLAMA_N","512")' \
'def embed_one(t):' \
' import hashlib,numpy as np;h=hashlib.sha256(t.encode()).digest();seed=int.from_bytes(h[:8],"big")%(2**31-1);r=np.random.default_rng(seed);v=r.normal(0.0,1.0,768).astype(np.float32);n=float(np.linalg.norm(v));return (v/n) if n>0 else v' \
'def vec_lit(v): return "["+",".join(f"{x:.6f}" for x in v.tolist())+"]"' \
'def retrieve(q,k):' \
' import psycopg2; c=psycopg2.connect(DSN);cur=c.cursor();cur.execute("""SELECT ch.chunk_id,left(ch.text,900),d.title,s.uri,ch.embedding <=> %s::vector AS dist FROM content.chunks ch JOIN content.documents d ON d.doc_id=ch.doc_id LEFT JOIN content.sources s ON s.source_id=d.source_id ORDER BY dist ASC LIMIT %s;""",(q,k));r=cur.fetchall();cur.close();c.close();return r' \
'def build_prompt(q,h):' \
' parts=[]' \
' for i,(cid,prev,title,uri,dist) in enumerate(h,1):' \
' meta=[m for m in [f"title: {title}" if title else None,f"path: {uri}" if uri else None,f"chunk_id: {cid} dist: {dist:.4f}"] if m];parts.append(f"### Context {i} ({' | '.join(meta)})\\n{prev}")' \
' ctx="\\n\\n".join(parts);sys_inst="You are a concise assistant. Use the provided context only. If the answer is not in the context, say you do not find it."' \
' return f"[SYSTEM]\\n{sys_inst}\\n\\n[CONTEXT]\\n{ctx}\\n\\n[USER]\\n{q}\\n\\n[ASSISTANT]\\n"' \
'def run_llama(m,p,chat):' \
' cmd=[LLAMA_BIN,"-m",m,"-p",p,"-n",LLAMA_NTOK,"--temp","0.2","--top_p","0.9","-c",LLAMA_CTX,"-ngl",LLAMA_NGL,"-t",LLAMA_T,"-b",LLAMA_BATCH];' \
' import subprocess;return subprocess.call(cmd) if chat else (lambda o:(o.stdout.strip() if o.returncode==0 else (_ for _ in ()).throw(RuntimeError(o.stderr.strip()))))(subprocess.run(cmd,capture_output=True,text=True))' \
'def main():' \
' import argparse;ap=argparse.ArgumentParser();ap.add_argument("model");ap.add_argument("question");ap.add_argument("k",nargs="?",type=int,default=TOP_K);ap.add_argument("--chat",action="store_true");a=ap.parse_args();' \
' qv=embed_one(a.question);h=retrieve(vec_lit(qv),a.k);' \
' if not h: print("No results.");sys.exit(0)' \
' p=build_prompt(a.question,h);r=run_llama(a.model,p,a.chat);' \
' import sys;sys.exit(r) if isinstance(r,int) else print("\\n=== ANSWER ===\\n"+r)' \
'if __name__=="__main__": main()' > ~/Desktop/rag_min.py && chmod +x ~/Desktop/rag_min.py

Tell me which one you try and what the terminal prints; once the file is in place, the rest is just running it with your Q6_K path (and you’ll keep your 32k, full offload settings).

Short answer: yes — that file is valid (Python 3.10+). There’s nothing syntactically wrong with it, and the logic is sound.

What can still make it “look like it won’t run” are external blockers, not the code itself. Here are the only two likely culprits at this point and the exact fixes:

---

1) It stalls before any output → embedding model spin-up

Even though we wrapped it in a try/except, importing sentence_transformers can take a while if it’s present in your venv. To force the fast deterministic embedder (and skip any ST load entirely), set this once for your shell:

export EMBED_MODEL="__DISABLE__"
export HF_HUB_OFFLINE=1

(You can go back to nomic-ai/nomic-embed-text-v1.5 later if you want.)

---

2) It never reaches llama.cpp → wrong binary/env

Make sure the binary path is correct and the GPU flags are on:

export GGML_CUDA=1
export LLAMA_BIN="$HOME/ougway_env/llama.cpp/build/bin/llama-cli"
"$LLAMA_BIN" -h | head -n 3   # should print usage

If that prints usage, you’re good.

---

Run it “wide open” with your model (non-interactive RAG answer)

export TS_DSN="dbname=tokenspace user=darren host=localhost password=LittleOugway"
export LLAMA_CTX=32768
export LLAMA_NGL=999
export LLAMA_BATCH=32
export PYTHONUNBUFFERED=1

python ~/Desktop/rag_answer.py \
  "$HOME/ougway_env/llama.cpp/models/qwen3/qwen3-8b-q6_k.gguf" \
  "Summarize the Flower Lattice activation rule."

Or RAG → interactive chat (REPL)

python ~/Desktop/rag_answer.py \
  "$HOME/ougway_env/llama.cpp/models/qwen3/qwen3-8b-q6_k.gguf" \
  "Summarize the Flower Lattice activation rule." \
  --chat

---

Quick sanity checks if it still doesn’t print

• Verify the file on disk didn’t get mangled by pasting:
  bash
wc -l ~/Desktop/rag_answer.py
sed -n '1,30p' ~/Desktop/rag_answer.py
sed -n '120,220p' ~/Desktop/rag_answer.py
  (You should see the header, and later the def main(): block. If those ranges don’t show up, the paste got truncated—tell me what the last visible line is and I’ll regenerate a shorter file.)

• Check DB connectivity (fast, won’t modify anything):
  bash
psql "dbname=tokenspace user=darren host=localhost password=LittleOugway" -c "SELECT 1;"

If the binary shows usage, the DB returns 1, and EMBED_MODEL="__DISABLE__" is set, this script will run and print an answer. If it doesn’t, paste the last 10 lines it prints so I can zero in on the exact step it’s stalling on (embed → retrieve → llama).