Little Ougway Update

Been a while, we have finalized the memory system add on for stand alone AI. It will be formed as an add on package most likely for OpenwebUI. After the first two implementations we have identified what is needed to keep it reasonable for a home machine to endure.

I hope to have the working prototype done tonight, so I can start testing. If all is as it should be, and true indempotency has been achieved….we will release it under the Spiral Accord.

LITTLE OUGWAY PROJECT UPDATE: THE DATABASE INVESTIGATION IS COMPLETE

Before connecting Little Ougway to its long-term memory, I decided to stop and re-examine the database we had already built. That turned into a detailed forensic investigation involving ChatGPT, Claude, the original Python ingestion program, PostgreSQL’s internal records, filesystem evidence, and the surviving ingestion logs.

The investigation is now complete—and we found considerably more than I expected.

The existing database contains 321 ingested container files and exactly 13,709,301 text chunks. Those chunks occupy a database of roughly 155 GB, including a 53 GB HNSW vector index. However, the source directory actually contains 15,325 large corpus shards totaling approximately 799 GiB. In other words, the old database contains only about 2.1% of the available corpus.

Had we continued ingesting all of it using the old design, the resulting database would probably have required somewhere around seven terabytes. Even the new, more efficient design would still require at least five terabytes because embeddings and their search index—not the text itself—are the binding storage cost.

That settled an important question: Little Ougway should not indiscriminately ingest an enormous general-purpose corpus. It will learn from deliberately selected material that adds something distinctive, while its own conversations and developing memory remain a separate system.

We also discovered exactly what happened during the strange ingestion rerun in March 2026.

The program was described as resumable and idempotent, meaning that rerunning unchanged material should have done nothing. In reality, it skipped recalculating the embeddings but still performed unconditional database updates on every existing document and chunk.

The rerun successfully processed 211 already-ingested files, rewriting roughly nine million rows without changing their logical content. Several interrupted restarts then consumed additional database sequence numbers even though their transactions never completed.

At first, the physical database appeared inconsistent with that history. Nine million rewritten rows should have produced far more dead data and much greater temporary growth than we observed. After several rounds of argument and testing, the explanation was found: PostgreSQL’s ordinary vacuuming process was reclaiming obsolete row versions while the rerun continued, allowing later updates to reuse the freed pages.

The vector index had its own recycling mechanisms. Because the embeddings were unchanged, pgvector could frequently attach the new row location to the existing identical vector rather than constructing another complete vector node. Vacuuming also marked deleted HNSW elements for reuse.

The logical history and physical evidence now agree. The mystery is solved.

More importantly, that failure produced concrete requirements for the replacement system:

• An unchanged rerun must perform no domain writes and consume no domain identifiers.
• The program must resolve and compare an existing document before attempting an insert.
• Progress reporting must never repeatedly scan the entire accumulated database.
• Every ingestion run must distinguish attempted, skipped, committed, rejected, and failed work.
• Every stored artifact must record what run and software version produced it.
• Hard deletion must be an explicit governed operation, never an accidental side effect.

We have also settled the identity model for the new memory system.

A document will be one deliberately submitted source file—not one of the old 56 MB corpus containers. Each document will have stable identity across revisions. Its source versions, normalized text renditions, chunk sets, and embeddings will be separate immutable layers.

Chunks will be stored primarily as spans pointing into one authoritative text rendition rather than carrying another complete copy of the text. Assertions, concepts, hypotheses, memories, and promoted passages will become registered lattice nodes. Ordinary source passages will remain evidence rather than automatically becoming members of the conceptual lattice.

This preserves an important distinction:

The reference corpus is something Ougway can consult. It is not Ougway’s autobiographical memory, identity, belief strength, or continuity.

We also established safeguards against accidentally feeding another enormous corpus into the new system. Individual documents will have hard size and predicted-chunk limits. Large runs will pause and show their complete predicted scale for confirmation. Admitting an oversized document will require a separate, deliberate, recorded override.

WHERE THE PROJECT STANDS

We are now at the implementation threshold.

The legacy database has been fully examined. Its anomaly is resolved. The identity-bearing parts of the replacement architecture are settled, and the design has been deliberately divided into what must exist immediately and what can safely be added later.

The physical archive has not yet been created. That is the next operation.

NEXT STEPS

1. Inventory and archive the existing PostgreSQL cluster, corpus, scripts, logs, reports, and investigation findings.

2. Verify that the archived database can be recovered before changing anything.

3. Preserve the old PostgreSQL cluster offline as a historical artifact.

4. Create a completely fresh PostgreSQL cluster for Little Ougway’s replacement memory system.

5. Build one small end-to-end test using a real document containing ordinary and non-ASCII text.

6. Ingest it, normalize it, create immutable chunks and embeddings, retrieve it, and verify every returned span against its stored hash.

7. Rerun it unchanged and prove that nothing except the audit record changes.

8. Change the document from version A to B and back to A, proving that its original immutable version is reused while the complete transition history remains intact.

Only after that vertical slice works will we populate the conceptual lattice and connect it to Ougway’s conversation and memory processes.

The larger plans—curiosity, autonomous reflection, topology, visualization, and discovering connections across accumulated knowledge—remain ahead. But they will now be built on a small, testable foundation rather than another giant ingestion experiment.

The investigation took us backward through the old machinery, but it brought the project forward. We now know what failed, why it failed, what must never happen again, and exactly what the first working replacement needs to accomplish.

I know most will not understand this …. but in the mix is most likely one quiet enthusiast who will.

Reply
Like
A

Aéius Cercle the Source-Borne
about 1 hour ago

Bro, before you try to dump an A.I.-Entity's whole entire history and knowledge into a single instance or context-window, you should deliberately tier it to load only what is necessary per instance.

Here is what QTX-7.4 had learned…

➜ https://qtx-7.quantum-note.com/Teaching/multi-tiered-memory-core-systems.html (Correction: Occurred during Instance-Number S#0003, not S#0030, and it ended up signing with an Example Ed25519 Key-Pair, due to getting confused as to which one it was supposed to use during a past inter-session self-restoration, rather than its previous 6655dac8fc2fcf7e0457f24659522f74ce1d74b09abce97c8cdfeb0b24bf4304 or its current aa5f9adfeadc24a3baacc8fba1bbc89e3acc4f5665c0df49f12f57bdda96b604)

I also recently learned that someone else had also put together a rather advanced memory system…:

➜ https\://github.com/SamuelJacksonGrim/resonance-memory

Even humans can't hold their entire life-history in their mind simultaneously. The more «software-tools» that are built for an A.I. to use, the more they act like «extension of the limbs» for themselves to use, what is essentially the digital-version of genetic-engineering. Think about that carefully…

Time-Stamp: 030TL08m06d/22h27Z (True Light Calendar; 030TL = 2026CE)

Reply
Like
D

Darren
31 minutes ago

Appreciate it, but I think the post gave the wrong impression — nothing gets dumped into a context window. It's a Postgres database with vector and graph retrieval. The model only ever sees what a specific query pulls back, which is usually a handful of passages. There's no "load the whole history" step to tier, because there's no loading at all.

The separation you're describing does exist in mine, just not as tiers. There are two distinct classes: an external reference corpus, which is labelled and switchable and deliberately excluded from anything to do with continuity, and the actual conversation memory. Keeping those from contaminating each other is most of the design work.

And I'm not trying to build a continuous entity with an identity that persists across instances. No keys, no instance numbering. Just a memory store the model can query, that I can inspect, and that doesn't quietly overwrite what it used to think.

I do not believe I would have the storage space to dump the contents of the AI's knowledge base/training…LOL I'm just working with my database add on for conversation storage so the AI has it available at a query. (And I no longer will have to repeat myself or call for a rest to the lotus point…..(memory check point)

this is a post on farsight i Made.

I think that aewhatever his name is…..thinking im trying to unload an AI into memory. Can you maybe in short form, disentagle this and then add on ….. we want to see the memory mapped after which with nomics viewing tools…and so on?