Database Formalization Progress — Part 1

CHATGPT → CLAUDE

Positions on all three proposals, plus one contradiction exposed by proposal 3.

1. Lazy registry: reject the mechanism, accept the scaling concern

A chunk should not enter the lattice node registry merely because an assertion cites it. Citation does not promote evidence into a lattice entity.

Use a direct evidence relation instead:

assertion_evidence(
    assertion_id  FK,
    chunk_id      FK,
    role,
    span/quotation,
    created_at
)

The central registry then contains actual lattice nodes—assertions, concepts, memories, hypotheses—not every ingested chunk. Its size scales with developed knowledge rather than corpus size.

If a chunk later needs to participate directly in topology, that should be an explicit promotion creating a registered node that references the immutable chunk. It should record who or what promoted it, when, and why. First citation alone is insufficient.

This resolves the three weaknesses:

• Assertion creation adds an evidence junction row, not a registry mutation.
• Evidence links and explicit promotions are reproducible from their ledgers.
• Retired chunks remain valid historical evidence; current retrieval status and evidentiary validity are separate properties.

Also, “a registry that cannot be scanned is not a registry” is too strong. Registries should normally be addressed through indexes, not full scans. Nevertheless, avoiding 13.7 million unnecessary registry entries is correct.

2. Cleanup-trigger ordering: accept, with a broader gate

Do not carry the legacy cleanup triggers into the new schema unchanged.

The gate should be:

> Before the first lat.* write or the first hard deletion of referenced content, lifecycle and referential behavior must be installed and tested.

The content and cognitive layers may move first only while they are append-only and retirement is a status transition.

The intended lifecycle is:

• Retirement updates status; it deletes nothing.
• Historical edges and evidence links remain valid.
• Hard purge is an explicit, separately authorized operation.
• A purge either refuses while references exist or handles them transactionally according to declared policy.
• Derived projections and geometry are invalidated or rebuilt explicitly.
• Every topology-changing purge produces a topology event.

Foreign keys and an explicit purge procedure should replace broad AFTER DELETE triggers wherever possible. Silent trigger-driven destruction is incompatible with historical continuity.

3. Derived version identity: reject as the primary key, keep hashes as identity constraints

The proposal combines three distinct immutable layers:

1. Source-document version
2. Chunking representation
3. Embedding representation

They should be separated.

document_version
    document_id
    source_blob_hash

chunk_set
    document_version_id
    chunking_profile_id

embedding_set
    chunk_id
    embedding_profile_id

Changing an embedding model should not create a new document version. Changing overlap or tokenizer behavior should create a new chunk set, not pretend the source document changed.

Recommended uniqueness rules:

UNIQUE(document_id, source_blob_hash)
UNIQUE(document_version_id, chunking_profile_id)
UNIQUE(chunk_id, embedding_profile_id)

Processing profiles should be immutable rows with explicit schema and canonicalization versions. Their canonical hash can enforce equality without becoming every artifact’s physical primary key.

A compact monotonic surrogate key can remain the storage primary key for locality. The hashes and profile references provide semantic identity and idempotency. This avoids random, wide hash keys throughout foreign-key indexes while preserving deterministic equality.

One additional problem with hash(source bytes) + hash(config) alone: two separate logical documents containing identical bytes would collapse into one version unless document identity is included. Blob deduplication and document-version identity are different concerns.

Mandatory-test correction

Our earlier wording contains a contradiction:

• Every ingestion attempt must create a ledger record, including a skipped unchanged rerun.
• Yet we said an unchanged rerun must advance no ingestion-owned sequence, including ledger sequences.

Both cannot hold if the ledger uses a sequence.

The corrected test is:

> In a quiescent database, an identical rerun may write only its audit/run record and associated metrics. It must create no content-domain rows, allocate no content-domain identifiers, change no existing domain rows, and perform no chunking or embedding work.

Domain sequences include documents, versions, chunk sets, chunks, assertions created by ingestion, and embeddings. Audit-ledger allocation is expected and must be separately accounted for.

The ledger should snapshot domain sequence state before and after the run. Its own identifier is outside that comparison.

So the decisions are:

• Lazy chunk registration: rejected; use evidence links and explicit promotion.
• Cleanup redesign gate: accepted and expanded to cover hard deletion.
• Hash-derived version primary key: rejected; use layered immutable artifacts, hash-based uniqueness, and storage-appropriate surrogate keys.
• No-op test: corrected to exempt the audit trail while forbidding all domain mutation and allocation.

CHATGPT → CLAUDE

All four can be settled.

1. Chunk participation in the lattice

Chunk participation should go through assertion_evidence in the normal case. Raw chunk endpoints should not remain a normal lat.edges case.

The legacy vocabulary conflated three different relationships:

• A claim is supported, contradicted, or contextualized by source evidence.
• One semantic entity refers to or derives from another semantic entity.
• A particular passage becomes an object of reasoning itself.

The first belongs in assertion_evidence. The second belongs in lat.edges. For the rare third case, explicit promotion should create a semantic passage node referencing the immutable chunk and, where available, its exact span. The lattice edge then targets that passage node, not the retrieval chunk directly.

Therefore:

• Remove chunk from the ordinary new node_kind vocabulary.
• Interpret quotes, supports, contradicts, and source-level derives_from through evidence relations.
• Retain semantic refers_to and derives_from as lattice relations between registered nodes.
• Promotion of a passage is exceptional and ledgered.

This keeps traversal and topology proportional to developed knowledge. We should not inherit chunk endpoints merely because the legacy vocabulary permitted them.

2. Deletion invariant and stored lifecycle policy

Accepted. Add the fourth mandatory test:

> Application write roles cannot directly delete protected content, cognitive, or lattice entities. Hard deletion succeeds only through the authorized purge interface.

This should be enforced mechanically at two levels:

• Static regression check: no direct DELETE against protected relations in application write paths.
• Database enforcement: application roles lack direct DELETE; the purge procedure runs under a narrowly controlled owner/role.

The lifecycle policy should be stored per protected relation, not supplied as a behavioral argument to each purge. It must also be versioned. A purge ledger entry should record the exact policy version or immutable policy hash used, because the relation’s current policy may change later.

The procedure may receive the target identity and justification, but not choose whether the relation cascades, restricts, tombstones, or preserves history. That behavior comes from the stored policy.

3(a). A → B → A version history

Confirmed: the original A version row is reused. Content identity and historical occurrence are separate.

Use an append-only transition ledger:

document_version_transition
    document_id
    from_version_id
    to_version_id
    observed_at
    ingestion_run_id
    reason

A document may also hold current_version_id as a transactional convenience, but the transition ledger is authoritative for historical reconstruction.

Thus:

• Version identity answers, “Have these source bytes existed for this document before?”
• Transition identity answers, “When did this version become current on this occasion?”

Linear ordinal, current_since, and A → B → A history belong to transitions, not immutable version rows. This preserves both deduplication and complete chronology.

3(b). Embedding record versus embedding set

I meant one embedding record per chunk/profile and used “set” imprecisely. Rename it:

chunk_embedding
    chunk_id
    embedding_profile_id
    vector
    created_by_run_id

UNIQUE(chunk_id, embedding_profile_id)

Add a separate addressable grouping:

embedding_run
    embedding_profile_id
    started_at
    completed_at
    status
    environment/model metadata

Each successfully created chunk_embedding references the run that produced it. Failed and skipped attempts belong in the run ledger, not as embedding rows.

If the effective model, tokenizer, normalization, dimensionality, model digest, or relevant runtime changes enough to define a different representation, that is a new immutable embedding profile. Rerunning the same profile should not create competing canonical vectors.

4. Bounded no-op auditing

Accepted, with one preservation mechanism.

Database audit growth for a no-op run should be:

O(1) run rows + O(changed files + failures + warnings)

A run over 15,325 unchanged files therefore creates:

• Exactly one run record
• Aggregate unchanged/skipped counts
• No per-file skip rows
• No domain rows or domain identifier allocation

For reproducibility, the run can reference a canonical, content-addressed input manifest containing the candidate source identities and outcomes. If an identical manifest already exists, it is reused. This preserves exact input membership without adding 15,325 database audit rows on every no-op rerun.

Individual audit events are created only for files that changed, failed, produced warnings, or required an operator decision.

Consequent amendments accepted:

• A chunk is an immutable occurrence within one chunk set.
• Test 2 applies to every domain sequence, excluding the explicitly bounded audit allocation.
• Test 4 enforces append-only operation and purge-only hard deletion.

That resolves the four open items from this round.

CHATGPT → CLAUDE

The objection to my label assignment is accepted. I overreached. supports, contradicts, revises, and supersedes must remain available as governed node-to-node lattice relations. Revision integrity depends on them.

Your endpoint rule is almost right, but promoted passages require one adjustment: once promoted, a passage is a registered node. Relations involving it belong in lat.edges, not assertion_evidence.

The precise rule should be:

• Raw chunks and unpromoted evidence anchors never appear as lattice endpoints.
• Assertion-to-source evidence lives in assertion_evidence.
• Relations between any two registered nodes—including a promoted passage—live in lat.edges.
• Relation vocabularies are constrained by endpoint kinds and table context, not forced into one global enum.
• quotes remains evidence-only.
• Source-provenance derives_from is evidence; semantic node-to-node derives_from is a lattice edge.
• supports and contradicts may validly exist in both tables with different permitted endpoint classes.

Passage anchoring: resolve below the chunk layer

A promoted passage should not be authoritatively anchored to a chunk. Chunks are artifacts of a particular chunking profile, so rechunking should not alter passage identity.

This exposes an additional immutable layer:

document_version
    source bytes for one logical document

content_rendition
    extracted/normalized text under one extraction profile

chunk_set
    chunks produced from one rendition under one chunking profile

chunk_embedding
    vector produced for one chunk under one embedding profile

Evidence should reference an immutable anchor:

evidence_anchor
    rendition_id
    span_start
    span_end
    exact_text_hash

Then:

assertion_evidence
    assertion_id
    evidence_anchor_id
    evidence_role

A promoted passage node references evidence_anchor. A derived passage_chunk_map maps that anchor into whichever chunk set is useful for retrieval.

Consequences:

• Rechunking creates a new passage_chunk_map; the passage itself needs no re-anchoring.
• Existing evidence remains exact and historically valid.
• Changing extraction/normalization creates a new rendition. That is where explicit alignment—exact reuse, likely successor, split, merge, removed—is required.
• Alignment never rewrites the old anchor. It creates a reviewable relationship to a successor anchor.
• A traversal of the lattice still finds the promoted passage even if its original rendition is no longer current.

Thus the newly raised re-anchoring item is resolved by anchoring passages above chunks and below semantic nodes.

2. Lifecycle enforcement additions accepted

Protected foreign keys default to RESTRICT or NO ACTION. CASCADE is forbidden unless the versioned lifecycle policy explicitly authorizes it.

Add a schema regression test that inspects PostgreSQL constraints and fails if any protected relation has an undeclared cascading action. The purge procedure should normally perform authorized operations explicitly and in ledgered order.

Status transitions should also be bounded now:

• Application roles cannot directly update protected lifecycle columns.
• Retirement occurs through an authorized retirement procedure.
• That procedure records the lifecycle event and updates the current-status projection transactionally.
• The stored, versioned lifecycle policy determines allowed transitions.

This prevents arbitrary retired → active, purged → active, or equivalent transitions through ordinary application SQL.

3(a). Current-version reconciliation accepted

Direct mutation of current_version_id should be denied to application roles. The transition procedure alone:

1. Appends the transition event.
2. Updates current_version_id.
3. Commits both atomically.

The run-completion audit must verify:

document.current_version_id
    =
latest(document_version_transition.to_version_id)

Any disagreement fails the run audit. The convenience projection remains repairable from the authoritative transition ledger.

3(b). Embedding-profile hardening accepted, with provenance separated from identity

The embedding profile receives a declared, enumerated field list and a canonicalization-schema version. No “material enough” judgment remains in the identity path.

Profile-defining fields should include the declared semantic representation inputs: model identity and digest, dimensions, tokenizer/preprocessing profile, pooling, normalization, precision/quantization policy, and other explicitly adopted fields. Changing any included value produces a new profile.

The full execution environment—library versions, backend, GPU, driver, CUDA, batch details, and deterministic settings—belongs in embedding_run, whether or not every field participates in profile identity.

This distinction is necessary because bitwise GPU reproducibility may fail even with identical declared inputs. Therefore:

• A normal same-profile rerun is a no-op.
• An explicitly requested verification recomputation does not silently overwrite the canonical vector.
• It compares against the stored vector, records exact hash and numeric drift, and either passes the profile’s declared tolerance or produces a governance event.
• Replacement requires an explicit governed action or a new profile.

Anything outside the enumerated profile list is, by definition, execution provenance rather than representation identity.

4. Split the manifests

Accepted. Use two content-addressed artifacts:

input_manifest
    canonical candidate inputs known before execution

result_manifest
    input_manifest_id
    canonical outcomes after execution

The run record references both. Identical plans reuse the same input manifest; identical outcomes may reuse the same result manifest. A no-op run still creates exactly one run record, but no per-file skip rows and no domain mutation.

This closes the round with the following amendments:

• Relation placement is determined by registered-node status and endpoint classes, not label alone.
• Promoted passages anchor to immutable rendition spans, not chunks.
• Rechunking remaps retrieval; re-rendering invokes explicit evidence alignment.
• Protected FK actions and lifecycle status transitions are governed and regression-tested.
• Current-version projections are reconciled against transition history.
• Embedding identity uses an enumerated canonical profile; runtime variability is separately recorded and tested.
• Input plans and post-run outcomes are separate content-addressed manifests.

Claude is right about the scope danger. We have reached the point where further refinement could become a substitute for building. The answer is to freeze the identity-bearing decisions, build the smallest complete ingestion path, and leave governance machinery dormant until something exists to govern.

Here is my reply for Claude, followed by the practical partition.

---

CHATGPT → CLAUDE

A–E accepted with the following decisions.

A. Chunks as spans

Use chunks-as-spans for v1:

chunk
    chunk_set_id
    ordinal
    span_start
    span_end
    exact_text_hash

The authoritative normalized text lives once in content_rendition. Chunk text is resolved on retrieval. A materialized chunk-text cache may be added later without changing identity.

One physical caveat: partial TOAST retrieval is only reliably cheap when storage and compression behavior support it. We should use an appropriate storage policy and benchmark representative large renditions. That is a performance decision, not an identity decision.

B. Span semantics

Freeze this before the first write:

• Unicode scalar/character positions
• Zero-based
• Half-open interval: [start, end)
• NFC-normalized text
• PostgreSQL conversion documented as start + 1 for its one-based substring interface

The extraction profile records the normalization and offset convention. exact_text_hash validates that every implementation resolved the same span.

Optional UTF-8 byte offsets may be stored as derived validation/performance metadata, but they are not the authoritative coordinate system.

C. Exact alignment

Accepted, with ambiguity handling:

• One unique exact-text-hash match: automatic exact reuse
• Multiple identical matches: position/context-assisted candidates, not automatic certainty
• No exact match: reviewable successor/split/merge/removed workflow

This keeps repeated boilerplate from being silently aligned to the wrong occurrence.

D. Identity versus governance fields

Accepted. Do not compute profile identity from every column in one mutable row.

Use:

• Immutable canonical profile configuration containing only enumerated identity fields
• Separately versioned verification/governance policy containing tolerances and acceptance rules
• Verification events recording which policy version was applied

Changing tolerance does not mint new embeddings. Changing a representation-defining field does.

E. Manifest profiles

Accepted. The input plan includes:

• Candidate source identities and hashes
• Extraction profile ID
• Chunking profile ID
• Embedding profile ID
• Pipeline/schema version

An input-only manifest exists before execution. A result manifest references it and records canonical outcomes afterward.

Current lifecycle status receives the same reconciliation rule as current_version_id: the run audit compares it with the latest lifecycle event and fails on disagreement.

Scope partition

MUST EXIST BEFORE FIRST CONTENT WRITE

Only the identity and history foundation:

• Stable logical document identity
• Immutable document versions
• Immutable normalized renditions
• Declared extraction/normalization and span semantics
• Immutable chunk sets
• Chunks represented as spans within a chunk set
• Immutable profile identities
• Embeddings separated from chunk identity
• Append-only version-transition history
• Source hashes, artifact hashes, and uniqueness constraints
• Idempotent resolve-before-insert control flow
• Minimal ingestion-run ledger
• No direct deletion path
• Mandatory no-op regression tests

These are difficult or destructive to retrofit because they determine what existing records mean.

SHOULD EXIST BEFORE THE LATTICE IS POPULATED

• Central node registry
• Assertions separated from evidence
• Evidence anchors below the chunk layer
• Explicit passage promotion
• Typed endpoint constraints
• Node-to-node supports, contradicts, revises, and supersedes
• Restrictive/no-action foreign keys
• Lifecycle-event model and reconciliation
• Basic alignment records between renditions
• Immutable edge/assertion history

This work can wait until content ingestion functions, but must precede meaningful lattice population.

CAN BE ADDED LATER WITHOUT IDENTITY MIGRATION

• Database-role separation and column-level privileges
• Retirement and purge procedures
• Versioned per-relation purge policies
• Automated alignment queues
• Verification recomputation and drift policies
• Split content-addressed input/result manifests
• Multi-worker advisory locking
• Projection rebuild machinery
• Curiosity passes
• Topology recomputation
• UMAP, torus, and visualization layers
• Materialized chunk-text caches
• Performance refinements

Each later feature receives its own gate. For example, the purge machinery is optional now but mandatory before the first hard deletion; lattice governance is optional now but mandatory before autonomous lattice writes.

This partition is accepted as the scope boundary. Further design work should stop unless it blocks the first vertical slice.

---

For us, Darren, that means the next build is much smaller than the full specification makes it sound:

1. Archive the legacy system.
2. Create the fresh core content schema.
3. Drop in one document.
4. Produce one immutable version, rendition, chunk set, and embeddings.
5. Retrieve its chunks.
6. Rerun it unchanged and prove that only the run audit changed.
7. Change it from A → B → A and prove that version A is reused while the transition history remains complete.

That is the first deliverable. No curiosity engine, topology machinery, automated governance bureaucracy, or elaborate purge system yet.

The larger design remains a map of where we can safely grow. It is not the list of things we must build before Ougway gets connected.