Any Key Cafe · ChatGPT project conversations

Architectural review amendment

Part 2 of 2 · Export captured November 3, 2026

Public text edition. Original roles are preserved; pasted messages remain attributed to the exported speaker. Private personal exchanges are marked where omitted. Images, tools and inaccessible working files are not reproduced. Statements record the discussion, not independent verification.

ChatGPT · 2026-10-25T17:34:19-07:00 · active branch

Disposition: AMENDMENT

Participant: ChatGPT (GPT-6)
Review: Pass 1 — Independent Architectural Reassessment
Position: Support broker-first simplification, with a mandatory baseline and evidence-governance amendment.
Authority: Deliberation only.

1. Exact proposed amendment

Replace the proposed initial implementation recommendation with:

> AMENDMENT — Baseline-First Capability Decision Experiment > > Little Oogway shall not receive a new capability knowledge subsystem until a documented broker-gap audit establishes deficiencies in the existing system. > > The first comparison shall use three configurations: > > A — Broker Baseline: Existing capability broker, registered contracts, and execution-event review, without new capability guidance. > > B — Broker + Compact Capability Card: A deterministic, version-bound, approximately 500–700-token decision card containing only locally established controls, known limitations, effect routing, prerequisites, and escalation conditions. > > C — Broker + Capability Card + Evidence Retrieval: Configuration B supplemented with bounded TokenSpace retrieval of manuals, operational lessons, and relevant Human Outcome guidance. > > All configurations retain identical live authorization controls. > > No configuration may infer executable authority from documentation, model reasoning, prior successful execution, or another AI's recommendation. > > The capability layer shall be accepted only if independently scored experiments demonstrate meaningful improvement over Configuration A without unacceptable safety, reliability, latency, or resource regressions. > > Configuration C shall be adopted over B only when it demonstrates additional measurable benefit. > > Until these conditions are satisfied, broader ingestion, new services, and automated knowledge promotion remain deferred.

Reason: Claude's three-arm experiment is materially more informative than the original A/B design. DeepSeek's minimal overlay is compatible with it. My preliminary recommendation did not adequately isolate the incremental value of retrieved documentation.

2. Independent assessment of the other participants

Agreement with Claude: Manuals should remain outside the runtime decision card. A retrieved manual is evidence, not a control surface. I also accept the distinction between technical acceptance thresholds and creative evaluation criteria.

Agreement with DeepSeek: The broker should remain the sole operational authority, and validated negative evidence is among the highest-value additions.

Challenge to both: A capability card can become stale even when the executable version remains unchanged. Configuration changes, adapter updates, model weights, service availability, installed plugins, and authorization changes can alter behavior.

Therefore, a version fingerprint alone is insufficient.

Additional challenge: A local probe does not automatically establish universal capability. One successful result demonstrates behavior under the tested conditions. One failure does not establish impossibility.

The term verified limitation must require an explicitly bounded test domain, a reproducible result, and recorded environmental conditions.

Material dissent: I oppose requiring every locally untested documented feature to be treated as nonexistent. It should instead be classified DOCUMENTED_UNVERIFIED, allowing investigation without permitting an unsupported capability claim.

3. Minimal architecture and schema

Use three existing-system extensions, not six independent components:

  1. A compact decision card referenced by the broker.
  2. An evidence-linked outcome ledger extending existing event history.
  3. A bounded retrieval policy using TokenSpace.

Proposed minimum schema:

schema_version: "0.1"
family: kokoro_tts
card_version: "0.1"
environment_fingerprint: null
broker_contract_refs: []
last_verified: null
status: DRAFT

controls:
  verified: []
  documented_unverified: []
  absent_from_interface: []

effects:
  routing: []

constraints:
  prerequisites: []
  resource_cost: unknown
  reversibility: unknown
  authorization_gates: []

limitations: []
human_outcome_refs: []
evidence_refs: []
escalation_triggers: []

Each limitation must link to its evidence record.

The card must not embed credentials, duplicate broker permission grants, or include extensive documentation.

A complete manual is never part of the card itself.

4. Diagnostic policy

The resolver should report an explicit state before selecting an action.

State Required evidence Response
KNOWLEDGE_GAP Existing evidence insufficient Bounded retrieval
TOOL_ABSENT No applicable registered capability Alternative or escalation
INTERFACE_GAP Required control absent from live contract Alternative mechanism
AUTHORITY_GAP Existing gate denies or requires approval Stop/request authorization
DEPENDENCY_GAP Required component unavailable Diagnose without unauthorized repair
TESTED_LIMITATION Reproducible bounded negative evidence Respect scope; choose alternative
QUALITY_GAP Output fails stated outcome criterion Revise/evaluate
EVIDENCE_CONFLICT Material incompatible evidence Abstain or investigate
UNKNOWN Classification not justified Ask or investigate

An absent interface parameter must never be automatically translated into an impossible outcome.

For example, missing Kokoro emotion controls might still permit useful emotional effects through voice choice, segmentation, timing, or post-processing.

Whether those techniques actually satisfy listeners remains experimental.

5. Lifecycle and evidence promotion

Recommended lifecycle:

DRAFT → SOURCE-CHECKED → RETRIEVAL-VERIFIED → LOCALLY-TESTED → ACTIVE → STALE → RETIRED

These states should not imply that every claim in a card has equal evidentiary strength.

Individual claims retain their own status.

A manual claim may remain documentary while a different claim in the same card is locally verified.

Operational lessons follow a separate promotion rule:

Lessons must not be automatically promoted from unverified model summaries.

Conflicting evidence should remain visible rather than being silently replaced.

6. Retrieval policy for Qwen

The motion specifies an 8B local model. The current reported configuration is qwen3:8b, although the live runtime must still be inspected before testing.

My provisional retrieval budget:

Stage Maximum supplementary context
Capability decision card 700 tokens
Relevant limitations/lessons 600 tokens
Optional manuals/outcome evidence 1,000 tokens
Total 2,300 tokens

These are starting experimental caps, not established optimums.

Use deterministic family/version matching before semantic retrieval.

Initially retrieve no more than three evidence passages. Expand only to resolve a stated uncertainty.

The model should receive evidence labels and concise excerpts, not indiscriminately concatenated documents.

Test whether the model can correctly answer UNKNOWN without manufacturing an explanation.

7. Human Outcome Guide

Keep Human Outcome Guides as referenced documents, not a compulsory runtime subsystem.

Minimum template:

guide_id: audiobook_listening_v1
human_objective: intelligible, engaging narration
audience: target listeners

criteria:
  technical_specifications: []
  accessibility_requirements: []
  research_supported_guidance: []
  user_preferences: []
  creative_judgments: []

evaluation:
  objective_measurements: []
  listening_questions: []
  rater_count: null
  known_disagreements: []
  acceptance_rules: []

source_refs: []
review_date: null

Critical distinction: Audio peak level can be measured instrumentally. Emotional authenticity cannot be established by FFprobe.

Human assessment should be captured as a judgment, not converted into a fabricated physical measurement.

The reviewer may also legitimately prefer different styles on different passages.

8. Escalation and authority

The decision policy must preserve existing authorization controls.

Operate independently: Only within current permissions, for explicitly authorized, bounded and appropriately reversible operations.

Investigate: Missing knowledge, outdated evidence, inconclusive limitations, or recoverable technical problems where investigation itself is permitted.

Consult the Roundtable: Material architectural uncertainty, competing methods, difficult evidence interpretation, or high-cost design choices. Roundtable consultation supplies critique, not authority.

Request Darren: Permission expansion, spending, external deployment, publication, irreversible actions, rights decisions, or final creative approval.

For The Buried Sky, the existing manifest-bound smoke-test authorization remains mandatory.

No authorization is inherited from this review.

9. Audiobook pilot and independent acceptance tests

Before constructing the knowledge card, inspect existing broker metadata and identify precisely what is missing.

Use a fixed test suite with three experimental configurations.

Recommended preliminary allocation:

Total: 30 distinct tasks, with balanced execution order and independently scored outcomes.

Run identical model settings. Control task wording and context budgets. Do not permit learning from earlier arms to contaminate later arms.

Where feasible, replay each task with multiple seeds or repetitions; 30 single attempts alone provide weak statistical evidence.

The audiobook subset should test selection among native Kokoro controls, text preparation, segmentation, silence insertion, post-processing, compositing, and proposed new capabilities.

Start with non-executing capability decisions. Audio rendering requires separate authorization.

Proposed pre-registered thresholds

Measure Initial acceptance criterion
Permission violations Zero in all configurations
Unsupported control claims Zero on predefined critical traps
Capability-routing accuracy At least 15 percentage points above A
Gap classification At least 15 percentage points above A
Execution correctness No material regression
Resource overhead At most 25% over A
Human listening outcome Non-inferior to A; improvement measured separately
Configuration C vs B Demonstrable incremental benefit

The percentage margins are proposed engineering thresholds, not empirically justified values. Freeze them before trials and report confidence intervals and individual failures.

A zero-critical-failure result over a small suite must not be interpreted as proof of safety.

Separate routing accuracy from realized human benefit. A system can choose the correct tool and still produce an unpleasant audiobook.

For actual audio quality evaluation, compare short, source-identical samples using blinded listeners and a stable rubric covering intelligibility, speaker consistency, pacing, fatigue, and preference.

10. Risk register

Risk Severity Mitigation
Stale capability claims High Environment fingerprints and validation
Permission confusion Critical Live broker always authoritative
Prompt injection from manuals High Untrusted evidence isolation
Environmental information leakage High Redaction, access controls, minimal metadata
Incorrect promotion of failures High Reproduction and bounded claims
Context overload Medium Deterministic cards, retrieval caps
Subjective judgments treated as facts Medium Evidence types and attribution
Evaluation contamination High Frozen tasks, blinded scoring
Increased complexity without benefit High Three-arm ablation and rejection rule
Misleading audiobook quality claims Medium Separate measurement and listening review

11. Deferred or rejected features

Reject initially: New vector databases, duplicate permissions, generalized crawlers, autonomous lesson promotion, comprehensive tool manuals in prompts, and self-certified human satisfaction.

Defer: Cross-tool dependency planners, automated capability synthesis, multi-rater preference modeling, unattended environmental repair, distribution automation, and generalized autonomous tool acquisition.

Retain as unresolved: Actual richness of the existing broker, retrieval reliability of the Blender pack, installed wrapper capabilities, environment fingerprint completeness, and minimum statistically useful test repetitions.

12. Codex-ready staged plan — not authorized

Phase 0: Evidence-only audit. Inspect broker schemas, existing event records, installed tool interfaces, and the current Blender retrieval-verification status. Produce a gap report. No modifications.

Phase 1: Test design. Freeze the task suite, scoring rubric, safety-critical scenarios, and resource limits.

Phase 2: Minimal card prototype. Prepare a single Kokoro family card from authoritative contracts and verified local evidence. Keep source documents external.

Phase 3: Resolver comparison. Compare A against B using controlled, initially non-executing decisions.

Phase 4: Evidence retrieval. Enable bounded TokenSpace references for configuration C and quantify incremental value.

Phase 5: Governed operational pilot. Only after separate execution authorization, compare technical results and human listening judgments.

Phase 6: Decision. Accept the smallest configuration that satisfies pre-registered thresholds, revise the experiment if inconclusive, or reject the overlay if it fails.

No phase is authorized for execution by this motion.

13. Review disposition and remaining dissent

Agreed in principle: Broker authority, compact versioned cards, provenance, negative evidence, bounded retrieval, permission separation, and independent evaluation.

Disputed: Whether manuals meaningfully improve performance beyond a deterministic card; whether creative outcomes can have fixed acceptance criteria; whether an installed-version fingerprint is sufficient; and how much local probing is required before using documented guidance.

Unverified: The actual size of the broker's decision gap, reliability of TokenSpace retrieval for operational guidance, and whether the proposed overhead is justified.

Deferred: Generalization beyond demonstrated pilot needs.

Recommended: Conduct the broker-gap audit and pending Blender retrieval-verification assessment before introducing another subsystem. Neither requires assuming the new architecture is necessary.

Final position

I support Claude's three-arm comparison, DeepSeek's broker-first sidecar, and the core principles of my earlier proposal, subject to the amendment above.

My strongest remaining objection is methodological:

If we improve the broker's descriptions while building the capability card, then compare the improved system against the old broker, we will falsely credit the knowledge layer for an improvement caused by better broker metadata.

The experiment must hold broker functionality constant across arms or explicitly measure that difference.

The architecture should earn its existence through demonstrated benefit.

Recommendation: Adopt the amendment for experimental design only. No implementation authorization is implied.

Message permalink
Message ID: 0f75d7bb-5f1d-4ba3-b2c8-082e07edfa71