🔬 Corpus x-ray and question SSOT
Abstract
The live bench remains the authority. A fresh full export now anchors one canonical row per published question, while every surviving local corpus layer is joined by the established 16-hex normalized stem hash. The frozen 5,247-question bundle is measured as pending; it was not imported by this work.
Fresh live measurement
The read-only exporter reproduced the historic fourteen-key contract in this order: id, stem, title, type, lang, source, wqd_id, wqd_source, opentdb_id, correct_idx, choices, explanation, subjects, difficulty. Stems come from post_content, never clipped titles. Rows with malformed choices or an out-of-range answer would be retained with an exporter defect; this run had none.
sudo -n docker compose --profile cli run --rm \
-e QW_EXPORT_PATH=/var/www/html/qw-live-full-2026-08-30.json \
-v /tmp/qw_live_full_export.php:/tmp/qw_live_full_export.php \
wpcli wp eval-file /tmp/qw_live_full_export.php
sudo -n docker compose --profile cli run --rm wpcli \
wp post list --post_type=qw_question --post_status=publish --format=count| Measure | Fresh result |
|---|---|
| Published questions | 9,073 |
| Question-index rows | 9,073 |
| Drift | 0 |
| Question-subject rows | 9,319 |
| Subject terms | 483 |
| English / German / Spanish / French | 4,715 / 4,244 / 57 / 57 |
| Exporter defect rows | 0 |
| Export SHA-256 | be1d053c…a1c78b |
The independent post count also returned 9073; the export contains 9,073 unique post IDs and every row obeys the fourteen-key contract.
One canonical ledger
The canonical local ledger is 04 QUESTIONS/ssot/corpus-ssot.jsonl. It is generated, not hand-edited, from the fresh live export. Each row includes live content, qid, canonical stem hash, every matching provenance record, a defect list, and versioned prompt/reveal narration text and hashes.
Identity follows the existing corpus convention: Unicode NFKD, remove combining marks, case-fold, punctuation to spaces, collapse whitespace, SHA-256, first 16 hex characters. Inputs and their SHA-256 values are frozen in ssot-manifest.json.
| Layer | Measured relationship to the live spine |
|---|---|
| Fresh live export | 9,073 canonical rows |
| Legacy WXR retained ledgers | 8,337 source-bearing rows available for hash joins |
| Frozen authoring bundle | 5,247 rows; 3 already collide with live, 5,244 pending |
| Authoring reserve | 288 rows, correcting the provisional 289 claim |
| Older import packs | 5 packs / 410 questions |
| Historical overlap ledger | 763 cross-bank clusters available as provenance |
| Harvest duplicate ledger | 3,293 decisions; 429 live rows matched with its separate normalized-text key |
All 9,073 live rows have the authoritative live-postmeta provenance record. That authority marker is not counted as a local corpus join: 9,011 / 9,073 live rows match at least one independent local layer, while 62 do not. Local source layers add source references, origins, historical bank membership, and duplicate-decision provenance where their appropriate identity keys match.
Warning
The frozen 5,247-question bundle is overwhelmingly unimported. This audit did not import it. Import remains a separate operational decision.
Reconciliation and defects
The fresh count is exactly the prior 9,073 evidence baseline, so the count delta is zero. The known Mona Lisa duplicate fixture still resolves correctly: live qids 357 and 1915 form the only internal normalized-stem collision group.
| Defect census | Rows |
|---|---|
| Empty explanation | 8,377 |
| Ellipsis-terminated stem | 27 |
| Other configured validator/content defects | 0 |
The live distributions are:
- language:
de 4244,en 4715,es 57,fr 57; - difficulty:
easy 3070,normal 4062,hard 1941; - type:
multiple 7779,boolean 1294.
The authoring-wave deficit artifact records a frozen 111-question shortfall across 41 target cells. Recomputing those exact (subject, difficulty) cells from the fresh live ledger gives 998 missing questions across 34 cells; 7 of the original cells now meet target. The larger current shortfall is real: the frozen have values included the authoring supply intended to fill those cells, whereas the live corpus contains only 3 of the 5,247 selected rows. xray-report.json preserves every cell’s frozen have/short, live have/short, delta, and resolved state.
Prompt/reveal TTS split
Each canonical row has deterministic narration script version 1:
prompt_text: question scaffold, stem, and lettered choices only;reveal_text: localized correct-answer scaffold plus explanation when present;- independent SHA-256 hashes for prompt and reveal;
voice_slot: null, keeping voice selection outside content identity.
English, German, Spanish, and French use localized scaffold words. Boolean rows begin with a localized true-or-false cue. HTML is stripped and entities decoded; URL tokens are omitted. Empty explanations produce only the correct-answer sentence. The correct-answer scaffold is assertion-tested never to appear in the prompt.
tts-manifest.jsonl is a resumable dry-run queue with two pending assets per question.
| Planning estimate | Value |
|---|---|
| Assets | 18,146 |
| English characters | 1,009,572 |
| German characters | 932,340 |
| Spanish characters | 18,474 |
| French characters | 18,244 |
| Audio at 15 characters/second | 36.64 hours |
| Storage at 64 kbps | 1.06 GB |
These volume figures are planning estimates, not synthesized measurements.
Five spoken-text spot renders
The generated local report contains five labeled examples: a live boolean row, a live German row, a live empty-explanation row, a synthetic HTML/entity/URL fixture, and the live longest stem. The synthetic fixture is assertion-pinned: tags disappear, entities become spoken words, the URL token is omitted, and the prompt remains answer-reveal-free.
Voice audition
The live roster returned 28 multilingual voices. The bounded evaluation audition used two male candidates (Atlas, Castor) and two female candidates (Ara, Aurora) for both English and German. Three real SSOT prompts per candidate/language produced the maximum allowed 24 clips.
- 24 of 24 synthesis requests succeeded;
- smallest clip: 154,368 bytes;
- representative English and German clip for every candidate played sequentially with
afplay, all exit 0; - bytes, latency, qid, language, gender, and content hash are recorded in the local audition log.
No winner is asserted: selection belongs to the listener. Production synthesis is explicitly out of scope and requires separately authorized service credentials rather than the evaluation credential.
Verification
python3 '04 QUESTIONS/ssot/build_ssot.py' --verify
# {"duplicates":1,"frozen_live":3,"live_deficit":998,"local_layer_covered":9011,"rows":9073,"verify":"PASS"}Two independent generator runs produced the same TTS-manifest SHA-256: 2f2c707b…e0dabb0. The current full bench gate is 1..76, assertions=76 failures=0; the stale-path guard exits 0.
- related: question corpus forensics · authoring strategy · completed log