✍️ Question authoring strategy — 200/category with dual gates

Scope

This page fixes the authoring contract for growing the corpus toward ~200 published, grounded, mostly-easy multiple-choice questions per major category, with broad child-topic coverage. It complements the corpus contract (data shape, provenance) — this page is about writing new questions well and proving the result. Live counts as of 2026-08-19: 9,069 published questions, index 9,069, drift 0 (see legacy WXR import).

Why: the real bottleneck is subject depth

The sufficiency census showed only 2 of 456 subjects could satisfy the default 10-question English progressive quiz before the imports (census). The constraint is content per subject, not code. But can_satisfy alone is a floor, not a target — it passes at 10 questions and proves neither ~200/category nor balanced child coverage. Hence dual gates (below).

Format: author natively in question-pack v1

quizwizz.question-pack v1 is the robust, import-proven format (8,337 questions in one run). Author against QuestionValidator rules so nothing bounces:

• stem ≤2000 chars; 2–8 distinct choices ≤1000 each (distinctness is lowercase + whitespace-collapsed)
• exactly one correct_idx — no multi-select (the 7 qs_json rejects proved this)
• explanation ≤4000; difficulty easy|normal|hard; path-slug subjects (games-chess) reused from the existing term tree, never invented
• ≤100 questions per pack, one language per pack
• no raw HTML/< in choices — wp_strip_all_tags() empties <1%; write less than 1%

Plan the matrix before the prose

Per major category (Geography, Science, History, Entertainment, Sports, Arts, …):

1. Enumerate 8–12 child subtopics from the existing qw_subject tree.
2. Allocate quotas: 200 ≈ 12 children × ~17 questions, weighted to the famous core of each child.
3. Difficulty split ~70% easy / 25% normal / 5% hard.

The matrix cell — “Geography → rivers → easy → #14” — is the unit of work. This prevents clumping (50 capital questions, zero river questions) and drift into obscurity.

What makes a great question

1. One unambiguous answer — a domain expert answers instantly and never argues. Ban time-decaying facts (“current champion”), convention-dependent facts (continent counts), and stems where two options are arguably correct.
2. Same-kind distractors — wrong answers are the same category of thing (three other painters, not a river). Easy = famous fact + honest distractors, never one real option and three jokes.
3. Self-contained stem — no external context, no “which of the following”, no negatives in the easy tier.
4. Guessable at exactly 25% — no length/grammar leak toward the correct choice.
5. The explanation teaches. Pattern: confirm the fact in one sentence, then one adjacent surprise — the delayed “aha”. Example: Mercury is closest to the Sun — yet Venus is the hottest planet, because its CO₂ atmosphere traps heat better than proximity does. Explanation = fact + why/contrast + one hook, 2–3 sentences, never a stem restatement.

Production pipeline

Batched per matrix cell: draft → mechanical lint → fact-check → dedupe → pack → import → gates.

1. Draft 20 candidates per cell against a strict template (stem / 4 choices / correct_idx / explanation / source / difficulty).
2. Mechanical lint offline (reuse the build_import_packs.py pattern): schema, choice distinctness, length caps, banned patterns (negation, “all of the above”, “current/latest/recently”, HTML).
3. Fact-check as its own pass, each question against its stated source (qw_source is stored per question). Reject, don’t repair — drafting is cheap.
4. Dedupe by stem hash against live post_content — never post titles (321 of 679 titles were ellipsis-truncated; see forensics).
5. Import via QuestionPackImporter::import_json(), ≤100/pack, draft-quiz receipts as resume markers.

Acceptance: dual gates

Gate 1 — playability floor (automated, existing). can_satisfy(subject, default 10-question en quiz) passes for every target subject. Probe Wave 6 assertion 12 defends this.

Gate 2 — coverage ledger (the real target). A script emits a per-category table and exits non-zero on any red cell. Data sourcing matters: the index tables (wp_qwizz_question_index, wp_qwizz_question_subjects) are authoritative only for counts, language, difficulty, and subject membership; qw_source and qw_explanation live in post meta and stems in post_content, so those checks must join wp_posts/wp_postmeta (or load canonical question records via the service layer).

Check	Threshold	Source of truth
Category total	≥200 published	index tables
Per-child quota	every planned child ≥ its matrix quota; no child at 0	index + subjects tables
Easy share	60–80% easy (a band — catches drift both ways)	index tables
Source verification	100% of new rows carry qw_source; fact-check pass recorded per pack manifest	postmeta
Dedupe	0 stem-hash collisions vs live post_content	posts table
Explanation quality	0 empty explanations on new rows; ~10%/pack human sample confirms fact-plus-hook pattern	postmeta

“Done” for a category = both gates green, ledger output archived as evidence, and the row logged in the Completed log. Chasing Gate 1 alone would rationally dump 10 questions into each of 456 subjects — the ledger makes that strategy score zero.

Scaling: 10x, 100x, and across languages

Proven today, hypothesis beyond: the importer is proven at 8,337 questions in one sequential run (packs <=100, resumable via draft receipts); the read model is a flat indexed table with per-question lang; the same language-neutral qw_subject tree serves every language (en 4,711 / de 4,244 rows share the 483 terms); and both gates filter by lang, so a category is done per (category, lang) cell with no new machinery. Pack-level concurrency is untested — concurrent WordPress imports can contend on post/meta writes, term counting, cache invalidation, receipt/idempotency checks, and index maintenance. Treat 10x/100x as hypotheses gated by staged load tests (10k -> 100k -> 1M synthetic rows on a scratch bench): measure import throughput, lock waits/errors, DB size, quiz-query p95, reindex and backup time per stage, verify published count = index count with drift 0 after every wave, and raise import concurrency only from those measurements.

At 10x (~90k): fact-checking is the only human-linear step — tier it: LLM-drafted prose keeps the per-question check, dataset-derived questions shift verification to the generator (spot-check per template). Add near-duplicate detection (normalized-token or embedding similarity) to the offline lint — exact stem-hash misses paraphrases. Keep Gate-2 postmeta checks scoped to new rows per wave; whole-corpus checks stay on the index tables so wp_postmeta joins never rescan everything.

At 100x (~1M): templates over prose. The unit of authorship becomes the template x structured dataset (Wikidata-class tables): distractors drawn from the same dataset column are same-kind by construction in any language; explanations templated with the adjacent-surprise hook from a neighbouring field; qw_source becomes machine provenance (dataset + row). Verification effort = templates x language adapters, not questions. The matrix becomes data (cells reference template + dataset + quota) and a generator fills it. New quality risk is monotony, not wrong facts — cap per-template share per quiz.

Languages: UI locale (gettext, de at 1648/1648 parity) and content language (per-question lang) are separate axes. Universal facts generate per-language from the same dataset row via localized templates — the shared row prevents factual divergence, but not the rest: dataset labels can be missing or fall back to another language, grammar needs locale-specific inflection and articles, transliteration aliases can collapse into duplicate choices after normalization, and which distractors feel familiar differs by locale. So each language is not free: it is a versioned language adapter (template translation + label-source policy + morphology rules + alias normalization), validated 100% mechanically, with sampled native-speaker review per (template, locale) — not merely per template. Cheaper than authoring, never zero. Locale-native content (idioms, national history, local culture) is authored per language against its own matrix cells — that is where human editorial budget goes at every scale. Never freehand machine-translate finished questions: distinctness collapses and the validator cannot catch a wrong translated answer.

Correction — 2026-08-28

An earlier revision of this section claimed localized templates were “near-zero marginal cost” with “no drift between a translation and its fact”. Retracted same day: only factual divergence is prevented by the shared row; label fallback, morphology, alias duplication, and distractor familiarity all cost per-locale work (hence the language-adapter requirement above).

Cost curve: 1x = human drafting; 10x = LLM drafting + scripted lint + full fact-check; 100x = datasets + templates + per-template verification + sampled review. The dual gates never change — only the generator behind them does.

Anti-goals

• No new qw_subject slugs without an explicit decision.
• No multi-select questions until the validator supports them.
• No question counted “done” without a source and a non-empty explanation.
• No measuring success in raw question count — the metric is categories with both gates green.