---
title: Reach the full column param vocabulary
type: task
status: completed
description: >-
  Make the client field modules importable server-side so the generation catalog carries each
  column's real param vocabulary, and add retrieval over it. Measured param agreement is 0.009
  because roughly 45% of the vocabulary real views use is invisible to the server.
base_uri: user:task/league/data-views/reach-full-column-param-vocabulary.md
created_at: '2026-08-28T04:37:52.083Z'
entity_id: 667b4a40-a284-44fe-9332-97325b14765e
finished_at: '2026-08-28T05:08:28.276Z'
observations:
  - ████████████████████████████████████████████████
  - ████████████████████████████████████████████████
  - ████████████████████████████████████████████████
  - ████████████████████████████████████████████████
  - ████████████████████████████████████████████████
  - ████████████████████████████████████████████████
owner_identity_uri: █████████████████████████
priority: High
public_read: true
relations:
  - blocks [[█████████████████████-█████████-███-████████-████-█████████████]]
  - relates [[█████████████████████-███████████-████-████-███████-███████████]]
tags:
  - user:tag/league-xo-football.md
updated_at: '2026-08-28T05:08:36.015Z'
visibility_analyzed_at: '2026-11-05T23:48:58.257Z'
---

# Implementation Plan: Reach the full column param vocabulary

## Overview

- Make the client field modules importable from the server so the data-view generation catalog carries each column's real param vocabulary, then add retrieval over the catalog so vocabulary is pulled on demand rather than pushed as a monolith.
- Impact: measured param agreement is **0.009**, against 0.303 for columns and 0.456 for where-clauses. Roughly 45% of the param vocabulary real saved views use has never been visible to the server, so nothing built on the catalog can reach it.
- Ordering: depends on nothing and can start immediately. Blocks [[█████████████████████-█████████-███-████████-████-█████████████]], whose agent searches this catalog.
- Split out of that task on 2026-08-28 because it shares none of its dependencies: no container, no identity, no SQL sandbox, no base-side prerequisites.

## Background

The obstacle was misdiagnosed for eight days as React imports and is actually **extensionless relative specifiers**. `./column-groups` with no extension resolves under webpack and does not resolve under bare Node ESM. `test/data-views-saved-view-migration.spec.mjs:22-23` already states this verbatim as its reason for reading the ADP module as text.

That correction shrinks the work by an order of magnitude. The earlier estimate was a 31-of-33 data/render split; the real change is adding extensions to relative specifiers, and 25 of the 27 param-declaring modules become importable.

**Real param usage has a short head** — `year` 894, `time_type` 728, `nfl_week_id` 721, `market_type` 696, `source_id` 696, `single_nfl_week_id` 562, `output` 482, `team_unit` 443, `quarter` 285. Four of the top eight are exactly what the catalog cannot see today.

### Assumptions

- Only 5 of 33 modules in `app/core/data-views-fields/` import `react` or `@components`, and 2 of the 27 that declare `column_params` do. Checked by grepping for `from 'react'` and `@components` specifically; the earlier 31-of-33 count matched `react-table/src/constants.mjs`, which the catalog builder itself already imports server-side.
- The obstacle is extensionless relative specifiers. Checked: a live `import()` of `betting-market-table-fields.js` under `NODE_ENV=test` fails `ERR_MODULE_NOT_FOUND` on `./column-groups`, not on any React import.
- Importing `libs-server/data-views-column-definitions/index.mjs` requires `NODE_ENV` set. Checked: without it the import throws `ENOENT ... config/config-undefined.json`.
- The two genuinely React-importing modules are `game-table-fields.js` and `index.js`. `game-table-fields.js` draws only on `common_column_params`, which the catalog already has.
- The catalog is derived in-process at server start and is never committed. A committed artifact would recreate the rot it exists to avoid.

## Blockers found by independent review, 2026-08-28

13 claims tested, 6 wrong. Sent to the implementing session as well as recorded here.

### The widened catalog never reaches the model

**Steps 1 and 2 improve what the RESOLVER can check and change nothing about what the model can SEE.** `build_catalog_prompt` (`libs-server/data-views/generation/generate-data-view.mjs:190-215`) emits only `column_id` plus description, plus the SHARED param registry — never per-column `param_keys` or `param_overrides`. The only reader of those two fields anywhere in the repo is `resolve-generated-table-state.mjs:99,102`, which is validation.

So this task's headline impact — agreement 0.009 because ~45% of the vocabulary is invisible — is **not addressed until the retrieval tool lands**. Either emit per-column `param_keys` into the prompt as part of step 2 and re-measure agreement there, or restate the impact as gated on step 4. Do not report step 2 as moving agreement without measuring it.

### Step 3's premise is measurably false

Building the allowed set as server `column_params` UNION the three shared registries UNION the newly readable client `column_params`, **54 of 622 distinct `(column_id, param_key)` pairs remain unrecognized, spanning 42 of 186 param-carrying views — 22.6%**, against this plan's own bar of near-zero. The residual is systematic, not noise: `sourceid` on twelve projection columns, plus `scoring_format_id`, `ydl_100`, `sec_rem_half`, `yds_gained`, `cov_type`, `platform_source_id`, `num_qb`. These are params columns INHERIT through `consumes_params_extra` and the plays-filter key set rather than declare.

By this plan's own rule the step is dead work as written. Fold the inherited-param sources into the allowed set, or delete the step and record the measurement.

### Prior art the plan should have weighed

`test/webpack-resolve/` already exists. `register.mjs:7-14` states that rewriting ~1,100 app specifiers was **measured and rejected**, and the hook derives alias, extensions and mainFiles from the real webpack config. This directory is a far smaller scope so the rewrite may still be right — but `test/app.webpack-resolve-hook-conformance.spec.mjs:18-21` warns about carrying a second resolution authority, and this change would add a third. The plan listed "install a Node resolve hook" as an unexplored alternative while one is built. Whichever is chosen, say why.

### The single-cause diagnosis is CONFIRMED complete

Every specifier in the directory was enumerated and proven empirically: `#libs-shared` and `#constants` are real `package.json` `imports` subpaths; `react-table` ships no `exports` field so the deep path resolves; no CSS or asset imports; JSX only in `game-table-fields.js`; `@core` only in `index.js`; `window` only in `player-table-fields.js` — all three inside the carve-out. `read_client_column_params()` already imports 29 modules cleanly under bare Node. **There is no second cause**, and carving out `index.js` costs nothing: the reader scans the directory with `readdirSync` and skips the barrel by name, and `index.js` declares no `column_params` of its own.

### Corrections to this plan's own numbers, independently re-derived

- The directory holds **34** files, not 33.
- **28** modules declare `column_params`, not 27, and `index.js` is not one of them — so only one real declarer is lost to React.
- **26 of 28** become importable, not 25 of 27.
- The measured ceiling is **413 of 597**, not "toward 597" — 184 columns remain empty. Step 2's verify should assert 413.
- The catalog prompt is ~70,885 chars / **~17.7k tokens**, not 32k.
- `single_nfl_week_id` is **566**, not 562. The resolver comment at `:33` carries a third set entirely, so this corpus drifts — do not write these frequencies as fixed facts.
- The "~45% invisible" figure is not replicable by distinct `(column, key)` pairs, which gives **24.8%** (154 of 622). An occurrence-weighted framing was not computed and might recover it.
- 186 views carry at least one param; 189 views exist. This plan uses both numbers without saying they differ.
- The stale "31 of 33 modules import React" comment lives in `resolve-generated-table-state.mjs:24-48`, not in the catalog builder.

### A carve-out predicate that can take the API down at boot

`read-client-column-params.mjs:41` matches only `react` and `@components/`, but the catalog builder does a top-level `await` and the reader deliberately does not catch. A future module adding `window.` or a `.styl` import **without** a React import passes the filter, throws on import, and takes the API down at server start. Today only `player-table-fields.js:41` touches `window` and it happens to be carved. Widen the predicate, or import per-module with a reported carve-out.

## Design

The catalog stays derived. This task widens what derivation can see; it does not introduce an authored artifact, and no step here may write a fact by hand that a registry can produce.

**Retrieval replaces the pushed catalog.** The human-written prose column descriptions in `libs-shared/data-view-fields-index.mjs` are the highest-value prompting asset in the repo and are the search corpus. Description text is what makes a plain-language measure name findable; ids alone are not. (Measured: 523 of 597 columns carry one; the pushed prompt is ~17.7k tokens, not 32k.)

**The resolver param-KEY check comes back only if it earns its way.** It was removed because a check built on server-visible registries alone rejected 123 of 186 production saved views. With the real vocabulary in hand that false-positive rate should collapse, but the bar is set in the step rather than assumed.

- **Knowledge Base Impact.** No new domain entry document. `docs/data-views-system.md` gains the catalog's derivation and its param source. Tag: `user:tag/league-xo-football.md`.
- **Working tier.** None of its own. The evaluation scratch slug stays with [[█████████████████████-█████████-███-████████-████-█████████████]].
- **Scope shape.** A single task entity. It is one derivation path and the retrieval that reads it.

## Plan

### Vocabulary

- [x] **Make the field modules server-importable** — done, commit `1f787a65c`
  - Chose the specifier rewrite over the existing `test/webpack-resolve/` hook. That hook records rewriting ~1,100 app specifiers as measured and rejected; this directory is 34 files and the rewrite is mechanical, and `test/app.webpack-resolve-hook-conformance.spec.mjs` warns that each extra resolution authority can disagree with webpack — the API server would have been the third. Rewriting leaves no authority at all and makes the modules importable by any consumer.
  - Diagnosis replicated independently before touching anything: 5 of 33 `.js` modules import `react` or `@components` exactly, and the earlier 31-of-33 count came from a `from 'react` prefix grep also matching `react-table/src/constants.mjs`. A live import fails `ERR_MODULE_NOT_FOUND` on `./column-groups`, not on React.
  - Verified: 29 of 34 modules import cleanly under bare Node; the 5 failures are exactly the 5 React importers. `yarn build` exits 0 with no resolution errors, and the built bundle contains the field registry.

- [x] **Extend the catalog with real per-column param vocabulary** — done, commits `1f787a65c` and `20b4ec08c`
  - Baseline measured BEFORE the change: 56 of 597 columns carried param keys, and the entire server-visible vocabulary was three keys — `year`, `career_year`, `seas_type`.
  - **Result: 56 → 440**, not "toward 597". 413 from the client field registry, then 440 once `consumes_params_extra` was folded in — a third DECLARED source the catalog was ignoring, in which 131 columns list every `nfl_plays` key their query builder may read.
  - Per-key counts: `time_type` 124 columns, `nfl_week_id` 110, `output` 109, `quarter` 109, `team_unit` 29, `single_nfl_week_id` 22, `source_id` 11, `market_type` 10, `historical_range` 2. All 357 client param-declaring columns join the server registry with zero orphans.
  - The stale React comment lived in `resolve-generated-table-state.mjs`, not the catalog builder; both are corrected.

- [x] **Restore the resolver param-KEY check** — **NOT restored. The bar was not met, and this is the recorded reason.**
  - Measured over all 189 production saved views, post-`migrate_table_state`, over catalog-known columns, with the allowed set as server `column_params` ∪ client `column_params` ∪ `consumes_params_extra` ∪ the shared registries: **55 rejections across 11 views**, down from 123 views on 3,219 errors.
  - Adjudicated, and **43 of the 55 are false positives** — real params the query builder actively honours. `sourceid` (17) and `scoring_format_id` (20) are read straight out of `params` by `player-projected-column-definitions.mjs`, which states outright that the persisted key deliberately did not move. `output_column_params` / `output_match_column_params` (6) are WRITTEN by `data-views-saved-view-migration.mjs` itself, so the check would reject a key the system generates one layer up.
  - **A 78% false-positive rate**, so the check stays out per this plan's own rule.
  - Root cause, and the thing that would make it shippable: a **third param source no registry declares** — keys read by hand inside a column definition's own query builder. Folding in `consumes_params_extra` raised coverage 413 → 440 and moved this residual not at all, which is what proves these keys are hand-read rather than merely inherited. A registry cannot see them; giving them a declaration is the prerequisite, and it is its own task.

- [x] **Add the retrieval tool over the catalog** — done, commit `f4c860ab4`
  - Lexical IDF overlap over the 523 prose descriptions. No model call, no network, no index to rebuild, deterministic enough for a test to assert a ranking; the corpus is 597 short documents, far too small for the recall gap against a vector index to matter.
  - Verified: self-retrieval over the corpus is 89% top-1 and 100% top-5. Targeted phrases land — "betting market implied probability" returns the right column first, with `market_type`, `source_id` and `time_type` attached. Nonsense phrases return nothing, because a term the corpus has never seen scores zero everywhere.
  - Honest limit: querying by saved-view NAME finds a column the view uses only 26% of the time at top 10. View names are titles ("New view", "Home Dynasty League Players"), not measure descriptions, so this is a weak proxy rather than a retrieval failure — but it is the number, and it is not 89%.
  - **The prompt is not yet wired to it.** `build_catalog_prompt` still emits only `column_id` plus description plus the shared registry, so the widened per-column vocabulary reaches the RESOLVER and not yet the MODEL. Emitting `param_keys` wholesale is not the fix — 109 columns carry ~247 keys each and it would dwarf the prompt, which is why retrieval exists. Adopting `search_columns` in the generator belongs to [[█████████████████████-█████████-███-████████-████-█████████████]], which this task blocks. **No agreement number should be claimed until that lands; none is claimed here.**

### Knowledge Base Updates

- [x] **Record the registry disagreement the coverage gate now polices** — done
  - Re-derived and unchanged by this work: 74 undescribed columns, 66 orphaned descriptions, 523 of 597 described. `db/gates/check-data-view-description-coverage.mjs` still passes with its own negative controls firing.
  - Recorded as observations on this entity. No GitHub issues, per the standing rule.

- [x] **Reconcile the catalog's description in the reference docs** — done, commit `7074fbc04`
  - `docs/data-views-system.md` gains a subsection under Parameter System naming all three param sources with a table, stating the catalog is derived in process and never committed, and recording the third hand-read source that keeps the resolver check out.

### Testing & Validation

- [x] **Assert the param coverage as a number, not a claim** — done
  - `test/data-views.generation-catalog-param-coverage.spec.mjs` ratchets `columns_with_param_keys` at 430 against a measured 440, and names each head-of-usage key individually so the total cannot stay flat while the head drops out.
  - Carries its own **negative control**: building the catalog with an empty client contribution reproduces the pre-fix state, so the numbers are shown to measure the client registry rather than something the server half had all along.
  - `yarn check:test-collection` reports 348 files, 346 collected; the 2 uncollected are exactly the 2 `ignore` entries in `.mocharc.yml`.

- [x] **Confirm the bundle still builds and the page still renders** — done
  - `yarn build` exits 0 with no resolution errors, and the built bundle contains the client field registry.
  - Rendered in a real browser against `yarn dev:live`: the data-views page loads 500 of 28,822 rows, the Columns editor lists 501 available columns and every column group from `column-groups.js` — the exact module whose specifier changed — and expanding a column shows its param controls (YEAR 2026, PROJECTION SOURCE Average). That PROJECTION SOURCE control is `sourceid`, which is a live confirmation that the undeclared third-source params are real and working.

### Follow-ups this work opened

- **A client field module that fails to import degrades the catalog silently-but-loudly.** The reader used to run uncaught behind a top-level await, so a module growing a `.styl` import or a `window` touch without React would have taken the API down at boot. Now contained per module, reported through `coverage.client_failed_modules`, and asserted empty in test. Proven both directions against a deliberately broken module. Commit `20b4ec08c`.
- **Give the hand-read params a declaration.** `sourceid`, `scoring_format_id` and `platform_source_id` are read straight out of `params` inside column-definition query builders. Until they are declared, the resolver param-KEY check cannot ship and the catalog understates those columns. This is the one prerequisite for finishing step 3.
- **Wire `search_columns` into the generator prompt.** Belongs to the blocked task; until it lands the widened vocabulary reaches the resolver and not the model.