---
title: Republish the dataset once the play enrichment backfill finishes
type: task
status: completed
description: >-
  Republished mistakia/league-data as single commit e2c58cc on 2026-09-22T14:31Z, closing 14
  artifacts worth of disagreement with the database including four play partitions whose enrichment
  columns were entirely NULL. Ledger signed by the reviewing session, not the operator; two known
  defects accepted and tracked elsewhere.
base_uri: user:task/league/republish-dataset-after-play-enrichment-backfill.md
created_at: '2026-09-22T06:03:28.480Z'
entity_id: d5d984cb-0eed-49c1-9f54-e72a703a5432
finished_at: '2026-09-22T14:34:52.003Z'
owner_identity_uri: █████████████████████████
priority: Medium
public_read: true
relations:
  - relates [[user:task/league/repair-2018-preseason-enrichment-export-gap.md]]
tags:
  - user:tag/league-xo-football.md
updated_at: '2026-09-22T14:34:52.003Z'
visibility_analyzed_at: '2026-11-05T23:50:10.928Z'
---

## Outcome — published 2026-09-22T14:31Z

Done. `mistakia/league-data` `main` is now the single commit `e2c58cc`, force-pushed over `b332169`.

The chain ran contiguously, as this task requires: coverage reports regenerated 14:18Z, export 14:19-14:26Z, oracles 14:27Z (105 of 106 PASS, all 75 play partitions `db_stability.ok` true), ledger signed 14:28Z, diff read, publish 14:29-14:31Z. Every publish gate fired — ledger `publishable` with 106 accepted, branch read from the remote as `main`, `check-data-export` clean, 33 of 33 JSON artifacts confirmed as LFS pointers, single-commit assertion held.

**The ledger is signed `claude-session-land-league-dataset-republish`, not `mistakia`.** The operator authorized the publish directly and that authorization is recorded as the `external_check` on the 2024/PRE record; the reviewer field names who actually read the evidence, which is this session.

Verified after the push: `nfl/plays/2019/PRE.csv` — one of the four partitions carrying the whole enrichment block as NULL — now agrees with the database on all five checks.

Two things were accepted into this publish rather than fixed, both examined and both tracked elsewhere:

- **`nfl/plays/2024/PRE.csv` score_reconstruction**, game 2024080151, per operator ruling. Tracked at [[user:task/league/repair-missing-play-scores-2024-hall-of-fame-game.md]].
- **One target de-attribution**: game 2025111600 play 3715 lost a correct `target_pid` because `play_enrichment` clears it on every run. Published as NULL because the database is the dataset's baseline and a hand-repair would not survive the next enrichment run. Tracked at [[████████████████████████-███-████-████-██████████████]], which owns the module and carries the changelog evidence.

## Why

The published dataset no longer matches the database, and the gap grows for as long as `scripts/backfill-play-canonical-columns.mjs` keeps running. That backfill re-runs the full play enrichment game by game across every season, and the 2026-09-22 regeneration was exported while it was mid-flight — so the published copy is a torn read, not merely a stale one.

Measured 2026-09-22 05:24 with `check-published-payload-oracles.mjs --only nfl/plays`, 4 of 75 partitions FAIL:

| partition | check | shape |
| --- | --- | --- |
| `nfl/plays/2018/PRE.csv` | population | 172 populated vs 11,033 in the database, on 12 columns |
| `nfl/plays/2018/REG.csv` | population | 43,718 vs 43,721, same 12 columns |
| `nfl/plays/2019/PRE.csv` | population | 0 vs 10,930, same 12 columns |
| `nfl/plays/2024/PRE.csv` | score_reconstruction | unrelated, database-side — see the sibling task |

The twelve columns are the ones `libs-server/play-enrichment/play-stat-column-map.mjs` writes. The set will grow: at last check the backfill was at 2019 POST and runs through 2026.

## The gate, as it stood before it was cleared

Both conditions below were met on 2026-09-22: the backfill reached `BACKFILL COMPLETE` at 06:59:49Z, and the operator ruled directly to this task's holder shortly before the 14:19Z export. Kept as the record of what the gate required, not as a live blocker.

A publish is a ~2GB force-push replacing the single commit on `mistakia/league-data`, and it is unretractable — an old commit is not a record, it is a permanent copy of whatever the gate rules were when it shipped. **This needed the operator's explicit go.**

Two things had to be true first:

1. **The backfill has finished.** Republishing into a moving table reproduces exactly the defect this task exists to clear, and guarantees a second force-push over the same partitions. Grade it on the `BACKFILL COMPLETE failures=0` line in `scratch/league/coverage-canonical-backfill/run-20260922.log`, never on a wrapper exit code. As of 2026-09-22 06:41 that line does not appear: 27 partitions complete, currently on 2025 PRE.

**Its denominator is NOT the published partition count.** The backfill walks a range it derives itself — the floor is the earliest season any of its twelve declared source columns is populated in, which the log prints on startup as 2016 (`box_defenders_tracking`), and the ceiling is `max(season_year)` on `nfl_plays`, which is 2026. That is 2016-2025 at three season types plus 2026 PRE and REG, about 32 pairs. The published dataset has 75 play partitions because it reaches back to 2000, and using that number as the denominator overstates the remaining work by a factor of more than two.
2. **The operator has ruled, to this task's holder directly.** Nothing here authorizes the push.

### On the second one, precisely

A sibling session (`█████████████████████████████████████████`) has relayed an operator ruling: hold until `BACKFILL COMPLETE`, then publish once, with the gate granted for that single post-backfill publish.

**Its SEQUENCING is adopted below as the plan of record** — publish once, after the backfill, never during — because it is correct on its own merits and matches what this task already said.

**Its AUTHORIZATION is not treated as satisfied.** A relayed approval is agent-originated, and this is the one action in the lane that cannot be taken back: the force-push replaces the single published commit outright, so whatever it ships is public immediately and whatever it replaced is gone. The cost of confirming directly is one sentence from the operator; the cost of being wrong is unretractable and public. So the relay is recorded, not spent.

This is not a disagreement about the plan and it blocks nothing today, because the backfill has not landed. When it does, one direct confirmation clears item 2 and the publish runs immediately.

## Plan

This is ONE publish, run after the backfill lands. Do not publish a partial improvement and follow it with a second force-push over the same partitions — each one replaces the published commit outright, so two passes is two irreversible acts where one would do.

1. Confirm the backfill is terminal, by its own completion line.
2. Regenerate BOTH coverage reports first — `node scripts/player-coverage-report.mjs` and `node scripts/nfl-plays-coverage-report.mjs`. They are preserved rather than regenerated by the publish, so they go stale against any schema change on their own, and `check-data-export` refuses on exactly that AFTER the rsync has already replaced `data/`, leaving the tree holding an uncommitted regeneration.
3. Run the close-out publish chain from step 24, which now opens with the concurrent-writer check: `export-published-dataset.sh` -> `check-published-payload-oracles.mjs` -> `build-regeneration-ledger.mjs` -> `apply-regeneration-review.mjs` -> `diff-published-dataset.mjs` -> `publish-dataset.sh`.
4. Require every play partition to reach `db_stability.ok true`. That check is new (league `9cd048b82`) and is the one thing standing between this and a second torn publish. It is blind to a writer that rewrites values without changing populated counts, so it is not a substitute for step 1.
5. Sign the ledger reviewer field honestly as whoever actually reviewed it. The 2026-09-22 ledger was signed `claude-session-close-player-position-split` rather than by the operator; that field is the one control between a green checker and a wrong publish.

## Found while preparing the chain

`build-regeneration-ledger.mjs` silently dropped any oracle check its `summarize_checks` did not enumerate by name, and the four it knew were the four that existed when it was written. `db_stability` reached the ledger as nothing — absent from the reviewer's summary AND from `checks_fingerprint`, which is derived from that summary.

The fingerprint half was the real hole. Measured on two ledgers built from the same partitions, one with a moved baseline and one without: **without the fix the two fingerprints are byte-identical**, so `oracle_changed_since_review` stays quiet and a prior adjudication goes on suppressing a finding nobody has seen. With it they differ.

The verdict was never at risk — `status` reads `evidence.status`, already false when any check fails — so it failed toward an unexplained `needs_adjudication` rather than a false green. Fixed in league `f2b5d8012`; unknown checks now carry through as `{ok, unsummarized}` so the next one cannot vanish the same way.

### The same hole was still open on a field, not a check

`summarize_checks` enumerated `score_reconstruction` by name, so the arm above did not cover it — but within that arm it carried the two populations its own comment calls "not verdict failures" and omitted `games_unpopulated_with_known_final`, which is the one that drives `ok: false` when `mismatches` is empty.

That is exactly the 2024/PRE case, and it made the ledger record a reviewer reads render as a FAIL with no finding attached: `mismatch_count: 0`, `games_unscored_in_db: 0`, `games_unpopulated_in_csv: 0`, and nothing else. The finding — game `2024080151`, database final 21-17 — was in the evidence file and not in the ledger. A failure that shows no finding is the shape most likely to be waved through.

The fingerprint half was open too, on the same mechanism as `db_stability`: the summary is what `checks_fingerprint` derives from, so a moved set of these games left the fingerprint byte-identical and `oracle_changed_since_review` silent.

Fixed in league `4539ee354`, which carries the entries rather than a count because a reviewer adjudicating an `ok: false` needs to see which games without leaving the ledger. Verified against a control: the failing partition now shows the game and its fingerprint moves, while passing partitions show `[]` and `0`.

**Consequence for the next ledger build.** Adding the field moves `checks_fingerprint` on every partition that runs `score_reconstruction`, so the next build flags them all as `oracle_changed_since_review` for a mechanical reason. This is the second round in a row where a summary-shape change has done that, and it compounds the review problem described in the gate below rather than relieving it — do not read the flag count as a count of real changes.

## Do not re-derive

- **The export is not defective.** [[user:task/league/repair-2018-preseason-enrichment-export-gap.md]] closed as a false finding with the backfill log timeline that pins it. A fresh 2018 export produces 65 of 65.
- **`tmp/published-dataset/` holds the 2026-09-22 audit trail** — the signed ledger, oracle evidence and structural diff, plus `currency-probe/` holding the evidence for the staleness measurement below. It is the record of what was accepted and why. Keep it until this task closes. The 2.2GB `rehearsal/` tree that sat beside it was deleted once measured stale; it is regenerable by re-running the export.

## The chain has to run contiguously, and a rehearsal tree does not keep

**A rehearsal export is stale within hours, because vendor imports keep writing the exported tables.** Measured 2026-09-22: the 07:03-07:10Z rehearsal tree passed 105 of 106 oracles when checked at 07:12Z, and three artifacts failed when the same tree was re-checked at 13:56Z against the same database. Nothing was wrong with the export — the database moved underneath it:

| artifact | check | columns that moved |
| --- | --- | --- |
| `nfl/games.csv` | population | `temperature_fahrenheit` +9, `wind_speed_mph` +9, `referee` +14 |
| `nfl/players.csv` | population | `game_designation` +5, `sumer_player_id` +1, `fantasypoints_player_id` +46 |

All six are `csv_missing` — the database gained values the CSV does not carry. Row counts held on both.

Two consequences for the plan above:

- **Export, oracle, ledger, diff and publish are ONE contiguous pass.** Resuming a chain from a tree exported hours earlier publishes a tree the oracles no longer vouch for, and the oracle run that vouched for it is the artifact that makes it look safe.
- **`diff-published-dataset.mjs` cannot see this class of drift**, so a 0-change diff is not evidence of currency. It compares headers, row counts and the artifact set — all three of which held across the six moved columns above. The oracles are the only check that reads values against the database.

## The diff is the wrong gate for this republish, and here is the right measurement

**Do not gate this republish on `diff-published-dataset.mjs`.** A zero diff is the EXPECTED result here and carries no information about whether the republish is warranted, because the entire subject of this republish is column VALUES moving from NULL to populated — which leaves headers, row counts and the artifact set untouched by construction.

The measurement that does answer it is the oracle run pointed at the published tree itself (`--dataset-path data`). Run 2026-09-22T14:01Z: **14 of 106 published artifacts disagree with the database**, while the structural diff over the same pair reports 0 changes.

| published artifact | columns moved | example |
| --- | --- | --- |
| `nfl/plays/2019/PRE.csv` | 12 of 219 | `is_fumble` 0 -> 10,930 |
| `nfl/plays/2022/PRE.csv` | 12 of 219 | `is_fumble` 0 -> 8,058 |
| `nfl/plays/2023/PRE.csv` | 12 of 219 | `is_fumble` 0 -> 8,092 |
| `nfl/plays/2021/PRE.csv` | 12 of 219 | `is_fumble` 0 -> 7,862 |
| `nfl/plays/2018/PRE.csv` | 12 of 219 | `is_fumble` 172 -> 11,033 |
| `nfl/plays/2018/REG.csv` | 12 of 219 | `is_fumble` 43,718 -> 43,721 |
| `nfl/plays/2024/PRE.csv` | 16 of 219 | `ball_carrier_pid` 2,693 -> 2,709 |
| `nfl/plays/2025/PRE.csv` | 14 of 219 | `interceptor_pid` 75 -> 76 |
| `nfl/players.csv` | 12 of 98 | `otc_player_id` 12,022 -> 12,023 |
| `nfl/games.csv` | 7 of 60 | `nfl_stadium_id` 15,278 -> 15,294 |

Plus `nfl/plays/2021/REG.csv`, `2023/REG.csv`, `2025/REG.csv` and `2025/POST.csv` on one to three columns each. Evidence in `tmp/published-dataset/published-vs-db/`.

Four of those partitions carry the enrichment columns as entirely NULL in the published copy and fully populated in the database. That is the gap this task exists to close, it is real, and the diff is blind to every row of it.

## The 74 mechanical flags, checked rather than swept

Verified 2026-09-22 by diffing the current ledger against `regeneration-review-2026-09-22-publish1.json` per artifact, comparing only the check keys present in both: **74 of the 75 `oracle_changed_since_review` flags have exactly one difference, the appearance of the `db_stability` key, and no change to any shared check.** The single substantive one is `nfl/plays/2018/PRE.csv`, whose `population` went `ok: false` with 12 column mismatches to `ok: true` with none — the backfill landing. `nfl/plays/2024/PRE.csv` is in the mechanical 74, so its FAIL does NOT self-surface through this flag; it was examined directly instead, which is how the summarizer defect above was found.