---
title: Harden report_job terminal jobs-insert against post-run connection blips
type: task
status: completed
description: >-
  Make report-job.mjs resilient so a transient DB connection error on its terminal jobs-table audit
  insert cannot fail an otherwise-successful long-running import job.
base_uri: user:task/league/harden-report-job-terminal-jobs-insert.md
created_at: '2026-07-12T17:34:18.061Z'
entity_id: c091b633-bd99-4a79-9a8c-9aeb156df819
finished_at: '2026-07-12T18:33:33.604Z'
observations:
  - ████████████████████████████████████████████████
  - ████████████████████████████████████████████████
  - ████████████████████████████████████████████████
owner_identity_uri: █████████████████████████
priority: Low
public_read: true
tags:
  - user:tag/league-xo-football.md
  - user:tag/homelab.md
updated_at: '2026-07-12T18:33:59.556Z'
visibility_analyzed_at: '2026-11-06T00:17:56.004Z'
---

## Background

On 2026-07-12 05:48 UTC a supplier projections import's scheduled run (signal 120514) failed at `libs-server/report-job.mjs:62` — `await db('jobs').insert(...)` — with a knex error carrying `sql: undefined, bindings: undefined`, i.e. a pre-query connection-acquisition failure, not a bad query. Investigation confirmed the import pipeline and DB are otherwise healthy: the projections job succeeded on every recent scheduled day (07-02..07-11, 8/8), and a direct probe connects fine as the production writer role with the `jobs` table queryable. So the failure was a transient connection blip on the terminal audit write after the ~18-min run, not a config/auth problem. Signal 120514 is a runs-primitive pipeline_failure that auto-resolves on the next successful scheduled run.

The latent defect: `report_job`'s terminal `db('jobs').insert` has no reconnect/retry and runs after a long job during which the pooled connection can go stale (server-side idle timeout or a network blip). A single such blip fails an otherwise-successful import and pages a high-severity signal.

## Plan

- [ ] Wrap the terminal `db('jobs').insert` in `report_job` in a bounded retry that re-acquires a fresh connection on connection-class errors (KnexTimeoutError, "Connection terminated unexpectedly", ECONNRESET), so a stale-pool blip does not fail the job report.
- [ ] Consider pool hardening in `db/index.mjs` (knex/tarn): `pool.afterCreate` liveness ping and/or `acquireConnectionTimeout` tuning so long jobs validate a connection before the terminal write.
- [ ] Ensure a failed audit-row insert does not mask a successful import outcome — the job's real success/failure should reach the runs primitive independent of the local `jobs` bookkeeping insert.
- [ ] Verify against a simulated stale connection (or a forced pool destroy mid-run) that the terminal insert recovers.

## Notes

- Same terminal-insert fragility applies to every import that ends with `report_job` (every supplier and NFL import), so the fix is fleet-wide, not projections-specific.
- Relates to the surface-pipeline-failures discipline: the fix must keep the failure oracle honest — do not swallow genuine failures while adding resilience to the transient class.