Answer the review: a short answer makes more survivors, and must not be skipped

Three things this got wrong. The direction: truncating Gaia leaves the HYG rows
whose counterpart it dropped without one, so survivors rise — 10 886 today,
12 711 at half the rows, 16 258 at a third — which the comment claimed was the
other way, and which decides whether the 15 000 ceiling can be leaned on at all
(it catches a truncation past about two thirds, and nothing shallower).

The throw: `fetchStars` catches everything a source throws and skips it, so a
truncated CSV was reported as "the archive was unreachable" one step after
`writeStarAssets` had already overwritten the published catalogue. Marked with
`GaiaAnswerError` and rethrown there, so an answer that cannot be worked with
fails the run where it happened. Measured end to end in a throwaway working
directory, 300 000 rows in the cache: fails, names the cache file to delete,
assets untouched. With the rethrow taken back out again: assets written, then
"the archive was unreachable".

The row limit: `rows.length >= ROW_LIMIT` is true for every reduced
ETL_GAIA_ROW_LIMIT, so the tripwire fired on exactly the deliberate slice the
override exists for — and told the operator to raise it. Gated on the same flag
as its neighbour. `ETL_GAIA_ROW_LIMIT=20000` now runs through; without the gate
it dies on the limit it was given.

Also: the row floor names the one cache file it is about rather than a glob that
takes the Hipparcos cross-match with it, and says an edited query is a third
reason it can fire — DEFAULT_QUERY_ROWS now sits under the query it counts.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016jxMkwA2rbicdGxHosecYi
This commit is contained in:
2026-09-18 14:35:17 +02:00
co-authored by Claude Opus 5
parent 44f6a8d086
commit b7f277ea04
3 changed files with 50 additions and 27 deletions
+5 -2
View File
@@ -73,8 +73,11 @@ function validateStars(stars: StarRecord[]): void {
* lacks: bright stars it saturates on, red dwarfs past its magnitude cut. So the headroom left to
* the ceiling tracks the gap between those two cutoffs as much as Gaia's completeness.
*
* This bounds a merge that went wrong. It cannot bound a Gaia download that came back short: that
* makes *fewer* survivors, not more, and is guarded where it can be seen, in `fetchGaiaStars`.
* This bounds a merge that went wrong, and — loosely — a Gaia download that came back short: a
* truncated answer leaves the HYG rows whose counterpart it dropped without one, so survivors go
* *up*, not down. Measured against the published catalogue: 10 886 today, 11 004 at nine tenths of
* the rows, 12 711 at half, 16 258 at a third. So this ceiling only catches a truncation past about
* two thirds, and `fetchGaiaStars` catches the shallower ones with its own row floor.
*/
const MAX_UNMERGED_TWINS = 100;
const MAX_HYG_SURVIVORS = 15_000;