Answer the review: a short answer makes more survivors, and must not be skipped
Three things this got wrong. The direction: truncating Gaia leaves the HYG rows whose counterpart it dropped without one, so survivors rise — 10 886 today, 12 711 at half the rows, 16 258 at a third — which the comment claimed was the other way, and which decides whether the 15 000 ceiling can be leaned on at all (it catches a truncation past about two thirds, and nothing shallower). The throw: `fetchStars` catches everything a source throws and skips it, so a truncated CSV was reported as "the archive was unreachable" one step after `writeStarAssets` had already overwritten the published catalogue. Marked with `GaiaAnswerError` and rethrown there, so an answer that cannot be worked with fails the run where it happened. Measured end to end in a throwaway working directory, 300 000 rows in the cache: fails, names the cache file to delete, assets untouched. With the rethrow taken back out again: assets written, then "the archive was unreachable". The row limit: `rows.length >= ROW_LIMIT` is true for every reduced ETL_GAIA_ROW_LIMIT, so the tripwire fired on exactly the deliberate slice the override exists for — and told the operator to raise it. Gated on the same flag as its neighbour. `ETL_GAIA_ROW_LIMIT=20000` now runs through; without the gate it dies on the limit it was given. Also: the row floor names the one cache file it is about rather than a glob that takes the Hipparcos cross-match with it, and says an edited query is a third reason it can fire — DEFAULT_QUERY_ROWS now sits under the query it counts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016jxMkwA2rbicdGxHosecYi
This commit is contained in:
+5
-2
@@ -73,8 +73,11 @@ function validateStars(stars: StarRecord[]): void {
|
||||
* lacks: bright stars it saturates on, red dwarfs past its magnitude cut. So the headroom left to
|
||||
* the ceiling tracks the gap between those two cutoffs as much as Gaia's completeness.
|
||||
*
|
||||
* This bounds a merge that went wrong. It cannot bound a Gaia download that came back short: that
|
||||
* makes *fewer* survivors, not more, and is guarded where it can be seen, in `fetchGaiaStars`.
|
||||
* This bounds a merge that went wrong, and — loosely — a Gaia download that came back short: a
|
||||
* truncated answer leaves the HYG rows whose counterpart it dropped without one, so survivors go
|
||||
* *up*, not down. Measured against the published catalogue: 10 886 today, 11 004 at nine tenths of
|
||||
* the rows, 12 711 at half, 16 258 at a third. So this ceiling only catches a truncation past about
|
||||
* two thirds, and `fetchGaiaStars` catches the shallower ones with its own row floor.
|
||||
*/
|
||||
const MAX_UNMERGED_TWINS = 100;
|
||||
const MAX_HYG_SURVIVORS = 15_000;
|
||||
|
||||
Reference in New Issue
Block a user