Refuse a Gaia answer that came back short, and read the body inside the retry
The merge gate asks whether Gaia contributed any stars, never how many. The TAP service truncates on its own timeout and still serves a well-formed CSV with a 200, ordered by magnitude — so a half answer is the bright half, which is the half HYG overlaps. Every gate passes: Gaia stars are present, HYG survivors go down rather than up, unmerged twins can only fall. The weekly job would publish a catalogue missing two hundred thousand stars and the runner would cache it for the weeks after. `fetchGaiaStars` now refuses fewer than 95% of the 412 765 rows its query holds, as its sibling query already did, and refuses an answer that fills the row limit. `fetchText` retried the request but not the body: a connection reset part-way through the 57 MB CSV rejected out of the loop, with no wait and no second attempt. The read now happens inside it. Also corrected: the merge gate's account of the HYG survivors (two thirds of them are stars Gaia measures but the main query never downloads, since Gaia puts them past the 250 pc cutoff), and the refresh workflow's comment on what happens when the archive is unreachable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016jxMkwA2rbicdGxHosecYi
This commit is contained in:
@@ -49,9 +49,11 @@ jobs:
|
||||
key: gaia-dr3-${{ hashFiles('tools/etl/sources/gaia.ts') }}
|
||||
|
||||
# Every other source is fetched live on this fresh runner. A failed fetch fails the run by
|
||||
# design — no refresh is better than a partial one. That includes Gaia on a cold cache:
|
||||
# the ETL skips it when unreachable, and the merge gate in build.ts then refuses a
|
||||
# catalogue it contributed nothing to.
|
||||
# design — no refresh is better than a partial one. That includes Gaia on a cold cache, by
|
||||
# two different paths: its Hipparcos cross-match is required, so an unreachable archive
|
||||
# fails the run from fetchStars itself, while its main query is skipped when unreachable and
|
||||
# the merge gate in build.ts then refuses a catalogue it contributed nothing to. An archive
|
||||
# that answers short rather than not at all is caught in fetchGaiaStars.
|
||||
- name: Rebuild the datasets
|
||||
run: npm run etl
|
||||
|
||||
|
||||
Reference in New Issue
Block a user