avalon-vanguard/star-map
2
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Carry the Hipparcos parallax errors between weekly refreshes, as the Gaia answers already are

Browse Source 
a81dd49 added a query of public.hipparcos_newreduction on the ESA archive, cached as
hipparcos-errors-<hash>.csv, and fetchStars requires it: a failed or short answer throws. The
refresh workflow carries only tools/etl/.cache/gaia-dr3-*.csv between runs, a glob that file does
not match, so every weekly run fetched its 117 955 rows live from the archive the cache exists to
spare, and an ESA outage would have failed the refresh even with every Gaia answer cached. Before
a81dd49 a warm cache meant no request to that archive at all.

The cache step now lists both globs. Its key gains "-hipparcos": actions/cache keys are immutable,
and an entry already saved under the old key would be restored without the new file and never
saved again. Checked with Node's path.matchesGlob against the local cache (gaia-dr3-54fdbc7a,
gaia-dr3-hip-785b92fc and hipparcos-errors-f846b045 match, the archive's own files do not) and by
parsing the workflow with PyYAML. A workflow file has no unit test to fail without the change.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Senrokai
2026-09-29 20:46:29 +02:00
co-authored by Claude Opus 5.5
parent 08bf8b18a6
commit 4ebe663a84
1 changed files with 9 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/data-refresh.yml
+9 -4
View File
@@ -41,12 +41,17 @@ jobs:
# Gaia DR3 is a frozen release: the same query returns the same bytes (a live re-fetch has # Gaia DR3 is a frozen release: the same query returns the same bytes (a live re-fetch has
# reproduced stars.bin exactly), so its responses are carried from one run to the next # reproduced stars.bin exactly), so its responses are carried from one run to the next
# rather than re-downloaded every week from an archive that times out under load. The key # rather than re-downloaded every week from an archive that times out under load. So is
# follows gaia.ts, where the queries are written, so a changed query is fetched afresh. # van Leeuwen's 2007 Hipparcos reduction, as frozen and on the same archive, whose parallax
# errors fetchStars requires: left out, it was fetched live every week, and an ESA outage would
# have failed the refresh with every Gaia answer cached. The key follows gaia.ts, where the
# queries are written, so a changed query is fetched afresh.
- uses: actions/cache@v4 - uses: actions/cache@v4
with: with:
path: tools/etl/.cache/gaia-dr3-*.csv path: |
key: gaia-dr3-${{ hashFiles('tools/etl/sources/gaia.ts') }} tools/etl/.cache/gaia-dr3-*.csv
tools/etl/.cache/hipparcos-errors-*.csv
key: gaia-dr3-hipparcos-${{ hashFiles('tools/etl/sources/gaia.ts') }}
# Every other source is fetched live on this fresh runner. A failed fetch fails the run by # Every other source is fetched live on this fresh runner. A failed fetch fails the run by
# design — no refresh is better than a partial one. That includes Gaia on a cold cache, by # design — no refresh is better than a partial one. That includes Gaia on a cold cache, by