Write the star assets once, after the archive's hosts are added, not before them as well

Since 494fb56 and 4c8e4a0, fetchStars wrote stars.bin, stars-meta.bin and stars-index.json, and
fetchExoplanets wrote them again with the 3 277 hosts it adds from the archive and the 574 Gaia
designations it renames, after the live Horizons and archive fetches in between. A run stopped
between the two writes, or fetchStars.ts run on its own as the README documented, left 452 294
stars on disk with no archive star and 574 hosts back to their designations, and the existing
exoplanets.json pointing 4 237 planets at stars that were not there. It happened on this branch:
the dev server served those files.

fetchStars now returns the stars and writes nothing; fetchExoplanets, which build.ts runs after
it, is the only writer. fetchStars.ts run on its own runs fetchExoplanets, which fetches the
stars itself. Checked from cache: `tsx tools/etl/fetchStars.ts` exits 0 and writes 455 571 stars,
the published files byte for byte; build.ts with fetchSolarSystem made to throw fails with
"simulated Horizons outage" and leaves every data file unchanged, where before it left the
452 294-star catalogue. The README's table now names fetchExoplanets.ts as what writes them.

The ETL has no unit harness; these two runs are the check.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-09-30 16:21:03 +02:00
co-authored by Claude Opus 5.5
parent 073fb9f718
commit 734b048c9e
2 changed files with 9 additions and 6 deletions
+2 -2
View File
@@ -200,9 +200,9 @@ re-runs are cheap and offline-friendly; set `ETL_FORCE_REFRESH=1` to bypass the
| Script | Source | Output | | Script | Source | Output |
| --- | --- | --- | | --- | --- | --- |
| `fetchStars.ts` | HYG database, plus any other positional catalogue wired in (see below) | `stars.bin`, `stars-meta.bin`, `stars-index.json` | | `fetchStars.ts` | HYG database, plus any other positional catalogue wired in (see below) | the catalogue stars, handed to `fetchExoplanets.ts` |
| `fetchSolarSystem.ts` | JPL Horizons / SSD | `bodies.json` | | `fetchSolarSystem.ts` | JPL Horizons / SSD | `bodies.json` |
| `fetchExoplanets.ts` | NASA Exoplanet Archive (TAP) | `exoplanets.json` | | `fetchExoplanets.ts` | NASA Exoplanet Archive (TAP), and the stars above with the hosts it adds | `exoplanets.json`, `stars.bin`, `stars-meta.bin`, `stars-index.json` |
| `fetchDeepSky.ts` | OpenNGC | `deepsky.json` | | `fetchDeepSky.ts` | OpenNGC | `deepsky.json` |
The star catalogue ships as two binary column stores plus a small JSON file, not as an array of The star catalogue ships as two binary column stores plus a small JSON file, not as an array of
+7 -4
View File
@@ -71,8 +71,11 @@ function spectralTypeOf(row: Record<string, string>): string {
/** /**
* Downloads the HYG (Hipparcos/Yale/Gliese) stellar database, places each star along its * Downloads the HYG (Hipparcos/Yale/Gliese) stellar database, places each star along its
* equatorial direction (epoch J2000.0) at whichever of its Hipparcos and Gaia distances has the * equatorial direction (epoch J2000.0) at whichever of its Hipparcos and Gaia distances has the
* smaller error, keeps the ones either survey puts within range, unions the other positional * smaller error, keeps the ones either survey puts within range, and unions the other positional
* sources, and writes `stars.bin` (packed positions) + `stars-index.json` (everything else). * sources. Writes nothing: fetchExoplanets adds the hosts the catalogue lacks and renames the
* ones known only by a designation, then writes the star assets once. Written here as well, they
* stood on disk without those 3 277 stars whenever a run stopped between the two — and that
* catalogue left 4 237 planets pointing at stars it did not have.
*/ */
export async function fetchStars(): Promise<StarRecord[]> { export async function fetchStars(): Promise<StarRecord[]> {
console.log(`Fetching HYG star catalog (distance cutoff: ${DISTANCE_CUTOFF_PC} pc)...`); console.log(`Fetching HYG star catalog (distance cutoff: ${DISTANCE_CUTOFF_PC} pc)...`);
@@ -162,7 +165,6 @@ export async function fetchStars(): Promise<StarRecord[]> {
const merged = await mergeWithOtherSources(stars); const merged = await mergeWithOtherSources(stars);
merged.sort((a, b) => a.id - b.id); merged.sort((a, b) => a.id - b.id);
writeStarAssets(merged);
return merged; return merged;
} }
@@ -224,8 +226,9 @@ export function writeStarAssets(stars: StarRecord[]): void {
writeFileSync(dataPath('stars-index.json'), JSON.stringify(index)); writeFileSync(dataPath('stars-index.json'), JSON.stringify(index));
} }
// On its own it writes what the ETL would, hosts included; imported late, since fetchExoplanets imports this module.
if (require.main === module) { if (require.main === module) {
fetchStars().catch((error) => { import('./fetchExoplanets').then(({ fetchExoplanets }) => fetchExoplanets()).catch((error) => {
console.error(error); console.error(error);
process.exitCode = 1; process.exitCode = 1;
}); });