Write the star assets once, after the archive's hosts are added, not before them as well
Since494fb56and4c8e4a0, fetchStars wrote stars.bin, stars-meta.bin and stars-index.json, and fetchExoplanets wrote them again with the 3 277 hosts it adds from the archive and the 574 Gaia designations it renames, after the live Horizons and archive fetches in between. A run stopped between the two writes, or fetchStars.ts run on its own as the README documented, left 452 294 stars on disk with no archive star and 574 hosts back to their designations, and the existing exoplanets.json pointing 4 237 planets at stars that were not there. It happened on this branch: the dev server served those files. fetchStars now returns the stars and writes nothing; fetchExoplanets, which build.ts runs after it, is the only writer. fetchStars.ts run on its own runs fetchExoplanets, which fetches the stars itself. Checked from cache: `tsx tools/etl/fetchStars.ts` exits 0 and writes 455 571 stars, the published files byte for byte; build.ts with fetchSolarSystem made to throw fails with "simulated Horizons outage" and leaves every data file unchanged, where before it left the 452 294-star catalogue. The README's table now names fetchExoplanets.ts as what writes them. The ETL has no unit harness; these two runs are the check. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -71,8 +71,11 @@ function spectralTypeOf(row: Record<string, string>): string {
|
||||
/**
|
||||
* Downloads the HYG (Hipparcos/Yale/Gliese) stellar database, places each star along its
|
||||
* equatorial direction (epoch J2000.0) at whichever of its Hipparcos and Gaia distances has the
|
||||
* smaller error, keeps the ones either survey puts within range, unions the other positional
|
||||
* sources, and writes `stars.bin` (packed positions) + `stars-index.json` (everything else).
|
||||
* smaller error, keeps the ones either survey puts within range, and unions the other positional
|
||||
* sources. Writes nothing: fetchExoplanets adds the hosts the catalogue lacks and renames the
|
||||
* ones known only by a designation, then writes the star assets once. Written here as well, they
|
||||
* stood on disk without those 3 277 stars whenever a run stopped between the two — and that
|
||||
* catalogue left 4 237 planets pointing at stars it did not have.
|
||||
*/
|
||||
export async function fetchStars(): Promise<StarRecord[]> {
|
||||
console.log(`Fetching HYG star catalog (distance cutoff: ${DISTANCE_CUTOFF_PC} pc)...`);
|
||||
@@ -162,7 +165,6 @@ export async function fetchStars(): Promise<StarRecord[]> {
|
||||
|
||||
const merged = await mergeWithOtherSources(stars);
|
||||
merged.sort((a, b) => a.id - b.id);
|
||||
writeStarAssets(merged);
|
||||
return merged;
|
||||
}
|
||||
|
||||
@@ -224,8 +226,9 @@ export function writeStarAssets(stars: StarRecord[]): void {
|
||||
writeFileSync(dataPath('stars-index.json'), JSON.stringify(index));
|
||||
}
|
||||
|
||||
// On its own it writes what the ETL would, hosts included; imported late, since fetchExoplanets imports this module.
|
||||
if (require.main === module) {
|
||||
fetchStars().catch((error) => {
|
||||
import('./fetchExoplanets').then(({ fetchExoplanets }) => fetchExoplanets()).catch((error) => {
|
||||
console.error(error);
|
||||
process.exitCode = 1;
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user