Draw each HYG star at Gaia's distance, and keep the ones Hipparcos misplaced

HYG and Gaia were both cut at 250 pc, each on its own distance. A star
Hipparcos put at 200 pc and Gaia at 300 was kept by the first, never
downloaded from the second, and drawn at 200. That is where 83% of the
9 691 mid-magnitude HYG stars without a Gaia counterpart came from, and at
the median Hipparcos had them a third too close. The mirror case, Hipparcos
outside and Gaia inside, dropped the HYG row and left its Gaia entry
anonymous.

Gaia's own Hipparcos cross-match (hipparcos2_best_neighbour, a fixed DR3
table of 99 525 rows) gives a usable Gaia distance for 97 751 of them.
placementDistancePc keeps a star either survey puts inside the cutoff, and
draws every kept star at the better measurement, inside the cutoff or not.
57 121 HYG stars now sit at Gaia's distance. 6 833 of them are past 250 pc:
Zet Per 230 -> 259 pc, 35 Ori 137 -> 330, 44 Cnc 223 -> 613, and the
farthest, HIP 69445, at 8.7 kpc. 3 666 stars that Hipparcos put outside are
now kept, and 3 656 of them give a Gaia entry its name.

The cross-match is required rather than skipped when unreachable. Without
it, every one of those stars would move back to its Hipparcos distance, and
the published map would flip with the archive's availability. The ESA TAP
answered it with a 500 at first and in 102 s on the next try. So fetches
now retry 5xx and network failures twice, after 30 s and 120 s, in the
fetch every source goes through. The refresh job also carries the Gaia DR3
responses from run to run in the Actions cache: the release is frozen, and
a live re-fetch has already reproduced stars.bin byte for byte.

423 651 stars (+10), 61 168 HYG rows folded into Gaia entries (+3 656),
351 597 unnamed designations (-3 656). 10 886 HYG survivors and 23 unmerged
pairs under an arcsecond, both inside the merge gate's ceilings. The same
1 972 exoplanets have a host; KELT-4 A b and MWC 758 c now sit on their
named star.

The HUD's "Radius" becomes "Survey radius": 250 pc is where Gaia is
surveyed to, and no longer the edge of the map.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016jxMkwA2rbicdGxHosecYi

tools/etl/fetchStars.ts

@@ -1,8 +1,9 @@
 import { writeFileSync } from 'node:fs';
 
-import { mergeStarCatalogues } from '../../src/app/shared/astro/star-merge';
+import { mergeStarCatalogues, placementDistancePc } from '../../src/app/shared/astro/star-merge';
 import { encodeStarCatalog } from '../../src/app/shared/models/star-catalog';
 import { StarRecord, SUN_STAR_ID } from '../../src/app/shared/models/star.model';
+import { fetchGaiaDistancesByHip } from './sources/gaia';
 import { positionalSources } from './sources/registry';
 import { PARALLAX_PRECISION_MAS } from './sources/star-sources';
 import { parseCsvObjects, parseOptionalNumber } from './lib/csv';
@@ -19,13 +20,14 @@ const HYG_UNKNOWN_DISTANCE_PC = 100000; // HYG's placeholder for unmeasured/unre
 const UNKNOWN_MAGNITUDE = 15;
 
 /**
- * Stars within this distance (parsecs) of the Sun are kept for the galaxy view.
+ * Stars either survey places within this distance (parsecs) of the Sun are kept for the galaxy
+ * view; `placementDistancePc` decides which distance a kept star is drawn at.
  *
- * Set at the range HYG's own measurements reach rather than at a round number. 98.6% of its
- * rows carry a Hipparcos identifier, and Hipparcos parallaxes are good to roughly a
- * milliarcsecond — so at 250 pc (4 mas) a star's distance is uncertain by some tens of per
- * cent, and beyond it the catalogue is plotting noise. Note that only the *radial* placement
- * blurs: a star's direction on the sky stays exact at any distance.
+ * Set at the range Hipparcos's own measurements reach rather than at a round number: its
+ * parallaxes are good to roughly a milliarcsecond, so at 250 pc (4 mas) a distance is uncertain
+ * by some tens of per cent. That is why it is not applied to the Hipparcos distance alone:
+ * Gaia puts 6 833 of the stars Hipparcos places inside it outside, and 3 666 the other way
+ * round. Only the *radial* placement blurs; a star's direction on the sky stays exact.
  *
  * The catalogue is also magnitude-limited, so this is not a volume-complete sample beyond about
  * 50 pc: it thins to the intrinsically bright, which is the same selection the naked eye makes.
@@ -58,27 +60,35 @@ function resolveName(row: Record<string, string>): string {
 }
 
 /**
- * Downloads the HYG (Hipparcos/Yale/Gliese) stellar database, takes each star's equatorial
- * Cartesian position (parsecs, epoch J2000.0), filters by distance, unions the other positional
- * sources, and writes `stars.bin` (packed positions) + `stars-index.json` (everything else).
+ * Downloads the HYG (Hipparcos/Yale/Gliese) stellar database, places each star along its
+ * equatorial direction (epoch J2000.0) at the better of its Hipparcos and Gaia distances, keeps
+ * the ones either survey puts within range, unions the other positional sources, and writes
+ * `stars.bin` (packed positions) + `stars-index.json` (everything else).
  */
 export async function fetchStars(): Promise<StarRecord[]> {
   console.log(`Fetching HYG star catalog (distance cutoff: ${DISTANCE_CUTOFF_PC} pc)...`);
   const csv = await fetchTextCached(HYG_CSV_URL, 'hygdata_v41.csv');
   const rows = parseCsvObjects(csv);
+  // Not skipped when unreachable, unlike the positional sources below; see its own comment.
+  const gaiaPcByHip = await fetchGaiaDistancesByHip();
 
   const stars: StarRecord[] = [];
+  let atGaiaDistance = 0;
+  let pastCutoff = 0;
 
   for (const row of rows) {
     const id = Number(row['id']);
-    const distancePc = Number(row['dist']);
 
     if (id === SUN_STAR_ID) {
       stars.push({ id, name: 'Sol', x: 0, y: 0, z: 0, magnitude: parseOptionalNumber(row['mag']) ?? UNKNOWN_MAGNITUDE, spectralType: row['spect'] || 'G2V', colorIndex: parseOptionalNumber(row['ci']) ?? null });
       continue;
     }
 
-    if (!Number.isFinite(distancePc) || distancePc >= HYG_UNKNOWN_DISTANCE_PC || distancePc > DISTANCE_CUTOFF_PC) {
+    const hygPc = Number(row['dist']);
+    const hipparcosPc = Number.isFinite(hygPc) && hygPc > 0 && hygPc < HYG_UNKNOWN_DISTANCE_PC ? hygPc : undefined;
+    const gaiaPc = row['hip'] ? gaiaPcByHip.get(Number(row['hip'])) : undefined;
+    const distancePc = placementDistancePc(hipparcosPc, gaiaPc, DISTANCE_CUTOFF_PC);
+    if (distancePc === null) {
       continue;
     }
 
@@ -89,26 +99,37 @@ export async function fetchStars(): Promise<StarRecord[]> {
     // once brought to the same epoch — have it. 1813 stars differ by over an arcsecond, and the
     // Cartesian columns are the ones Gaia agrees with for 1155 of them against 156 (one of those,
     // HIP 57146, has x/y/z 161″ from its own ra/dec and stays double).
+    //
+    // Only their direction is used. They sit at HYG's own distance, or at its 100 000 pc
+    // placeholder where it has none, and are carried along that direction to the one chosen above.
     const x = Number(row['x']);
     const y = Number(row['y']);
     const z = Number(row['z']);
-    if (![x, y, z].every(Number.isFinite)) {
+    const length = Math.hypot(x, y, z);
+    if (![x, y, z].every(Number.isFinite) || length === 0) {
       continue;
     }
+    const scale = distancePc / length;
+    if (gaiaPc !== undefined) {
+      atGaiaDistance++;
+    }
+    if (distancePc > DISTANCE_CUTOFF_PC) {
+      pastCutoff++;
+    }
 
     stars.push({
       id,
       name: resolveName(row),
-      x,
-      y,
-      z,
+      x: x * scale,
+      y: y * scale,
+      z: z * scale,
       magnitude: parseOptionalNumber(row['mag']) ?? UNKNOWN_MAGNITUDE,
       spectralType: row['spect'] || 'Unknown',
       colorIndex: parseOptionalNumber(row['ci']) ?? null
     });
   }
 
-  console.log(`  kept ${stars.length} stars (of ${rows.length} in the catalog).`);
+  console.log(`  kept ${stars.length} stars (of ${rows.length} in the catalog): ${atGaiaDistance} at Gaia's distance, ${pastCutoff} of them past ${DISTANCE_CUTOFF_PC} pc.`);
 
   const merged = await mergeWithOtherSources(stars);
   merged.sort((a, b) => a.id - b.id);

tools/etl/lib/http.ts

@@ -17,17 +17,38 @@ export async function fetchTextCached(url: string, cacheKey: string): Promise<st
   }
 
   console.log(`  fetching ${url}`);
-  const response = await fetch(url);
-  if (!response.ok) {
-    throw new Error(`Failed to fetch ${url}: ${response.status} ${response.statusText}`);
-  }
-  const text = await response.text();
+  const text = await fetchText(url);
 
   mkdirSync(dirname(cachePath), { recursive: true });
   writeFileSync(cachePath, text, 'utf-8');
   return text;
 }
 
+/**
+ * How long to wait before each retry of a failed request. The archives this reads are public
+ * services that time out under load — the Gaia TAP has answered a five-row join in two and a
+ * half minutes and a full one with a 500 — and a weekly refresh that gives up on the first of
+ * those publishes nothing that week.
+ */
+const RETRY_DELAYS_MS = [30_000, 120_000];
+
+async function fetchText(url: string): Promise<string> {
+  for (let attempt = 0; ; attempt++) {
+    const response = await fetch(url).catch((error: unknown) => (error instanceof Error ? error : new Error(String(error))));
+    if (!(response instanceof Error) && response.ok) {
+      return response.text();
+    }
+    const reason = response instanceof Error ? response.message : `${response.status} ${response.statusText}`;
+    // A 4xx is the request's own fault, and waiting will not change the answer.
+    const retryable = response instanceof Error || response.status >= 500;
+    if (!retryable || attempt >= RETRY_DELAYS_MS.length) {
+      throw new Error(`Failed to fetch ${url}: ${reason}`);
+    }
+    console.log(`  ${reason}; trying again in ${RETRY_DELAYS_MS[attempt] / 1000} s`);
+    await new Promise((resolve) => setTimeout(resolve, RETRY_DELAYS_MS[attempt]));
+  }
+}
+
 /** Convenience wrapper around {@link fetchTextCached} that parses the cached response as JSON. */
 export async function fetchJsonCached<T>(url: string, cacheKey: string): Promise<T> {
   return JSON.parse(await fetchTextCached(url, cacheKey)) as T;

tools/etl/sources/gaia.ts

@@ -125,3 +125,52 @@ export async function fetchGaiaStars(): Promise<StarRecord[]> {
   console.log(`  kept ${stars.length} Gaia stars (of ${rows.length} rows).`);
   return stars;
 }
+
+/**
+ * DR3's Hipparcos cross-match is a fixed table of 99 525 rows, 97 751 of them with a usable
+ * parallax. Far fewer means the answer was an error page served with a 200, or was cut short,
+ * and either would pass for "Gaia does not know these stars" and put every one of them back at
+ * its Hipparcos distance.
+ */
+const MIN_USABLE_HIP_DISTANCES = 90_000;
+
+/**
+ * Gaia's distance for every Hipparcos star it has a usable parallax for, keyed by HIP number.
+ *
+ * Taken from the archive's own cross-match (`hipparcos2_best_neighbour`) rather than from
+ * matching positions here, since Gaia's team made that identification star by star with the
+ * proper motions and photometry in hand. It is deliberately not bounded by distance: the stars
+ * it exists for are the ones Hipparcos put inside the map and Gaia puts outside, which the main
+ * query above never fetches.
+ *
+ * Required rather than best effort. Without it every HYG star falls back to its Hipparcos
+ * distance, the 6 833 that Gaia puts past 250 pc move back inside, and the published map would
+ * flip between the two with the archive's availability.
+ */
+export async function fetchGaiaDistancesByHip(): Promise<Map<number, number>> {
+  const query = [
+    'select top 200000 b.original_ext_source_id as hip, g.parallax, g.parallax_over_error',
+    'from gaiadr3.hipparcos2_best_neighbour b join gaiadr3.gaia_source g on g.source_id = b.source_id'
+  ].join(' ');
+  const url = `${GAIA_TAP_URL}?REQUEST=doQuery&LANG=ADQL&FORMAT=csv&QUERY=${encodeURIComponent(query)}`;
+  console.log('Fetching Gaia DR3 distances for Hipparcos stars (archive cross-match)...');
+  const rows = parseCsvObjects(await fetchTextCached(url, `gaia-dr3-hip-${createHash('sha1').update(url).digest('hex').slice(0, 8)}.csv`));
+
+  const distances = new Map<number, number>();
+  for (const row of rows) {
+    const hip = parseOptionalNumber(row['hip']);
+    const parallaxMas = parseOptionalNumber(row['parallax']);
+    const overError = parseOptionalNumber(row['parallax_over_error']);
+    if (hip !== undefined && parallaxMas !== undefined && parallaxMas > 0 && overError !== undefined && overError > 1 / MAX_PARALLAX_ERROR_RATIO) {
+      distances.set(hip, 1000 / parallaxMas);
+    }
+  }
+  if (distances.size < MIN_USABLE_HIP_DISTANCES) {
+    throw new Error(
+      `the Hipparcos cross-match gave ${distances.size} usable distances (of ${rows.length} rows), not the ~97 751 it holds; ` +
+        'delete tools/etl/.cache/gaia-dr3-hip-*.csv once the archive answers properly'
+    );
+  }
+  console.log(`  ${distances.size} Hipparcos stars have a Gaia distance (of ${rows.length} cross-matched).`);
+  return distances;
+}

tools/etl/sources/registry.ts

@@ -14,7 +14,8 @@ export const STAR_SOURCES: readonly StarSource[] = [
     name: 'HYG database (Hipparcos, Yale Bright Star, Gliese)',
     role: 'positional',
     endpoint: 'https://raw.githubusercontent.com/astronexus/HYG-Database',
-    contributes: 'A complete, named, spectrally classified bright-star catalogue with parallaxes — 68388 stars within 250 pc.',
+    contributes:
+      'A complete, named, spectrally classified bright-star catalogue. Its Hipparcos parallaxes give way to Gaia’s wherever Gaia has a usable one, so its stars sit where the better measurement puts them.',
     unimplementedBecause: null
   },
   {