Say which band each star was measured in, whose catalogue it is, and how sure its distance is

The readout named Hipparcos, Yale and Gliese while 378 775 of the 455 608 stars are described by Gaia DR3, printed one "Magnitude" for G and V alike, and gave every distance to the parsec. The band cannot be read off the star's source, which records whose position it has: 62 002 stars Gaia places keep HYG's V and B-V, and the 575 hosts renamed after their planets are Gaia's, in G. stars-meta.bin gains two byte columns, 14 to 16 bytes a star (6 378 512 to 7 289 728 bytes; gzip -9 2 804 049 to 3 110 991). One holds the magnitude's band (V 76 555 stars, G 378 744, none 309, whose magnitude is a stand-in), which colour the colour index is (B-V 75 203, BP-RP 376 703), and whether the distance is Gaia's parallax. The other holds the distance's relative error as its square root in 255ths: a step is 0.08 % of distance at 1 %, 0.35 % at 20 %, and 100 % is the top. encode, decode, BYTES_PER_STAR_META and the build.ts round trip cover both; no workflow reads the format. Where the errors come from: - Gaia rows keep the parallax_error their query already fetched: median 0.3 %, 90th percentile 1.2 %, at most 20 %, the query's own cut. - A HYG star at Gaia's distance takes the cross-match's parallax_over_error, and a star merged into a Gaia entry keeps that entry's error with its position. - The 3 067 Hipparcos stars that keep their Hipparcos distance, Rigel, Deneb and Alnilam among them, take e_plx from van Leeuwen's 2007 reduction: a new cached query of public.hipparcos_newreduction on the ESA archive, whose 117 955 rows HYG's distances invert. - The archive's stars take sy_disterr1/2 from pscomppars, in a query and cache file of their own so the composite rows already cached were not refetched. - 439 distances have no published error: 357 Gliese rows and 82 archive hosts. Of the errors, 392 786 are 1 % or less and are not printed; 61 156 print as "117 ± 12 pc" to the distance's own digits; 1 196 between 20 and 100 %, and 30 past it, print as the range the parallax gives, since a symmetric error in parallax is a lopsided one in distance. The star card (measured on the dev server) now reads, for example: - Rigel: 265 ± 23 pc, V 0.18, B-V -0.03, source HYG. - Deneb: 433 ± 60 pc. - Alnilam: "476 pc to 833 pc" (Hipparcos 1.65 ± 0.45 mas). - Gaia DR3 5612323414549657984: 111 ± 2 pc, G 4.63, BP-RP -0.15, source Gaia DR3. - Proxima Centauri: 1.30 pc, V 11.01, source "HYG, Gaia DR3 distance". - TRAPPIST-1: G 15.62, BP-RP 4.90, source Gaia DR3. - Kepler-186: V 15.14, source NASA Exoplanet Archive. The neighbourhood's subtitle reads "Gaia DR3 378,775 · HYG 73,556 · NASA Exoplanet Archive 3,277", counted by the catalogue describing each star. build.ts validateStars now fails a catalogue with more than 1 000 stars without a band (309 today) or without a distance error (439). Dropping G from the Gaia rows gave 379 040 without a band, and dropping their parallax_error gave 441 216 without an error; both runs failed. Decoding the catalogue in Node took a median 29 ms before and 24 ms after (nine runs each, within noise). In the app, five cold boots gave a 654-786 ms long task after the data landed and the HUD at 1.83-2.07 s. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-24 23:26:27 +02:00
parent 4c8e4a02f3
commit a81dd491ad
15 changed files with 450 additions and 45 deletions
@@ -20,6 +20,15 @@ function assertCondition(condition: boolean, message: string): void {
  }
 }

+/**
+ * Stars whose magnitude is a stand-in, and distances published without an error. Measured 309 and
+ * 439: the 44 Gaia sources with no G and the 265 archive hosts with neither V nor G; the 357 Gliese
+ * distances (no Hipparcos parallax behind them) and 82 archive hosts the archive gives no error
+ * for. Losing either field loses it for hundreds of thousands of stars.
+ */
+const MAX_STARS_WITHOUT_BAND = 1_000;
+const MAX_STARS_WITHOUT_DISTANCE_ERROR = 1_000;
+
 function validateStars(stars: StarRecord[]): void {
  assertCondition(stars.length > 0, 'No stars were produced.');

@@ -49,7 +58,28 @@ function validateStars(stars: StarRecord[]): void {
    assertCondition(decoded[i].id === stars[i].id && decoded[i].name === stars[i].name, `Star catalogue round-trip altered record ${i}.`);
    assertCondition(decoded[i].spectralType === stars[i].spectralType, `Star catalogue round-trip lost the spectral type of star ${stars[i].id}.`);
    assertCondition(decoded[i].colorIndex === null === (stars[i].colorIndex === null), `Star catalogue round-trip changed whether star ${stars[i].id} has a colour index.`);
+    assertCondition(
+      decoded[i].magnitudeBand === stars[i].magnitudeBand && decoded[i].colorSystem === stars[i].colorSystem && decoded[i].distanceFromGaia === !!stars[i].distanceFromGaia,
+      `Star catalogue round-trip changed the photometry of star ${stars[i].id}.`
+    );
+    // Stored as its square root in 255ths, up to 100 %; see `star-catalog.ts`.
+    const error = stars[i].distanceError;
+    assertCondition(
+      error === undefined ? decoded[i].distanceError === undefined : Math.abs(Math.sqrt(decoded[i].distanceError!) - Math.sqrt(Math.min(1, error))) <= 0.5 / 255 + 1e-9,
+      `Star catalogue round-trip changed the distance error of star ${stars[i].id}.`
+    );
  }
+
+  // What each star says it was measured in. A band or an error dropped on the way still encodes,
+  // decodes and draws; it shows only as a card reading "Not measured" for a star that was.
+  const withoutBand = stars.filter((star) => star.magnitudeBand === undefined).length;
+  assertCondition(withoutBand <= MAX_STARS_WITHOUT_BAND, `${withoutBand} stars have no magnitude band (at most ${MAX_STARS_WITHOUT_BAND} expected) — the band is being lost.`);
+  const withoutError = stars.filter((star) => star.id !== SUN_STAR_ID && star.distanceError === undefined).length;
+  assertCondition(
+    withoutError <= MAX_STARS_WITHOUT_DISTANCE_ERROR,
+    `${withoutError} stars have no distance error (at most ${MAX_STARS_WITHOUT_DISTANCE_ERROR} expected) — the parallax errors are being lost.`
+  );
+  console.log(`  ${withoutBand} stars with a stand-in magnitude; ${withoutError} distances without a published error.`);
 }

 /**
@@ -77,6 +77,13 @@ const COMPOSITE_COLUMNS = [
 // pl_name is unique here too, one row per planet.
 const COMPOSITE_URL = `${TAP_BASE_URL}?query=select+${COMPOSITE_COLUMNS}+from+pscomppars+order+by+pl_name&format=csv`;
 const COMPOSITE_CACHE_FILE = `exoplanet-archive-pscomppars-${createHash('sha1').update(COMPOSITE_URL).digest('hex').slice(0, 8)}.csv`;
+/**
+ * The same table's distance errors, for the stars placed from the archive. A query of its own,
+ * cached apart, so that asking for them did not refetch the columns above: the archive changes
+ * daily, and a new answer would have moved every figure the catalogue was checked against.
+ */
+const DISTANCE_ERRORS_URL = `${TAP_BASE_URL}?query=select+pl_name,sy_disterr1,sy_disterr2+from+pscomppars+order+by+pl_name&format=csv`;
+const DISTANCE_ERRORS_CACHE_FILE = `exoplanet-archive-disterr-${createHash('sha1').update(DISTANCE_ERRORS_URL).digest('hex').slice(0, 8)}.csv`;

 /**
 * Where the ids of the stars only the archive places begin: past Gaia's two ranges, and under
@@ -109,6 +116,7 @@ export async function fetchExoplanets(stars?: StarRecord[]): Promise<{ exoplanet
  const csv = await fetchTextCached(TAP_URL, CACHE_FILE);
  const rows = parseCsvObjects(csv);
  const composite = new Map(parseCsvObjects(await fetchTextCached(COMPOSITE_URL, COMPOSITE_CACHE_FILE)).map((row) => [row['pl_name'], row]));
+  const distanceErrors = new Map(parseCsvObjects(await fetchTextCached(DISTANCE_ERRORS_URL, DISTANCE_ERRORS_CACHE_FILE)).map((row) => [row['pl_name'], row]));

  let matched = 0;
  // Catalogue stars known only by their Gaia designation, which take the archive's host name —
@@ -144,21 +152,29 @@ export async function fetchExoplanets(stars?: StarRecord[]): Promise<{ exoplanet
        // Carried back from the archive's epoch like any Gaia row, and placed at `sy_dist`.
        const j2000 = propagateProperMotion(raDeg, decDeg, pmRaMasPerYear ?? 0, pmDecMasPerYear ?? 0, ARCHIVE_TO_CATALOGUE_YEARS);
        // V where the archive has it, as HYG's stars are; else Gaia's G, the band the catalogue's
-        // Gaia stars are already in. The assets carry no band, so it is counted below instead.
+        // Gaia stars are already in.
        const v = host('sy_vmag');
        const g = host('sy_gaiamag');
        const b = host('sy_bmag');
        const temperatureK = host('st_teff');
-        bands[v !== undefined ? 'V' : g !== undefined ? 'G' : 'none']++;
+        const band = v !== undefined ? 'V' : g !== undefined ? 'G' : undefined;
+        bands[band ?? 'none']++;
+        // B-V where the archive has both magnitudes, else the effective temperature's; with
+        // neither, null leaves the colour to the spectral type, as for any other star.
+        const colorIndex = b !== undefined && v !== undefined ? b - v : temperatureK !== undefined ? temperatureToColorIndex(temperatureK) : null;
+        // The archive gives the distance's error as two one-sided ones; their mean, relative.
+        const errors = distanceErrors.get(row['pl_name']);
+        const [above, below] = [parseOptionalNumber(errors?.['sy_disterr1']), parseOptionalNumber(errors?.['sy_disterr2'])];
        archiveStar = {
          id: ARCHIVE_ID_BASE + archiveStars.size,
          name: row['hostname'],
          ...raDegDecDistanceToXyz(j2000.raDeg, j2000.decDeg, distancePc),
          magnitude: v ?? g ?? UNKNOWN_MAGNITUDE,
+          ...(band === undefined ? {} : { magnitudeBand: band }),
          spectralType: compositeRow?.['st_spectype'] || 'Unknown',
-          // B-V where the archive has both magnitudes, else the effective temperature's; with
-          // neither, null leaves the colour to the spectral type, as for any other star.
-          colorIndex: b !== undefined && v !== undefined ? b - v : temperatureK !== undefined ? temperatureToColorIndex(temperatureK) : null,
+          colorIndex,
+          ...(colorIndex === null ? {} : { colorSystem: 'B-V' as const }),
+          ...(above === undefined || below === undefined ? {} : { distanceError: (Math.abs(above) + Math.abs(below)) / 2 / distancePc }),
          source: ARCHIVE_SOURCE
        };
        archiveStars.set(row['hostname'], archiveStar);
@@ -3,7 +3,7 @@ import { writeFileSync } from 'node:fs';
 import { mergeStarCatalogues, placementDistancePc } from '../../src/app/shared/astro/star-merge';
 import { encodeStarCatalog } from '../../src/app/shared/models/star-catalog';
 import { StarRecord, SUN_STAR_ID } from '../../src/app/shared/models/star.model';
-import { fetchGaiaDistancesByHip, GaiaAnswerError } from './sources/gaia';
+import { fetchGaiaDistancesByHip, fetchHipparcosParallaxErrors, GaiaAnswerError } from './sources/gaia';
 import { positionalSources } from './sources/registry';
 import { PARALLAX_PRECISION_MAS } from './sources/star-sources';
 import { parseCsvObjects, parseOptionalNumber } from './lib/csv';
@@ -71,7 +71,8 @@ export async function fetchStars(): Promise<StarRecord[]> {
  const csv = await fetchTextCached(HYG_CSV_URL, 'hygdata_v41.csv');
  const rows = parseCsvObjects(csv);
  // Not skipped when unreachable, unlike the positional sources below; see its own comment.
-  const gaiaPcByHip = await fetchGaiaDistancesByHip();
+  const gaiaByHip = await fetchGaiaDistancesByHip();
+  const hipparcosErrors = await fetchHipparcosParallaxErrors();

  const stars: StarRecord[] = [];
  let atGaiaDistance = 0;
@@ -81,14 +82,16 @@ export async function fetchStars(): Promise<StarRecord[]> {
    const id = Number(row['id']);

    if (id === SUN_STAR_ID) {
-      stars.push({ id, name: 'Sol', x: 0, y: 0, z: 0, magnitude: parseOptionalNumber(row['mag']) ?? UNKNOWN_MAGNITUDE, spectralType: row['spect'] || 'G2V', colorIndex: parseOptionalNumber(row['ci']) ?? null });
+      stars.push({ id, name: 'Sol', x: 0, y: 0, z: 0, magnitude: parseOptionalNumber(row['mag']) ?? UNKNOWN_MAGNITUDE, magnitudeBand: 'V', spectralType: row['spect'] || 'G2V', colorIndex: parseOptionalNumber(row['ci']) ?? null, colorSystem: 'B-V' });
      continue;
    }

    const hygPc = Number(row['dist']);
    const hipparcosPc = Number.isFinite(hygPc) && hygPc > 0 && hygPc < HYG_UNKNOWN_DISTANCE_PC ? hygPc : undefined;
-    const gaiaPc = row['hip'] ? gaiaPcByHip.get(Number(row['hip'])) : undefined;
-    const magnitude = parseOptionalNumber(row['mag']) ?? UNKNOWN_MAGNITUDE;
+    const gaia = row['hip'] ? gaiaByHip.get(Number(row['hip'])) : undefined;
+    const gaiaPc = gaia?.distancePc;
+    const magnitudeV = parseOptionalNumber(row['mag']);
+    const magnitude = magnitudeV ?? UNKNOWN_MAGNITUDE;
    const distancePc = placementDistancePc(hipparcosPc, gaiaPc, magnitude, DISTANCE_CUTOFF_PC);
    if (distancePc === null) {
      continue;
@@ -119,6 +122,11 @@ export async function fetchStars(): Promise<StarRecord[]> {
      pastCutoff++;
    }

+    // Gaia's error with Gaia's distance, Hipparcos's with its own; a Gliese row, with neither, has
+    // no published error, and its distance is as often photometric as measured.
+    const distanceError = gaia?.relativeError ?? (row['hip'] ? hipparcosErrors.get(Number(row['hip'])) : undefined);
+    const colorIndex = parseOptionalNumber(row['ci']);
+
    stars.push({
      id,
      name: resolveName(row),
@@ -126,8 +134,12 @@ export async function fetchStars(): Promise<StarRecord[]> {
      y: y * scale,
      z: z * scale,
      magnitude,
+      ...(magnitudeV === undefined ? {} : { magnitudeBand: 'V' as const }),
      spectralType: row['spect'] || 'Unknown',
-      colorIndex: parseOptionalNumber(row['ci']) ?? null,
+      colorIndex: colorIndex ?? null,
+      ...(colorIndex === undefined ? {} : { colorSystem: 'B-V' as const }),
+      ...(distanceError === undefined ? {} : { distanceError }),
+      distanceFromGaia: gaia !== undefined,
      // Only for the Gliese-only rows, whose positions are what the merge needs the motion to see
      // past. A Hipparcos position is good to under an arcsecond; given its motion too, 15 stars
      // took their co-moving companion's Gaia entry, and the companion was kept twice.
@@ -133,9 +133,9 @@ const MIN_ROW_SHARE = 0.95;
 export class GaiaAnswerError extends Error {}

 /**
- * Gaia publishes no spectral classifications, but `bp_rp` is a colour index on the same footing
- * as HYG's `ci` — so the app's existing colour and spectral-class handling works unchanged, and
- * the spectral type is left as unknown rather than invented from the colour.
+ * Gaia publishes no spectral classifications. Its `bp_rp` is a colour index, though not HYG's
+ * B−V — `colorSystem` says which — and the spectral type is left as unknown rather than
+ * invented from it.
 */
 const UNKNOWN_SPECTRAL_TYPE = 'Unknown';

@@ -209,10 +209,13 @@ function rowsToStars(rows: readonly Record<string, string>[], idBase: number): S
      return;
    }

+    const parallaxErrorMas = parseOptionalNumber(row['parallax_error']);
    const pmRaMasYr = parseOptionalNumber(row['pmra']);
    const pmDecMasYr = parseOptionalNumber(row['pmdec']);
    const j2000 = propagateProperMotion(raDeg, decDeg, pmRaMasYr ?? 0, pmDecMasYr ?? 0, CATALOGUE_EPOCH - GAIA_DR3_EPOCH);
    const { x, y, z } = raDegDecDistanceToXyz(j2000.raDeg, j2000.decDeg, distancePc);
+    const magnitudeG = parseOptionalNumber(row['phot_g_mean_mag']);
+    const colorIndex = parseOptionalNumber(row['bp_rp']);
    stars.push({
      id: idBase + index,
      name: `Gaia DR3 ${row['source_id']}`,
@@ -222,9 +225,13 @@ function rowsToStars(rows: readonly Record<string, string>[], idBase: number): S
      // Only the nearby query has sources without a G, 45 of them, and they are given its limit.
      // That errs bright: 26 of the 31 that 2MASS measured are at J 12.6–15.7, fainter still in G
      // for stars this red, and 5 at J 7–8.
-      magnitude: parseOptionalNumber(row['phot_g_mean_mag']) ?? MAGNITUDE_LIMIT,
+      magnitude: magnitudeG ?? MAGNITUDE_LIMIT,
+      ...(magnitudeG === undefined ? {} : { magnitudeBand: 'G' as const }),
      spectralType: UNKNOWN_SPECTRAL_TYPE,
-      colorIndex: parseOptionalNumber(row['bp_rp']) ?? null,
+      colorIndex: colorIndex ?? null,
+      ...(colorIndex === undefined ? {} : { colorSystem: 'BP-RP' as const }),
+      ...(parallaxErrorMas === undefined ? {} : { distanceError: parallaxErrorMas / parallaxMas }),
+      distanceFromGaia: true,
      source: 'gaia',
      pmRaMasYr,
      pmDecMasYr
@@ -243,7 +250,8 @@ function rowsToStars(rows: readonly Record<string, string>[], idBase: number): S
 const MIN_USABLE_HIP_DISTANCES = 90_000;

 /**
- * Gaia's distance for every Hipparcos star it has a usable parallax for, keyed by HIP number.
+ * Gaia's distance for every Hipparcos star it has a usable parallax for, and that distance's
+ * relative error, keyed by HIP number.
 *
 * Taken from the archive's own cross-match (`hipparcos2_best_neighbour`) rather than from
 * matching positions here, since Gaia's team made that identification star by star with the
@@ -255,7 +263,7 @@ const MIN_USABLE_HIP_DISTANCES = 90_000;
 * distance, the 6 833 that Gaia puts past 250 pc move back inside, and the published map would
 * flip between the two with the archive's availability.
 */
-export async function fetchGaiaDistancesByHip(): Promise<Map<number, number>> {
+export async function fetchGaiaDistancesByHip(): Promise<Map<number, { distancePc: number; relativeError: number }>> {
  const query = [
    'select top 200000 b.original_ext_source_id as hip, g.parallax, g.parallax_over_error',
    'from gaiadr3.hipparcos2_best_neighbour b join gaiadr3.gaia_source g on g.source_id = b.source_id'
@@ -264,13 +272,13 @@ export async function fetchGaiaDistancesByHip(): Promise<Map<number, number>> {
  console.log('Fetching Gaia DR3 distances for Hipparcos stars (archive cross-match)...');
  const rows = parseCsvObjects(await fetchTextCached(url, `gaia-dr3-hip-${createHash('sha1').update(url).digest('hex').slice(0, 8)}.csv`));

-  const distances = new Map<number, number>();
+  const distances = new Map<number, { distancePc: number; relativeError: number }>();
  for (const row of rows) {
    const hip = parseOptionalNumber(row['hip']);
    const parallaxMas = parseOptionalNumber(row['parallax']);
    const overError = parseOptionalNumber(row['parallax_over_error']);
    if (hip !== undefined && parallaxMas !== undefined && parallaxMas > 0 && overError !== undefined && overError > 1 / MAX_PARALLAX_ERROR_RATIO) {
-      distances.set(hip, 1000 / parallaxMas);
+      distances.set(hip, { distancePc: 1000 / parallaxMas, relativeError: 1 / overError });
    }
  }
  if (distances.size < MIN_USABLE_HIP_DISTANCES) {
@@ -282,3 +290,33 @@ export async function fetchGaiaDistancesByHip(): Promise<Map<number, number>> {
  console.log(`  ${distances.size} Hipparcos stars have a Gaia distance (of ${rows.length} cross-matched).`);
  return distances;
 }
+
+/** The new Hipparcos reduction holds 117 955 stars, and every one has a parallax error. */
+const MIN_HIPPARCOS_ERRORS = 110_000;
+
+/**
+ * The relative error of every Hipparcos parallax, keyed by HIP number, for the stars that keep
+ * their Hipparcos distance: 3 067 of them, Rigel and Deneb among them, where Gaia saturates.
+ *
+ * From van Leeuwen's 2007 reduction, which the ESA archive hosts beside Gaia and HYG's distances
+ * are the inverse of. HYG publishes the distance and not its error.
+ */
+export async function fetchHipparcosParallaxErrors(): Promise<Map<number, number>> {
+  const url = `${GAIA_TAP_URL}?REQUEST=doQuery&LANG=ADQL&FORMAT=csv&QUERY=${encodeURIComponent('select top 200000 hip, plx, e_plx from public.hipparcos_newreduction order by hip')}`;
+  console.log('Fetching Hipparcos parallax errors (new reduction)...');
+  const rows = parseCsvObjects(await fetchTextCached(url, `hipparcos-errors-${createHash('sha1').update(url).digest('hex').slice(0, 8)}.csv`));
+
+  const errors = new Map<number, number>();
+  for (const row of rows) {
+    const hip = parseOptionalNumber(row['hip']);
+    const parallaxMas = parseOptionalNumber(row['plx']);
+    const errorMas = parseOptionalNumber(row['e_plx']);
+    if (hip !== undefined && parallaxMas !== undefined && parallaxMas > 0 && errorMas !== undefined) {
+      errors.set(hip, errorMas / parallaxMas);
+    }
+  }
+  if (errors.size < MIN_HIPPARCOS_ERRORS) {
+    throw new Error(`the Hipparcos reduction gave ${errors.size} parallax errors (of ${rows.length} rows), not the ~117 955 it holds; delete tools/etl/.cache/hipparcos-errors-*.csv once the archive answers properly`);
+  }
+  return errors;
+}