Give every planet with a distance a star, from the archive where the catalogue has none

After matching, 4 237 planets still had no star: their hosts are too faint for either Gaia
query (fainter than G 12 past 50 pc) or too far (past 250 pc), Kepler-186 at 177.6 pc among
them. fetchExoplanets now adds one star per such host from the archive's own figures, 3 277
of them, whenever the archive gives a position and a distance:

- position carried back from J2016, where the archive publishes it (741 of the 746 matched
  hosts moving over 100 mas/yr sit nearer their star carried back, a median 0.11" against 3.47"
  as published), and placed at sy_dist;
- magnitude in V (2 999 hosts), else Gaia G (13), the band the catalogue's Gaia stars are
  already in; 265 have neither, 128 KMT, 95 OGLE and 32 MOA microlensing hosts at a median
  6.2 kpc among them, and take the ETL's faint stand-in of 15;
- colour as B-V from the archive's B and V, else from st_teff through a new
  temperatureToColorIndex (Ballesteros 2012, inverted; the Sun's 5 772 K gives 0.65), else
  left to st_spectype;
- ids from 1 070 000 000, past Gaia's two ranges and under the 2^30 validateStars enforces;
  source "exoplanet-archive".

Hosts past 250 pc are included: 399 of the 3 277 are within 250 pc, 1 962 between 250 pc and
1 kpc, 916 beyond. The drawn budget still chooses what is drawn (70 000 of 455 608).

Planets with a star: 2 090 -> 6 327 of 6 354. The other 27 have no distance in either table
(Luhman 16 A, mu2 Sco, PSR B1620-26 among them). Systems in the Solar Neighbourhood readout:
1 450 -> 4 736. validateExoplanets now refuses a catalogue where fewer than 99.5 % of planets
have a host (measured 99.58 %); dropping the added stars fails it at 2 090, and dropping the
composite fill at 6 227. No added star sits within an arcsecond of a catalogue star
(validateMerge's twins stay at 23); 5 of the 399 within 250 pc have one within a minute of arc,
VHS J125601.92-125723.9 (archive 12.7 pc) 5.8" from a Gaia entry at 21.2 pc the likeliest
duplicate.

In the app, searching TRAPPIST-1 or Kepler-186 and picking the star enters a system with its
seven and five planets drawn. gzip -9: stars.bin 5 030 741 -> 5 067 864 B, stars-meta.bin 2 784 614 -> 2 804 064,
stars-index.json 4 003 584 -> 4 015 882, exoplanets.json 342 467 -> 446 532 (host parameters,
both commits).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

src/app/shared/astro/spectral.spec.ts

@@ -1,6 +1,6 @@
 import { describe, expect, it } from 'vitest';
 
-import { parseSpectralClass, SPECTRAL_CLASSES, spectralTypeToColorIndex } from './spectral';
+import { parseSpectralClass, SPECTRAL_CLASSES, spectralTypeToColorIndex, temperatureToColorIndex } from './spectral';
 
 describe('parseSpectralClass', () => {
   it('reads a clean class and subclass', () => {
@@ -83,3 +83,20 @@ describe('spectralTypeToColorIndex', () => {
     expect(spectralTypeToColorIndex('')).toBeNull();
   });
 });
+
+describe('temperatureToColorIndex', () => {
+  it("puts the Sun's temperature at its own B-V and a cool dwarf redder", () => {
+    expect(temperatureToColorIndex(5772)).toBeCloseTo(0.65, 2);
+    expect(temperatureToColorIndex(3400)).toBeCloseTo(1.79, 2);
+  });
+
+  it('stays inside the range the spectral classes span', () => {
+    expect(temperatureToColorIndex(2566)).toBe(2);
+    expect(temperatureToColorIndex(50000)).toBe(-0.33);
+  });
+
+  it('has no answer for a temperature that is not one', () => {
+    expect(temperatureToColorIndex(0)).toBeNull();
+    expect(temperatureToColorIndex(Number.NaN)).toBeNull();
+  });
+});

src/app/shared/astro/spectral.ts

@@ -87,3 +87,23 @@ export function spectralTypeToColorIndex(spectralType: string | null | undefined
 
   return from + (to - from) * (subclass / 10);
 }
+
+/**
+ * B-V colour index for an effective temperature, for stars the Exoplanet Archive gives a
+ * temperature but no B magnitude. Inverts Ballesteros (2012), T = 4600 K · (1 / (0.92 (B-V) +
+ * 1.7) + 1 / (0.92 (B-V) + 0.62)), a blackbody fit good to a few per cent from A to early M: it
+ * puts the Sun's 5 772 K at 0.65, which is the Sun's own. Clamped to the range the class anchors
+ * above span, because the fit runs on past it — TRAPPIST-1's 2 566 K would come out at 2.7.
+ */
+export function temperatureToColorIndex(temperatureK: number): number | null {
+  if (!Number.isFinite(temperatureK) || temperatureK <= 0) {
+    return null;
+  }
+  // With x = 0.92 (B-V) and k = T / 4600 the fit is k x² + (2.32 k - 2) x + (1.054 k - 2.32) = 0,
+  // whose larger root is the physical one.
+  const k = temperatureK / 4600;
+  const b = 2.32 * k - 2;
+  const c = 1.054 * k - 2.32;
+  const x = (-b + Math.sqrt(b * b - 4 * k * c)) / (2 * k);
+  return Math.min(BEYOND_M, Math.max(COLOR_INDEX_ANCHORS.O, x / 0.92));
+}

src/app/shared/models/exoplanet.model.ts

@@ -7,7 +7,9 @@ import { OrbitalElements } from './body.model';
 export interface ExoplanetRecord {
   id: string;
   /**
-   * The host's star-catalogue id, the HYG or Gaia star it was matched to; null when none is.
+   * The host's star-catalogue id: a HYG or Gaia star it was matched to, or else a star the ETL
+   * added from the archive's own figures. Null only when the archive gives no position and
+   * distance to place one with.
    */
   hostStarId: number | null;
   hostStarName: string;

tools/etl/build.ts

@@ -155,6 +155,14 @@ function validateBodies(bodies: BodyRecord[]): void {
   assertCondition(planetCount === 8, `Expected 8 planets, found ${planetCount}.`);
 }
 
+/**
+ * The share of planets that must have a star on the map: 6 327 of 6 354 did when the ETL began
+ * adding the hosts the catalogue lacks from the archive's own figures, up from 2 071. The other
+ * 27 have no distance in either archive table, so nothing can place them; the floor leaves room
+ * for a few more of those, not for the matching or the additions to stop working.
+ */
+const MIN_HOSTED_SHARE = 0.995;
+
 function validateExoplanets(exoplanets: ExoplanetRecord[], stars: StarRecord[]): void {
   assertCondition(exoplanets.length > 0, 'No exoplanets were produced.');
   const starsById = new Map(stars.map((star) => [star.id, star]));
@@ -189,6 +197,10 @@ function validateExoplanets(exoplanets: ExoplanetRecord[], stars: StarRecord[]):
   }
 
   console.log(`  ${crossReferenced}/${exoplanets.length} exoplanets have a host star on the map.`);
+  assertCondition(
+    crossReferenced >= exoplanets.length * MIN_HOSTED_SHARE,
+    `Only ${crossReferenced} of ${exoplanets.length} exoplanets have a host star (at least ${MIN_HOSTED_SHARE * 100} % expected) — hosts are no longer being matched or added.`
+  );
 
   // How many can be propagated at their real rate rather than as if the host were the Sun.
   const withPeriod = exoplanets.filter((exoplanet) => exoplanet.periodDays !== undefined).length;
@@ -266,7 +278,7 @@ async function build(): Promise<void> {
   console.log();
   const bodies = await fetchSolarSystem();
   console.log();
-  // Names the Gaia designations that host planets, so it is this list, not the one above, that is published.
+  // Adds the hosts the catalogue lacks, so it is this list, not the one above, that is published.
   const { exoplanets, stars } = await fetchExoplanets(catalogueStars);
   console.log();
   const deepSky = await fetchDeepSky();

tools/etl/fetchExoplanets.ts

@@ -1,7 +1,9 @@
 import { createHash } from 'node:crypto';
 import { writeFileSync } from 'node:fs';
 
+import { propagateProperMotion, raDegDecDistanceToXyz } from '../../src/app/shared/astro/coordinates';
 import { buildStarNameIndex, resolveHostStarId } from '../../src/app/shared/astro/host-star-matching';
+import { temperatureToColorIndex } from '../../src/app/shared/astro/spectral';
 import { ExoplanetRecord } from '../../src/app/shared/models/exoplanet.model';
 import { isDesignation } from '../../src/app/shared/models/star-catalog';
 import { StarRecord } from '../../src/app/shared/models/star.model';
@@ -76,10 +78,27 @@ const COMPOSITE_COLUMNS = [
 const COMPOSITE_URL = `${TAP_BASE_URL}?query=select+${COMPOSITE_COLUMNS}+from+pscomppars+order+by+pl_name&format=csv`;
 const COMPOSITE_CACHE_FILE = `exoplanet-archive-pscomppars-${createHash('sha1').update(COMPOSITE_URL).digest('hex').slice(0, 8)}.csv`;
 
+/**
+ * Where the ids of the stars only the archive places begin: past Gaia's two ranges, and under
+ * the 2^30 `validateStars` holds every id to.
+ */
+const ARCHIVE_ID_BASE = 1_070_000_000;
+const ARCHIVE_SOURCE = 'exoplanet-archive';
+/**
+ * The archive's positions are at Gaia's epoch, J2016, not the catalogue's J2000: of the 746
+ * matched hosts moving over 100 mas a year, 741 sit nearer their star once carried back sixteen
+ * years (a median 0.11″ from it, against 3.47″ as published). The matcher tries both epochs; a
+ * star placed from the archive has to pick one.
+ */
+const ARCHIVE_TO_CATALOGUE_YEARS = 2000 - 2016;
+/** As in `fetchStars`: faint, for a host the archive gives neither a V nor a G magnitude. */
+const UNKNOWN_MAGNITUDE = 15;
+
 /**
  * Downloads confirmed exoplanets from the NASA Exoplanet Archive (`Planetary Systems` TAP
- * table), cross-references each host star to the star catalogue, and writes `exoplanets.json`
- * together with the star assets, whose Gaia designations take their host's name. Returns both.
+ * table), cross-references each host star to the star catalogue, adds a star from the archive's
+ * own figures for each host the catalogue lacks but the archive places, and writes
+ * `exoplanets.json` together with the star assets, which those additions change. Returns both.
  */
 export async function fetchExoplanets(stars?: StarRecord[]): Promise<{ exoplanets: ExoplanetRecord[]; stars: StarRecord[] }> {
   console.log('Fetching confirmed exoplanets from the NASA Exoplanet Archive...');
@@ -95,6 +114,8 @@ export async function fetchExoplanets(stars?: StarRecord[]): Promise<{ exoplanet
   // Catalogue stars known only by their Gaia designation, which take the archive's host name —
   // the only way "TRAPPIST-1" or "Teegarden's Star" can be found by search.
   const renamed = new Map<number, string>();
+  const archiveStars = new Map<string, StarRecord>();
+  const bands = { V: 0, G: 0, none: 0 };
   const exoplanets: ExoplanetRecord[] = rows.map((row, index) => {
     const compositeRow = composite.get(row['pl_name']);
     // `parseOptionalNumber`, not `Number`: a blank cell would otherwise become 0, which is a
@@ -106,7 +127,7 @@ export async function fetchExoplanets(stars?: StarRecord[]): Promise<{ exoplanet
     const pmRaMasPerYear = host('sy_pmra');
     const pmDecMasPerYear = host('sy_pmdec');
 
-    const hostStarId = resolveHostStarId(
+    let hostStarId = resolveHostStarId(
       { hostname: row['hostname'], raDeg, decDeg, distancePc, pmRaMasPerYear, pmDecMasPerYear, parallaxMas: host('sy_plx') },
       knownStars,
       nameIndex
@@ -117,6 +138,32 @@ export async function fetchExoplanets(stars?: StarRecord[]): Promise<{ exoplanet
       if (star?.source === 'gaia' && isDesignation(star) && !renamed.has(hostStarId)) {
         renamed.set(hostStarId, row['hostname']);
       }
+    } else if ([raDeg, decDeg, distancePc].every(Number.isFinite) && distancePc > 0) {
+      let archiveStar = archiveStars.get(row['hostname']);
+      if (!archiveStar) {
+        // Carried back from the archive's epoch like any Gaia row, and placed at `sy_dist`.
+        const j2000 = propagateProperMotion(raDeg, decDeg, pmRaMasPerYear ?? 0, pmDecMasPerYear ?? 0, ARCHIVE_TO_CATALOGUE_YEARS);
+        // V where the archive has it, as HYG's stars are; else Gaia's G, the band the catalogue's
+        // Gaia stars are already in. The assets carry no band, so it is counted below instead.
+        const v = host('sy_vmag');
+        const g = host('sy_gaiamag');
+        const b = host('sy_bmag');
+        const temperatureK = host('st_teff');
+        bands[v !== undefined ? 'V' : g !== undefined ? 'G' : 'none']++;
+        archiveStar = {
+          id: ARCHIVE_ID_BASE + archiveStars.size,
+          name: row['hostname'],
+          ...raDegDecDistanceToXyz(j2000.raDeg, j2000.decDeg, distancePc),
+          magnitude: v ?? g ?? UNKNOWN_MAGNITUDE,
+          spectralType: compositeRow?.['st_spectype'] || 'Unknown',
+          // B-V where the archive has both magnitudes, else the effective temperature's; with
+          // neither, null leaves the colour to the spectral type, as for any other star.
+          colorIndex: b !== undefined && v !== undefined ? b - v : temperatureK !== undefined ? temperatureToColorIndex(temperatureK) : null,
+          source: ARCHIVE_SOURCE
+        };
+        archiveStars.set(row['hostname'], archiveStar);
+      }
+      hostStarId = archiveStar.id;
     }
 
     return {
@@ -152,12 +199,19 @@ export async function fetchExoplanets(stars?: StarRecord[]): Promise<{ exoplanet
     };
   });
 
-  const allStars = knownStars.map((star) => (renamed.has(star.id) ? { ...star, name: renamed.get(star.id)! } : star));
+  // Appended after the catalogue, whose ids all sit below ARCHIVE_ID_BASE, so the list stays in
+  // the id order `fetchStars` sorted it into.
+  const added = [...archiveStars.values()];
+  const allStars = [...knownStars.map((star) => (renamed.has(star.id) ? { ...star, name: renamed.get(star.id)! } : star)), ...added];
   writeStarAssets(allStars);
 
   ensureDataDir();
   writeFileSync(dataPath('exoplanets.json'), JSON.stringify(exoplanets));
-  console.log(`  wrote ${exoplanets.length} exoplanets (${matched} on a catalogue star, ${renamed.size} Gaia designations named after their host).`);
+  console.log(
+    `  wrote ${exoplanets.length} exoplanets: ${matched} on a catalogue star (${renamed.size} Gaia designations named after their host), ` +
+      `${exoplanets.filter((exoplanet) => exoplanet.hostStarId !== null).length - matched} on ${added.length} stars added from the archive ` +
+      `(${added.filter((star) => Math.hypot(star.x, star.y, star.z) <= 250).length} within 250 pc; magnitude in V for ${bands.V}, in G for ${bands.G}, none for ${bands.none}).`
+  );
   return { exoplanets, stars: allStars };
 }