Draw the whole catalogue, and build the aggregation the rest would need

Two things, one verified and one that cannot be. The render budget is now the whole catalogue: 68388 stars, one instanced draw call, which is what a GPU should be asked to do. The budget itself stays, because the catalogue is meant to grow past what any machine should draw at once — Gaia alone could contribute a million — and at that point the selection is what keeps the field legible rather than a grey wash. A `?stars=` override handles the machines that cannot, including the software rasterizer the end-to-end suite runs against, whose frame rate is two orders of magnitude below a real GPU's and which was measuring the rasterizer rather than the app. The aggregation is the second thing, and none of it has run. Every ESA, NOIRLab, SDSS and Euclid endpoint is unreachable from here — only GitHub raw is, which is why HYG and OpenNGC are the current sources. So this is infrastructure and a Gaia query written against the published DR3 schema, not data. What the framework encodes is that these surveys are not interchangeable. The distinction is not size but whether a catalogue knows how far away its objects are, because a 3D map cannot place a star it only has a direction for. Gaia is the only one of the five that can add stars here, because it is the only one that measures parallaxes. DECaPS2 has fifty times Gaia's object count and photometry alone — not one of its 3.32 billion objects can be placed in depth. Euclid's bulge is 8 kpc away, where a parallax is microarcseconds; its contribution would be imagery. SDSS-V and SAGA are keyed to stars something else already places, so they enrich rather than extend. Those roles are recorded as data the ETL prints, not as prose that can drift. Overlapping catalogues are reconciled on direction rather than on 3D proximity, which is the one non-obvious part. Two surveys agree on a star's direction to within an arcsecond and disagree on its distance by tens of per cent, so a star at 200 pc is 50 pc from itself between catalogues while being unmistakably the same object. Matching in 3D would need a tolerance so loose it swallowed real neighbours. The better parallax wins where both reach; where only one does, the star stays. Names become dense-with-holes with a source dictionary, because a survey catalogue has no proper names — writing "Gaia DR3 4472832130942575872" once per star would cost 25 MB per million to repeat what two adjacent fields already say. An empty entry costs three bytes and is regenerated on load. The Sun needed its own case in the merge: it sits at the origin, has no direction to compare, and appears in every catalogue. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WaySiNst4HhDXBHnMy8p5G
2026-08-05 08:51:54 +00:00
parent 29fd92d118
commit efa9e4084a
18 changed files with 772 additions and 25 deletions
@@ -0,0 +1,108 @@
+import { raDegDecDistanceToXyz } from '../../../src/app/shared/astro/coordinates';
+import { StarRecord } from '../../../src/app/shared/models/star.model';
+import { parseCsvObjects, parseOptionalNumber } from '../lib/csv';
+import { fetchTextCached } from '../lib/http';
+
+/**
+ * Gaia DR3, via the ESA archive's TAP service.
+ *
+ * The only one of the large modern surveys that can add stars to a *3D* map, because it is the
+ * only one that measures parallaxes for them. Its 1.8 billion sources are roughly 1% of the
+ * Galaxy — no catalogue is close to the rest — but within a few hundred parsecs it is complete
+ * in a way Hipparcos never was, and its parallaxes are fifty times more precise.
+ *
+ * **This has never been run.** Every ESA, NOIRLab, SDSS and Euclid endpoint is unreachable from
+ * the environment this was written in, so the query below is written against the published DR3
+ * schema and has not been executed against it. Treat the column names as the first thing to
+ * check if a real run misbehaves.
+ */
+
+const GAIA_TAP_URL = 'https://gea.esac.esa.int/tap-server/tap/sync';
+
+/**
+ * How far out to take Gaia, in parsecs, and the faintest star to keep.
+ *
+ * Both exist to bound the download rather than the science. Gaia's parallaxes stay useful far
+ * past anything this map draws, so the limit here is a payload decision: the catalogue is baked
+ * into a static asset that a browser downloads before the first frame.
+ */
+const DISTANCE_CUTOFF_PC = Number(process.env['ETL_GAIA_DISTANCE_PC'] ?? 250);
+const MAGNITUDE_LIMIT = Number(process.env['ETL_GAIA_MAGNITUDE_LIMIT'] ?? 12);
+const ROW_LIMIT = Number(process.env['ETL_GAIA_ROW_LIMIT'] ?? 500000);
+
+/**
+ * Relative parallax error above which a star is dropped: a parallax measured to worse than 20%
+ * gives a distance that is not worth plotting, and inverting a noisy parallax biases it badly.
+ */
+const MAX_PARALLAX_ERROR_RATIO = 0.2;
+
+/** Parallax in milliarcseconds for a given distance — the query's cutoff, expressed as Gaia has it. */
+function parallaxFloorMas(distancePc: number): number {
+  return 1000 / distancePc;
+}
+
+function buildQuery(): string {
+  return [
+    `select top ${ROW_LIMIT}`,
+    'source_id, ra, dec, parallax, parallax_error, phot_g_mean_mag, bp_rp',
+    'from gaiadr3.gaia_source',
+    `where parallax > ${parallaxFloorMas(DISTANCE_CUTOFF_PC).toFixed(6)}`,
+    `and parallax_over_error > ${(1 / MAX_PARALLAX_ERROR_RATIO).toFixed(1)}`,
+    `and phot_g_mean_mag < ${MAGNITUDE_LIMIT}`,
+    'order by phot_g_mean_mag asc'
+  ].join(' ');
+}
+
+/**
+ * Gaia publishes no spectral classifications, but `bp_rp` is a colour index on the same footing
+ * as HYG's `ci` — so the app's existing colour and spectral-class handling works unchanged, and
+ * the spectral type is left as unknown rather than invented from the colour.
+ */
+const UNKNOWN_SPECTRAL_TYPE = 'Unknown';
+
+/**
+ * Gaia source ids are 19 digits and there are no proper names, so a star's identity here is its
+ * catalogue designation. The app's star ids are 32-bit, which a Gaia source id overflows, so the
+ * two are kept apart: `id` is assigned within this run's own range and the designation carries
+ * the real identifier in the name.
+ */
+const GAIA_ID_BASE = 1_000_000_000;
+
+export async function fetchGaiaStars(): Promise<StarRecord[]> {
+  const url = `${GAIA_TAP_URL}?REQUEST=doQuery&LANG=ADQL&FORMAT=csv&QUERY=${encodeURIComponent(buildQuery())}`;
+  console.log(`Fetching Gaia DR3 (within ${DISTANCE_CUTOFF_PC} pc, G < ${MAGNITUDE_LIMIT}, at most ${ROW_LIMIT} rows)...`);
+
+  const csv = await fetchTextCached(url, 'gaia-dr3.csv');
+  const rows = parseCsvObjects(csv);
+  const stars: StarRecord[] = [];
+
+  rows.forEach((row, index) => {
+    const parallaxMas = parseOptionalNumber(row['parallax']);
+    const raDeg = parseOptionalNumber(row['ra']);
+    const decDeg = parseOptionalNumber(row['dec']);
+    if (!parallaxMas || parallaxMas <= 0 || raDeg === undefined || decDeg === undefined) {
+      return;
+    }
+
+    const distancePc = 1000 / parallaxMas;
+    if (distancePc > DISTANCE_CUTOFF_PC) {
+      return;
+    }
+
+    const { x, y, z } = raDegDecDistanceToXyz(raDeg, decDeg, distancePc);
+    stars.push({
+      id: GAIA_ID_BASE + index,
+      name: `Gaia DR3 ${row['source_id']}`,
+      x,
+      y,
+      z,
+      magnitude: parseOptionalNumber(row['phot_g_mean_mag']) ?? MAGNITUDE_LIMIT,
+      spectralType: UNKNOWN_SPECTRAL_TYPE,
+      colorIndex: parseOptionalNumber(row['bp_rp']) ?? null,
+      source: 'gaia'
+    });
+  });
+
+  console.log(`  kept ${stars.length} Gaia stars (of ${rows.length} rows).`);
+  return stars;
+}
@@ -0,0 +1,80 @@
+import { fetchGaiaStars } from './gaia';
+import { StarSource } from './star-sources';
+
+/**
+ * Every catalogue this pipeline knows about, wired in or not, with an honest note on each.
+ *
+ * Kept as data so `npm run etl` can print what it actually used rather than what a README claims
+ * it uses. The four unimplemented entries are not placeholders for missing work: three of them
+ * cannot contribute stars to a 3D map at all, for the reason recorded against each.
+ */
+export const STAR_SOURCES: readonly StarSource[] = [
+  {
+    id: 'hyg',
+    name: 'HYG database (Hipparcos, Yale Bright Star, Gliese)',
+    role: 'positional',
+    endpoint: 'https://raw.githubusercontent.com/astronexus/HYG-Database',
+    contributes: 'A complete, named, spectrally classified bright-star catalogue with parallaxes — 68388 stars within 250 pc.',
+    unimplementedBecause: null
+  },
+  {
+    id: 'gaia',
+    name: 'Gaia DR3',
+    role: 'positional',
+    endpoint: 'https://gea.esac.esa.int/tap-server/tap/sync',
+    contributes:
+      'Parallaxes fifty times more precise than Hipparcos, for 1.8 billion sources — the only survey that can add stars to a 3D map, because it is the only one that measures how far away they are.',
+    unimplementedBecause: null,
+    fetch: fetchGaiaStars
+  },
+  {
+    id: 'decaps2',
+    name: 'DECaPS2 (Dark Energy Camera Plane Survey)',
+    role: 'backdrop',
+    endpoint: 'https://datalab.noirlab.edu/tap',
+    contributes:
+      'The deepest optical census of the southern galactic plane: 3.32 billion objects across 130 degrees, in the dust-obscured region every other catalogue thins out in.',
+    unimplementedBecause:
+      'Photometric only — no parallaxes, so not one of its 3.32 billion objects can be placed in depth. Fifty times Gaia’s object count and zero stars this map can position. It would enter as a direction-only backdrop layer, alongside the deep-sky shell.'
+  },
+  {
+    id: 'sdss5-mwm',
+    name: 'SDSS-V Milky Way Mapper',
+    role: 'enrichment',
+    endpoint: 'https://api.sdss.org',
+    contributes:
+      'All-sky optical and infrared spectroscopy: effective temperatures, surface gravities, metallicities and radial velocities, which is real spectral classification rather than a colour index standing in for one.',
+    unimplementedBecause:
+      'Adds no positions. It is keyed to targets Gaia already places, so it joins onto an existing catalogue rather than extending it — worth having once Gaia is in, and worth nothing before.'
+  },
+  {
+    id: 'euclid-q2-bulge',
+    name: 'Euclid Galactic Bulge Survey (Q2)',
+    role: 'backdrop',
+    endpoint: 'https://easidr.esac.esa.int/sas',
+    contributes: 'High-resolution imagery and astrometry of the crowded inner bulge, around 8 kpc away.',
+    unimplementedBecause:
+      'At 8 kpc a parallax is a few microarcseconds, so this is astrometry without usable distances for a map of this kind. Its natural use here is imagery — a real photograph of the bulge on the galactic view, in place of part of the procedural model.'
+  },
+  {
+    id: 'saga',
+    name: 'SAGA (Stellar Abundances for Galactic Archaeology)',
+    role: 'enrichment',
+    endpoint: 'https://sagadatabase.jp',
+    contributes: 'Compiled elemental abundances for metal-poor stars — the chemical record of how the Galaxy assembled.',
+    unimplementedBecause:
+      'A compilation keyed to stars other catalogues place, covering tens of thousands of objects rather than millions. Like SDSS-V it enriches; it cannot extend the map on its own.'
+  }
+];
+
+export function positionalSources(): readonly StarSource[] {
+  return STAR_SOURCES.filter((source) => source.role === 'positional' && source.fetch !== undefined);
+}
+
+/** A one-line-per-source report of what the pipeline can and cannot draw on. */
+export function describeSources(): string {
+  return STAR_SOURCES.map((source) => {
+    const status = source.unimplementedBecause === null ? (source.fetch ? 'wired in' : 'wired in (built separately)') : 'declared, not fetched';
+    return `  [${source.role}] ${source.name} — ${status}`;
+  }).join('\n');
+}
@@ -0,0 +1,45 @@
+import { StarRecord } from '../../../src/app/shared/models/star.model';
+
+/**
+ * The catalogues this map can draw on, and what each of them can actually contribute.
+ *
+ * The distinction that matters is not size. It is whether a catalogue knows how *far away* its
+ * objects are, because a 3D map cannot place a star it only has a direction for. That splits the
+ * available surveys into three roles which are not interchangeable, and a survey being enormous
+ * says nothing about which role it fills — DECaPS2 has fifty times Gaia's object count and
+ * cannot place a single one of them in depth.
+ */
+export type StarSourceRole =
+  /** Has a direction *and* a distance, usually from parallax. Can put a star in the scene. */
+  | 'positional'
+  /** Has stellar parameters keyed to an identifier. Adds knowledge about stars already placed. */
+  | 'enrichment'
+  /** Has directions but no usable distance. Can only be painted on the backdrop shell. */
+  | 'backdrop';
+
+export interface StarSource {
+  readonly id: string;
+  readonly name: string;
+  readonly role: StarSourceRole;
+  /** Where the data comes from, for the credits and for anyone re-running the pipeline. */
+  readonly endpoint: string;
+  /** What this source adds that the others do not. */
+  readonly contributes: string;
+  /**
+   * Why it is not yet wired in, or `null` when it is. Kept as data rather than as a comment so
+   * the ETL can print an honest summary of what actually ran.
+   */
+  readonly unimplementedBecause: string | null;
+  /** Fetches this source's stars. Absent for sources that are declared but not implemented. */
+  readonly fetch?: () => Promise<StarRecord[]>;
+}
+
+/**
+ * Precision of the parallax each positional source measures with, in milliarcseconds — which is
+ * what decides how far out its distances stay meaningful, and which of two catalogues to believe
+ * when both contain the same star.
+ */
+export const PARALLAX_PRECISION_MAS: Readonly<Record<string, number>> = {
+  hyg: 1,
+  gaia: 0.02
+};