Draw the whole catalogue, and build the aggregation the rest would need
Two things, one verified and one that cannot be. The render budget is now the whole catalogue: 68388 stars, one instanced draw call, which is what a GPU should be asked to do. The budget itself stays, because the catalogue is meant to grow past what any machine should draw at once — Gaia alone could contribute a million — and at that point the selection is what keeps the field legible rather than a grey wash. A `?stars=` override handles the machines that cannot, including the software rasterizer the end-to-end suite runs against, whose frame rate is two orders of magnitude below a real GPU's and which was measuring the rasterizer rather than the app. The aggregation is the second thing, and none of it has run. Every ESA, NOIRLab, SDSS and Euclid endpoint is unreachable from here — only GitHub raw is, which is why HYG and OpenNGC are the current sources. So this is infrastructure and a Gaia query written against the published DR3 schema, not data. What the framework encodes is that these surveys are not interchangeable. The distinction is not size but whether a catalogue knows how far away its objects are, because a 3D map cannot place a star it only has a direction for. Gaia is the only one of the five that can add stars here, because it is the only one that measures parallaxes. DECaPS2 has fifty times Gaia's object count and photometry alone — not one of its 3.32 billion objects can be placed in depth. Euclid's bulge is 8 kpc away, where a parallax is microarcseconds; its contribution would be imagery. SDSS-V and SAGA are keyed to stars something else already places, so they enrich rather than extend. Those roles are recorded as data the ETL prints, not as prose that can drift. Overlapping catalogues are reconciled on direction rather than on 3D proximity, which is the one non-obvious part. Two surveys agree on a star's direction to within an arcsecond and disagree on its distance by tens of per cent, so a star at 200 pc is 50 pc from itself between catalogues while being unmistakably the same object. Matching in 3D would need a tolerance so loose it swallowed real neighbours. The better parallax wins where both reach; where only one does, the star stays. Names become dense-with-holes with a source dictionary, because a survey catalogue has no proper names — writing "Gaia DR3 4472832130942575872" once per star would cost 25 MB per million to repeat what two adjacent fields already say. An empty entry costs three bytes and is regenerated on load. The Sun needed its own case in the merge: it sits at the origin, has no direction to compare, and appears in every catalogue. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WaySiNst4HhDXBHnMy8p5G
This commit is contained in:
@@ -318,3 +318,9 @@ Real photography where it exists, and a surface reasoned from measurements where
|
|||||||
- Raise the distance cutoff from 50 pc to 250 pc — where Hipparcos parallaxes stop being trustworthy — taking the catalogue from 8750 stars to 68388.
|
- Raise the distance cutoff from 50 pc to 250 pc — where Hipparcos parallaxes stop being trustworthy — taking the catalogue from 8750 stars to 68388.
|
||||||
- Give the star field a render budget: every star inside 25 pc plus the brightest of the rest. Search, navigation and the cross-reference still see the whole catalogue.
|
- Give the star field a render budget: every star inside 25 pc plus the brightest of the rest. Search, navigation and the cross-reference still see the whole catalogue.
|
||||||
- Store each exoplanet's host coordinates and re-resolve the cross-reference against the current catalogue at build time, so widening the star list rescues systems without re-downloading the archive. Renderable systems: 371 to 609.
|
- Store each exoplanet's host coordinates and re-resolve the cross-reference against the current catalogue at build time, so widening the star list rescues systems without re-downloading the archive. Renderable systems: 371 to 609.
|
||||||
|
|
||||||
|
### ✓ Step 12: Aggregate additional surveys, and draw the whole catalogue
|
||||||
|
- Raise the render budget to the full catalogue, with a `?stars=` override for machines (and test runs) that cannot draw it.
|
||||||
|
- Add a source registry with roles — positional, enrichment, backdrop — recording what each named survey can and cannot contribute, as data the ETL prints rather than as prose in a README.
|
||||||
|
- Add a Gaia DR3 TAP fetcher and a catalogue merge that matches on direction rather than 3D proximity, prefers the better parallax, and records provenance per star.
|
||||||
|
- Make names dense-with-holes and add a source dictionary, so a survey-scale catalogue with no proper names does not cost 25 MB per million stars to say what its id already says.
|
||||||
|
|||||||
@@ -169,7 +169,7 @@ re-runs are cheap and offline-friendly; set `ETL_FORCE_REFRESH=1` to bypass the
|
|||||||
|
|
||||||
| Script | Source | Output |
|
| Script | Source | Output |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| `fetchStars.ts` | HYG database (Hipparcos/Yale/Gliese) | `stars.bin`, `stars-meta.bin`, `stars-index.json` |
|
| `fetchStars.ts` | HYG database, plus any other positional catalogue wired in (see below) | `stars.bin`, `stars-meta.bin`, `stars-index.json` |
|
||||||
| `fetchSolarSystem.ts` | JPL Horizons / SSD | `bodies.json` |
|
| `fetchSolarSystem.ts` | JPL Horizons / SSD | `bodies.json` |
|
||||||
| `fetchExoplanets.ts` | NASA Exoplanet Archive (TAP) | `exoplanets.json` |
|
| `fetchExoplanets.ts` | NASA Exoplanet Archive (TAP) | `exoplanets.json` |
|
||||||
| `fetchDeepSky.ts` | OpenNGC | `deepsky.json` |
|
| `fetchDeepSky.ts` | OpenNGC | `deepsky.json` |
|
||||||
@@ -185,6 +185,40 @@ and both the ETL and the app use it, so the writer and the reader cannot drift a
|
|||||||
|
|
||||||
`ETL_STAR_DISTANCE_PC` (default `250`) sets the star-field distance cutoff.
|
`ETL_STAR_DISTANCE_PC` (default `250`) sets the star-field distance cutoff.
|
||||||
|
|
||||||
|
### On aggregating other surveys
|
||||||
|
|
||||||
|
`tools/etl/sources/registry.ts` lists every catalogue the pipeline knows about, and `npm run etl`
|
||||||
|
prints it. Sources are wired in by role, because the roles are not interchangeable:
|
||||||
|
|
||||||
|
| Source | Role | What it adds |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| HYG | positional | The named, spectrally classified bright-star spine — 68 388 stars within 250 pc. |
|
||||||
|
| Gaia DR3 | positional | Parallaxes fifty times more precise, for 1.8 billion sources. |
|
||||||
|
| DECaPS2 | backdrop | 3.32 billion objects across the southern galactic plane. |
|
||||||
|
| SDSS-V Milky Way Mapper | enrichment | Spectroscopic temperatures, gravities, metallicities, radial velocities. |
|
||||||
|
| Euclid Bulge Survey (Q2) | backdrop | High-resolution imagery and astrometry of the inner bulge. |
|
||||||
|
| SAGA | enrichment | Compiled elemental abundances for metal-poor stars. |
|
||||||
|
|
||||||
|
The distinction that matters is not size — it is whether a catalogue knows how **far away** its
|
||||||
|
objects are, because a 3D map cannot place a star it only has a direction for. **Gaia is the only
|
||||||
|
one of these that can add stars to this map**, because it is the only one that measures
|
||||||
|
parallaxes. DECaPS2 has fifty times Gaia's object count and photometry alone: not one of its
|
||||||
|
3.32 billion objects can be placed in depth, so it can only ever be a direction-only backdrop
|
||||||
|
beside the deep-sky shell. Euclid's bulge sits 8 kpc away, where a parallax is microarcseconds —
|
||||||
|
its natural contribution here is imagery, not positions. SDSS-V and SAGA are keyed to stars
|
||||||
|
another catalogue already places; they enrich what is there and cannot extend it.
|
||||||
|
|
||||||
|
Where two positional catalogues overlap they are reconciled by `star-merge.ts`, which matches on
|
||||||
|
**direction** rather than on 3D proximity. Two surveys agree on a star's direction to within an
|
||||||
|
arcsecond and disagree on its distance by tens of per cent — so a star at 200 pc can be 50 pc
|
||||||
|
from itself between catalogues while being unmistakably the same object. Where both have a star,
|
||||||
|
the one with the better parallax wins; where only one reaches, the star is still there.
|
||||||
|
|
||||||
|
Only HYG has ever run. Every ESA, NOIRLab, SDSS and Euclid endpoint is unreachable from the
|
||||||
|
environment this was developed in, so the Gaia query is written against the published DR3 schema
|
||||||
|
and has not been executed against it. A source that cannot be reached is reported and skipped
|
||||||
|
rather than failing the build.
|
||||||
|
|
||||||
### On how many stars
|
### On how many stars
|
||||||
|
|
||||||
Not many, against the Galaxy. It holds 100–400 billion stars and this map ships 68 388 of them —
|
Not many, against the Galaxy. It holds 100–400 billion stars and this map ships 68 388 of them —
|
||||||
|
|||||||
@@ -9,7 +9,10 @@ test.describe('Camera-flight transitions (click-to-select)', () => {
|
|||||||
// per-test budget covers that only just, and stopped covering it once a second flight-heavy
|
// per-test budget covers that only just, and stopped covering it once a second flight-heavy
|
||||||
// spec started running alongside this one.
|
// spec started running alongside this one.
|
||||||
test.setTimeout(90_000);
|
test.setTimeout(90_000);
|
||||||
await page.goto('/');
|
// A reduced star field, because this suite runs against a software rasterizer two orders of
|
||||||
|
// magnitude slower than a GPU, and what is under test here is the navigation state machine
|
||||||
|
// rather than how fast 68388 sprites rasterize. See `starRenderBudgetFromUrl`.
|
||||||
|
await page.goto('/?stars=4000');
|
||||||
|
|
||||||
const canvas = page.getByTestId('scene-canvas');
|
const canvas = page.getByTestId('scene-canvas');
|
||||||
await expect(canvas).toBeVisible();
|
await expect(canvas).toBeVisible();
|
||||||
|
|||||||
@@ -2,9 +2,12 @@ import { expect, test } from '@playwright/test';
|
|||||||
|
|
||||||
import { backButtonLocator } from './support/wait-for-back-button';
|
import { backButtonLocator } from './support/wait-for-back-button';
|
||||||
|
|
||||||
|
// A reduced star field throughout: this suite runs against a software rasterizer two orders of
|
||||||
|
// magnitude slower than a GPU, and what is under test is navigation and state rather than how
|
||||||
|
// fast the field rasterizes. See `starRenderBudgetFromUrl`.
|
||||||
test.describe('Galaxy view', () => {
|
test.describe('Galaxy view', () => {
|
||||||
test('boots the app, initializes the 3D scene, and starts in the galaxy overview (no system controls shown)', async ({ page }) => {
|
test('boots the app, initializes the 3D scene, and starts in the galaxy overview (no system controls shown)', async ({ page }) => {
|
||||||
await page.goto('/');
|
await page.goto('/?stars=4000');
|
||||||
|
|
||||||
await expect(page.getByTestId('scene-canvas')).toBeVisible();
|
await expect(page.getByTestId('scene-canvas')).toBeVisible();
|
||||||
await expect(page.getByPlaceholder('Search stars, planets, exoplanets…')).toBeVisible();
|
await expect(page.getByPlaceholder('Search stars, planets, exoplanets…')).toBeVisible();
|
||||||
@@ -16,7 +19,7 @@ test.describe('Galaxy view', () => {
|
|||||||
// Two multi-second camera flights, either side of a software-rendered scene bootstrap, add
|
// Two multi-second camera flights, either side of a software-rendered scene bootstrap, add
|
||||||
// up to more than the default per-test budget.
|
// up to more than the default per-test budget.
|
||||||
test.setTimeout(90_000);
|
test.setTimeout(90_000);
|
||||||
await page.goto('/');
|
await page.goto('/?stars=4000');
|
||||||
await expect(page.getByTestId('scene-canvas')).toBeVisible();
|
await expect(page.getByTestId('scene-canvas')).toBeVisible();
|
||||||
|
|
||||||
await page.getByRole('button', { name: 'Milky Way' }).click();
|
await page.getByRole('button', { name: 'Milky Way' }).click();
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ import { backButtonLocator } from './support/wait-for-back-button';
|
|||||||
|
|
||||||
test.describe('Search-driven navigation', () => {
|
test.describe('Search-driven navigation', () => {
|
||||||
test('selecting a star result flies into that system and shows the back-to-galaxy control', async ({ page }) => {
|
test('selecting a star result flies into that system and shows the back-to-galaxy control', async ({ page }) => {
|
||||||
await page.goto('/');
|
await page.goto('/?stars=4000');
|
||||||
const searchInput = page.getByPlaceholder('Search stars, planets, exoplanets…');
|
const searchInput = page.getByPlaceholder('Search stars, planets, exoplanets…');
|
||||||
await searchInput.fill('Proxima Centauri');
|
await searchInput.fill('Proxima Centauri');
|
||||||
|
|
||||||
@@ -18,7 +18,7 @@ test.describe('Search-driven navigation', () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
test('selecting a body result navigates straight to its detail route and shows real NASA data', async ({ page }) => {
|
test('selecting a body result navigates straight to its detail route and shows real NASA data', async ({ page }) => {
|
||||||
await page.goto('/');
|
await page.goto('/?stars=4000');
|
||||||
const searchInput = page.getByPlaceholder('Search stars, planets, exoplanets…');
|
const searchInput = page.getByPlaceholder('Search stars, planets, exoplanets…');
|
||||||
await searchInput.fill('Earth');
|
await searchInput.fill('Earth');
|
||||||
|
|
||||||
@@ -35,7 +35,7 @@ test.describe('Search-driven navigation', () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
test('typing fewer than two characters shows no results, and Escape clears the query', async ({ page }) => {
|
test('typing fewer than two characters shows no results, and Escape clears the query', async ({ page }) => {
|
||||||
await page.goto('/');
|
await page.goto('/?stars=4000');
|
||||||
const searchInput = page.getByPlaceholder('Search stars, planets, exoplanets…');
|
const searchInput = page.getByPlaceholder('Search stars, planets, exoplanets…');
|
||||||
|
|
||||||
await searchInput.fill('E');
|
await searchInput.fill('E');
|
||||||
|
|||||||
@@ -21,7 +21,7 @@ import { galacticNormal, PolarGridPlane, TetherField } from './grid-plane';
|
|||||||
import { MilkyWayRenderer } from './milky-way-renderer';
|
import { MilkyWayRenderer } from './milky-way-renderer';
|
||||||
import { starGlowExtentAu, starMarkerRadiusAu, systemFrameRadiusAu, systemFramingDistanceAu, systemViewDirection } from './system-framing';
|
import { starGlowExtentAu, starMarkerRadiusAu, systemFrameRadiusAu, systemFramingDistanceAu, systemViewDirection } from './system-framing';
|
||||||
import { HudReadout, StarmapHudComponent } from './starmap-hud.component';
|
import { HudReadout, StarmapHudComponent } from './starmap-hud.component';
|
||||||
import { colorIndexToRgb, StarFieldRenderer } from './star-field-renderer';
|
import { colorIndexToRgb, StarFieldRenderer, starRenderBudgetFromUrl } from './star-field-renderer';
|
||||||
import { LabeledPoint, StarLabelOverlay } from './star-label-overlay';
|
import { LabeledPoint, StarLabelOverlay } from './star-label-overlay';
|
||||||
import { SystemOrbitsRenderer } from './system-orbits-renderer';
|
import { SystemOrbitsRenderer } from './system-orbits-renderer';
|
||||||
|
|
||||||
@@ -340,7 +340,7 @@ export class GalaxySystemSceneComponent implements AfterViewInit, OnDestroy {
|
|||||||
this.bodies = bodies;
|
this.bodies = bodies;
|
||||||
this.exoplanets = exoplanets;
|
this.exoplanets = exoplanets;
|
||||||
|
|
||||||
this.starField = new StarFieldRenderer(stars, positions);
|
this.starField = new StarFieldRenderer(stars, positions, starRenderBudgetFromUrl(window.location.search));
|
||||||
this.galaxyGroup.add(this.starField.object);
|
this.galaxyGroup.add(this.starField.object);
|
||||||
|
|
||||||
this.milkyWay = new MilkyWayRenderer();
|
this.milkyWay = new MilkyWayRenderer();
|
||||||
|
|||||||
@@ -38,12 +38,17 @@ const PICK_NDC_SLOP = 0.01;
|
|||||||
* So the *data* is the catalogue and the *drawing* is a budget, and the two are allowed to
|
* So the *data* is the catalogue and the *drawing* is a budget, and the two are allowed to
|
||||||
* differ. Everything still exists for search, for flying to, and for hosting planets.
|
* differ. Everything still exists for search, for flying to, and for hosting planets.
|
||||||
*
|
*
|
||||||
* The figure is set low deliberately. A real GPU would draw the whole catalogue without
|
* Currently set to the whole catalogue, which is what a GPU should be asked to do — this is one
|
||||||
* noticing — this is one instanced draw call — but the value that matters is what a weak one
|
* instanced draw call, and a discrete card will not notice it. The budget still exists because
|
||||||
* does, and the software rasterizer this was measured against costs a third of its frame rate
|
* the catalogue is meant to grow past what any machine should draw at once: Gaia alone could
|
||||||
* per 12000 stars. Raise it freely on hardware that can take it; nothing else depends on it.
|
* contribute a million stars, and at that point the selection below is what keeps the field
|
||||||
|
* legible rather than a grey wash.
|
||||||
|
*
|
||||||
|
* Machines without a GPU do feel it. A software rasterizer measured here lost about a third of
|
||||||
|
* its frame rate per 12000 stars drawn; if that matters for a deployment, this is the one number
|
||||||
|
* to turn down.
|
||||||
*/
|
*/
|
||||||
export const STAR_RENDER_BUDGET = 12000;
|
export const STAR_RENDER_BUDGET = 68388;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Radius (parsecs) inside which every star is drawn regardless of brightness.
|
* Radius (parsecs) inside which every star is drawn regardless of brightness.
|
||||||
@@ -128,6 +133,20 @@ function createQuadGeometry(instanceCount: number): THREE.InstancedBufferGeometr
|
|||||||
* them. Returns them in catalogue order rather than in selection order, purely so the drawn set
|
* them. Returns them in catalogue order rather than in selection order, purely so the drawn set
|
||||||
* is stable and inspectable.
|
* is stable and inspectable.
|
||||||
*/
|
*/
|
||||||
|
/**
|
||||||
|
* Reads a render budget override off the page URL (`?stars=20000`), falling back to the default.
|
||||||
|
*
|
||||||
|
* Two uses, one real and one incidental. The real one is a deployment or a machine that cannot
|
||||||
|
* draw the whole catalogue — a number in a URL beats a rebuild. The incidental one is the
|
||||||
|
* end-to-end suite, which runs against a software rasterizer whose frame rate is two orders of
|
||||||
|
* magnitude below a real GPU's: those tests are checking navigation and state, and making them
|
||||||
|
* wait on a rasterizer measures nothing about the app.
|
||||||
|
*/
|
||||||
|
export function starRenderBudgetFromUrl(search: string, fallback = STAR_RENDER_BUDGET): number {
|
||||||
|
const requested = Number(new URLSearchParams(search).get('stars'));
|
||||||
|
return Number.isFinite(requested) && requested > 0 ? Math.floor(requested) : fallback;
|
||||||
|
}
|
||||||
|
|
||||||
export function selectDrawnStars(stars: readonly StarRecord[], budget = STAR_RENDER_BUDGET): Uint32Array {
|
export function selectDrawnStars(stars: readonly StarRecord[], budget = STAR_RENDER_BUDGET): Uint32Array {
|
||||||
if (stars.length <= budget) {
|
if (stars.length <= budget) {
|
||||||
return Uint32Array.from(stars.keys());
|
return Uint32Array.from(stars.keys());
|
||||||
|
|||||||
@@ -0,0 +1,148 @@
|
|||||||
|
import { describe, expect, it } from 'vitest';
|
||||||
|
|
||||||
|
import { raDegDecDistanceToXyz } from './coordinates';
|
||||||
|
import { StarRecord } from '../models/star.model';
|
||||||
|
import { directionCosine, isSameStar, mergeStarCatalogues } from './star-merge';
|
||||||
|
|
||||||
|
/** A star at a given sky position and distance, which is how catalogues actually report them. */
|
||||||
|
function at(id: number, raDeg: number, decDeg: number, distancePc: number, overrides: Partial<StarRecord> = {}): StarRecord {
|
||||||
|
const { x, y, z } = raDegDecDistanceToXyz(raDeg, decDeg, distancePc);
|
||||||
|
return { id, name: `star-${id}`, x, y, z, magnitude: 5, spectralType: 'G2V', colorIndex: 0.6, ...overrides };
|
||||||
|
}
|
||||||
|
|
||||||
|
const HIPPARCOS = { sourceId: 'hyg', parallaxPrecisionMas: 1 };
|
||||||
|
const GAIA = { sourceId: 'gaia', parallaxPrecisionMas: 0.02 };
|
||||||
|
|
||||||
|
describe('isSameStar', () => {
|
||||||
|
it('matches two catalogues reporting the same star', () => {
|
||||||
|
expect(isSameStar(at(1, 101.28, -16.71, 2.64), at(2, 101.28, -16.71, 2.63))).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('tolerates the distance disagreement two parallaxes actually have', () => {
|
||||||
|
// Hipparcos and Gaia routinely differ by tens of per cent at a few hundred parsecs. That
|
||||||
|
// disagreement is the reason to prefer one of them, not evidence they are different stars.
|
||||||
|
expect(isSameStar(at(1, 200, 10, 200), at(2, 200, 10, 260))).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('does not match two different stars that happen to be at the same distance', () => {
|
||||||
|
expect(isSameStar(at(1, 200, 10, 200), at(2, 200.5, 10, 200))).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('does not match along a line of sight when the distances genuinely conflict', () => {
|
||||||
|
// Same direction, one three times further away: a background star, not the same object.
|
||||||
|
expect(isSameStar(at(1, 200, 10, 100), at(2, 200, 10, 300))).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('matches on direction rather than on 3D proximity', () => {
|
||||||
|
// The distinction the merge rests on. These two are 60 pc apart in space and are the same
|
||||||
|
// star; a 3D-proximity test would have to be so loose it swallowed real neighbours.
|
||||||
|
const a = at(1, 45, 20, 200);
|
||||||
|
const b = at(2, 45, 20, 260);
|
||||||
|
expect(Math.hypot(a.x - b.x, a.y - b.y, a.z - b.z)).toBeGreaterThan(50);
|
||||||
|
expect(isSameStar(a, b)).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('treats two stars at the origin as the same, and one at the origin as unlike any other', () => {
|
||||||
|
const origin: StarRecord = { id: 0, name: 'Sol', x: 0, y: 0, z: 0, magnitude: -26.7, spectralType: 'G2V', colorIndex: 0.65 };
|
||||||
|
expect(isSameStar(origin, { ...origin, id: 1 })).toBe(true);
|
||||||
|
expect(isSameStar(origin, at(2, 45, 20, 10))).toBe(false);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('directionCosine', () => {
|
||||||
|
it('is one for the same direction and stays inside the domain of acos', () => {
|
||||||
|
expect(directionCosine(at(1, 45, 20, 5), at(2, 45, 20, 500))).toBeCloseTo(1, 12);
|
||||||
|
expect(Math.abs(directionCosine(at(1, 45, 20, 5), at(2, 225, -20, 5)))).toBeLessThanOrEqual(1);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('mergeStarCatalogues', () => {
|
||||||
|
it('keeps the better-measured catalogue where two overlap', () => {
|
||||||
|
// Gaia's parallax is fifty times more precise, so where both have a star, its position is
|
||||||
|
// Gaia's — regardless of which catalogue was passed first.
|
||||||
|
const shared = { raDeg: 101.28, decDeg: -16.71 };
|
||||||
|
const { stars, summary } = mergeStarCatalogues([
|
||||||
|
{ ...HIPPARCOS, stars: [at(1, shared.raDeg, shared.decDeg, 2.7)] },
|
||||||
|
{ ...GAIA, stars: [at(2, shared.raDeg, shared.decDeg, 2.64)] }
|
||||||
|
]);
|
||||||
|
|
||||||
|
expect(stars).toHaveLength(1);
|
||||||
|
expect(stars[0].id).toBe(2);
|
||||||
|
expect(stars[0].source).toBe('gaia');
|
||||||
|
expect(summary.duplicates).toBe(1);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('keeps a star the better catalogue does not reach', () => {
|
||||||
|
// The point of merging rather than replacing: Gaia is more precise but not a superset of
|
||||||
|
// everything, and a bright star it omits should not vanish from the map.
|
||||||
|
const { stars } = mergeStarCatalogues([
|
||||||
|
{ ...HIPPARCOS, stars: [at(1, 10, 10, 100)] },
|
||||||
|
{ ...GAIA, stars: [at(2, 200, -30, 50)] }
|
||||||
|
]);
|
||||||
|
|
||||||
|
expect(stars.map((star) => star.id).sort()).toEqual([1, 2]);
|
||||||
|
expect(stars.find((star) => star.id === 1)?.source).toBe('hyg');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('records where every star came from', () => {
|
||||||
|
const { stars, summary } = mergeStarCatalogues([
|
||||||
|
{ ...HIPPARCOS, stars: [at(1, 10, 10, 100), at(3, 20, 10, 100)] },
|
||||||
|
{ ...GAIA, stars: [at(2, 200, -30, 50)] }
|
||||||
|
]);
|
||||||
|
|
||||||
|
expect(summary.bySource).toEqual({ hyg: 2, gaia: 1 });
|
||||||
|
expect(new Set(stars.map((star) => star.source))).toEqual(new Set(['hyg', 'gaia']));
|
||||||
|
});
|
||||||
|
|
||||||
|
it('does not depend on the order the catalogues were given in', () => {
|
||||||
|
const shared = [at(1, 30, 5, 80)];
|
||||||
|
const better = [at(2, 30, 5, 79)];
|
||||||
|
const forwards = mergeStarCatalogues([{ ...HIPPARCOS, stars: shared }, { ...GAIA, stars: better }]);
|
||||||
|
const backwards = mergeStarCatalogues([{ ...GAIA, stars: better }, { ...HIPPARCOS, stars: shared }]);
|
||||||
|
|
||||||
|
expect(forwards.stars.map((s) => s.id)).toEqual(backwards.stars.map((s) => s.id));
|
||||||
|
});
|
||||||
|
|
||||||
|
it('leaves a star that already names its source alone', () => {
|
||||||
|
const { stars } = mergeStarCatalogues([{ ...GAIA, stars: [at(1, 10, 10, 100, { source: 'gaia-dr4' })] }]);
|
||||||
|
expect(stars[0].source).toBe('gaia-dr4');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('finds duplicates that straddle a sky-grid boundary', () => {
|
||||||
|
// The bucketing is an optimisation, and an optimisation that changes the answer is a bug.
|
||||||
|
// Every one of these sits on or beside a cell edge.
|
||||||
|
for (const [raDeg, decDeg] of [
|
||||||
|
[0, 0],
|
||||||
|
[0.5, 0.5],
|
||||||
|
[359.999, -0.0001],
|
||||||
|
[180, 89.9]
|
||||||
|
]) {
|
||||||
|
const { stars } = mergeStarCatalogues([
|
||||||
|
{ ...HIPPARCOS, stars: [at(1, raDeg, decDeg, 100)] },
|
||||||
|
{ ...GAIA, stars: [at(2, raDeg, decDeg, 100)] }
|
||||||
|
]);
|
||||||
|
expect(stars).toHaveLength(1);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
it('handles a single catalogue as a plain pass-through', () => {
|
||||||
|
const { stars, summary } = mergeStarCatalogues([{ ...HIPPARCOS, stars: [at(1, 10, 10, 100), at(2, 20, 20, 100)] }]);
|
||||||
|
expect(stars).toHaveLength(2);
|
||||||
|
expect(summary.duplicates).toBe(0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('handles no catalogues at all', () => {
|
||||||
|
expect(mergeStarCatalogues([]).stars).toEqual([]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('scales to catalogues large enough to matter', () => {
|
||||||
|
// The reason for the sky grid: the naive pairwise merge is quadratic, and these surveys are
|
||||||
|
// the size where that stops being an academic point.
|
||||||
|
const many = Array.from({ length: 20000 }, (_, i) => at(i, (i * 0.017) % 360, ((i * 0.031) % 160) - 80, 100));
|
||||||
|
const started = Date.now();
|
||||||
|
const { stars } = mergeStarCatalogues([{ ...HIPPARCOS, stars: many }, { ...GAIA, stars: many.map((s) => ({ ...s, id: s.id + 100000 })) }]);
|
||||||
|
|
||||||
|
expect(stars).toHaveLength(20000);
|
||||||
|
expect(Date.now() - started).toBeLessThan(10000);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,149 @@
|
|||||||
|
import { StarRecord } from '../models/star.model';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Merges star catalogues that overlap.
|
||||||
|
*
|
||||||
|
* Every all-sky survey contains the bright stars, so unioning two catalogues without matching
|
||||||
|
* them first would draw Sirius twice — in slightly different places, since two instruments never
|
||||||
|
* agree exactly. The merge therefore has to decide when two rows are the same object, and which
|
||||||
|
* of them to believe.
|
||||||
|
*
|
||||||
|
* Identity is decided on the sky rather than in space. Two catalogues agree closely on a star's
|
||||||
|
* *direction* — it is an angle, measured directly — and disagree much more on its *distance*,
|
||||||
|
* which comes from a parallax with real error bars. Matching on 3D proximity would therefore
|
||||||
|
* fail exactly where the catalogues are most useful: a star at 200 pc with a 25% distance
|
||||||
|
* disagreement is 50 pc from itself, while its direction is identical to within an arcsecond.
|
||||||
|
*/
|
||||||
|
|
||||||
|
const DEG_TO_RAD = Math.PI / 180;
|
||||||
|
|
||||||
|
/** Angular separation, in degrees, below which two entries are taken to be the same star. */
|
||||||
|
export const MERGE_ANGULAR_TOLERANCE_DEG = 1 / 3600;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* How far two distances may disagree, as a ratio, and still describe the same star. Generous on
|
||||||
|
* purpose: Hipparcos and Gaia routinely differ by tens of per cent at a few hundred parsecs, and
|
||||||
|
* that disagreement is the *reason* to prefer one, not evidence they are different objects.
|
||||||
|
*/
|
||||||
|
export const MERGE_DISTANCE_RATIO_TOLERANCE = 0.5;
|
||||||
|
|
||||||
|
export interface MergeCandidate {
|
||||||
|
readonly sourceId: string;
|
||||||
|
/** Lower is better — the parallax precision this source measures with, in milliarcseconds. */
|
||||||
|
readonly parallaxPrecisionMas: number;
|
||||||
|
readonly stars: readonly StarRecord[];
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface MergeSummary {
|
||||||
|
readonly total: number;
|
||||||
|
/** Entries dropped because a better-measured catalogue already had that star. */
|
||||||
|
readonly duplicates: number;
|
||||||
|
readonly bySource: Readonly<Record<string, number>>;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Unit direction of a star, which is the quantity catalogues actually agree on. */
|
||||||
|
function direction(star: StarRecord): [number, number, number] {
|
||||||
|
const length = Math.hypot(star.x, star.y, star.z);
|
||||||
|
return length === 0 ? [0, 0, 0] : [star.x / length, star.y / length, star.z / length];
|
||||||
|
}
|
||||||
|
|
||||||
|
function distanceOf(star: StarRecord): number {
|
||||||
|
return Math.hypot(star.x, star.y, star.z);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Buckets a direction onto a coarse sky grid, so a star only has to be compared against the
|
||||||
|
* handful of entries near it rather than against every star already merged.
|
||||||
|
*
|
||||||
|
* The cell is much larger than the match tolerance, so a pair straddling a boundary would be
|
||||||
|
* missed — which is why {@link neighbouringCells} checks the adjacent cells too.
|
||||||
|
*/
|
||||||
|
const SKY_CELL_DEG = 0.5;
|
||||||
|
|
||||||
|
function cellKey(raDeg: number, decDeg: number): string {
|
||||||
|
return `${Math.floor(raDeg / SKY_CELL_DEG)}:${Math.floor(decDeg / SKY_CELL_DEG)}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function skyAngles(star: StarRecord): { raDeg: number; decDeg: number } {
|
||||||
|
const [x, y, z] = direction(star);
|
||||||
|
return { raDeg: (Math.atan2(y, x) / DEG_TO_RAD + 360) % 360, decDeg: Math.asin(Math.max(-1, Math.min(1, z))) / DEG_TO_RAD };
|
||||||
|
}
|
||||||
|
|
||||||
|
function neighbouringCells(raDeg: number, decDeg: number): string[] {
|
||||||
|
const keys: string[] = [];
|
||||||
|
for (let dRa = -1; dRa <= 1; dRa++) {
|
||||||
|
for (let dDec = -1; dDec <= 1; dDec++) {
|
||||||
|
keys.push(cellKey(raDeg + dRa * SKY_CELL_DEG, decDeg + dDec * SKY_CELL_DEG));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return keys;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Cosine of the angle between two stars' directions. */
|
||||||
|
export function directionCosine(a: StarRecord, b: StarRecord): number {
|
||||||
|
const [ax, ay, az] = direction(a);
|
||||||
|
const [bx, by, bz] = direction(b);
|
||||||
|
return Math.max(-1, Math.min(1, ax * bx + ay * by + az * bz));
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Whether two entries describe the same star: same direction, and distances not in conflict. */
|
||||||
|
export function isSameStar(a: StarRecord, b: StarRecord): boolean {
|
||||||
|
const [near, far] = [distanceOf(a), distanceOf(b)].sort((p, q) => p - q);
|
||||||
|
|
||||||
|
// The Sun sits at the origin of this coordinate system and so has no direction at all, which
|
||||||
|
// the angular test below cannot speak about. Every catalogue contains it, so without this the
|
||||||
|
// merge would happily keep one Sun per source.
|
||||||
|
if (near === 0) {
|
||||||
|
return far === 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
const separationDeg = Math.acos(directionCosine(a, b)) / DEG_TO_RAD;
|
||||||
|
if (separationDeg > MERGE_ANGULAR_TOLERANCE_DEG) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
return (far - near) / near <= MERGE_DISTANCE_RATIO_TOLERANCE;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Unions the given catalogues, keeping one entry per star.
|
||||||
|
*
|
||||||
|
* Sources are taken in order of how precisely they measure parallax, best first, and a star is
|
||||||
|
* only added if no better-measured catalogue already has it. So where Gaia and Hipparcos
|
||||||
|
* overlap, the position is Gaia's; where only Hipparcos reaches, the star is still there.
|
||||||
|
*/
|
||||||
|
export function mergeStarCatalogues(candidates: readonly MergeCandidate[]): { stars: StarRecord[]; summary: MergeSummary } {
|
||||||
|
const ordered = [...candidates].sort((a, b) => a.parallaxPrecisionMas - b.parallaxPrecisionMas);
|
||||||
|
const merged: StarRecord[] = [];
|
||||||
|
const grid = new Map<string, StarRecord[]>();
|
||||||
|
const bySource: Record<string, number> = {};
|
||||||
|
let duplicates = 0;
|
||||||
|
|
||||||
|
for (const candidate of ordered) {
|
||||||
|
bySource[candidate.sourceId] = 0;
|
||||||
|
|
||||||
|
for (const star of candidate.stars) {
|
||||||
|
const { raDeg, decDeg } = skyAngles(star);
|
||||||
|
const alreadyPresent = neighbouringCells(raDeg, decDeg).some((key) => (grid.get(key) ?? []).some((existing) => isSameStar(existing, star)));
|
||||||
|
|
||||||
|
if (alreadyPresent) {
|
||||||
|
duplicates++;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
const withSource: StarRecord = { ...star, source: star.source ?? candidate.sourceId };
|
||||||
|
merged.push(withSource);
|
||||||
|
bySource[candidate.sourceId]++;
|
||||||
|
|
||||||
|
const key = cellKey(raDeg, decDeg);
|
||||||
|
const cell = grid.get(key);
|
||||||
|
if (cell) {
|
||||||
|
cell.push(withSource);
|
||||||
|
} else {
|
||||||
|
grid.set(key, [withSource]);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return { stars: merged, summary: { total: merged.length, duplicates, bySource } };
|
||||||
|
}
|
||||||
@@ -54,11 +54,15 @@ describe('encodeStarCatalog / decodeStarCatalog', () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
it('keeps the index free of anything that is not a string, since the numbers are elsewhere', () => {
|
it('keeps the index free of anything that is not a string, since the numbers are elsewhere', () => {
|
||||||
expect(Object.keys(encoded.index).sort()).toEqual(['count', 'names', 'spectralTypes']);
|
|
||||||
expect(encoded.index.count).toBe(STARS.length);
|
expect(encoded.index.count).toBe(STARS.length);
|
||||||
expect(encoded.index.names).toEqual(STARS.map((star) => star.name));
|
expect(encoded.index.names).toEqual(STARS.map((star) => star.name));
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it('writes no per-star source column when every star came from the same place', () => {
|
||||||
|
// It would be a couple of hundred kilobytes to say nothing.
|
||||||
|
expect(encoded.index.sourceIndices).toEqual([]);
|
||||||
|
});
|
||||||
|
|
||||||
it('is smaller than the array of objects it replaced', () => {
|
it('is smaller than the array of objects it replaced', () => {
|
||||||
// The whole reason for the format: the old encoding repeated eight key names per star.
|
// The whole reason for the format: the old encoding repeated eight key names per star.
|
||||||
const asObjects = JSON.stringify(STARS).length;
|
const asObjects = JSON.stringify(STARS).length;
|
||||||
@@ -91,3 +95,40 @@ describe('encodeStarCatalog / decodeStarCatalog', () => {
|
|||||||
expect(round[4999].id).toBe(4999);
|
expect(round[4999].id).toBe(4999);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|
||||||
|
describe('star catalogue provenance and derived names', () => {
|
||||||
|
const MIXED: StarRecord[] = [
|
||||||
|
{ id: 5, name: 'Sirius', x: 1, y: 0, z: 0, magnitude: -1.4, spectralType: 'A0', colorIndex: 0.0, source: 'hyg' },
|
||||||
|
// A survey star with no name of its own: what it is called is its catalogue designation.
|
||||||
|
{ id: 900, name: 'Gaia DR3 900', x: 0, y: 2, z: 0, magnitude: 11, spectralType: 'Unknown', colorIndex: 1.2, source: 'gaia' },
|
||||||
|
{ id: 901, name: 'Gaia DR3 901', x: 0, y: 0, z: 3, magnitude: 11.5, spectralType: 'Unknown', colorIndex: 1.3, source: 'gaia' }
|
||||||
|
];
|
||||||
|
|
||||||
|
const encoded = encodeStarCatalog(MIXED);
|
||||||
|
const decoded = decodeStarCatalog(encoded.index, encoded.positions, encoded.meta);
|
||||||
|
|
||||||
|
it('stores nothing for a name that is just the catalogue designation', () => {
|
||||||
|
// 25 bytes per star, per million stars, to repeat what two adjacent fields already say.
|
||||||
|
expect(encoded.index.names).toEqual(['Sirius', '', '']);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('regenerates those names exactly on the way back', () => {
|
||||||
|
expect(decoded.map((star) => star.name)).toEqual(['Sirius', 'Gaia DR3 900', 'Gaia DR3 901']);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('carries each star provenance through', () => {
|
||||||
|
expect(decoded.map((star) => star.source)).toEqual(['hyg', 'gaia', 'gaia']);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('writes the source column only once the stars differ', () => {
|
||||||
|
expect(encoded.index.sources.map((source) => source.id)).toEqual(['hyg', 'gaia']);
|
||||||
|
expect(encoded.index.sourceIndices).toEqual([0, 1, 1]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('keeps a real name even when the star has a source that could generate one', () => {
|
||||||
|
const named = encodeStarCatalog([{ ...MIXED[1], name: 'Some Proper Name' }]);
|
||||||
|
expect(named.index.names).toEqual(['Some Proper Name']);
|
||||||
|
expect(decodeStarCatalog(named.index, named.positions, named.meta)[0].name).toBe('Some Proper Name');
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|||||||
@@ -37,12 +37,42 @@ export const BYTES_PER_STAR_META =
|
|||||||
/** `stars-index.json`: everything that is a string, plus the count the columns are sized by. */
|
/** `stars-index.json`: everything that is a string, plus the count the columns are sized by. */
|
||||||
export interface StarCatalogIndex {
|
export interface StarCatalogIndex {
|
||||||
count: number;
|
count: number;
|
||||||
/** One per star, in catalogue order. */
|
/**
|
||||||
|
* One per star, in catalogue order — but empty where the name is simply the star's catalogue
|
||||||
|
* designation, which is regenerated on load from the source and the id.
|
||||||
|
*
|
||||||
|
* A survey-scale catalogue has no proper names to speak of. Gaia's designations are its
|
||||||
|
* 19-digit source ids, so writing "Gaia DR3 4472832130942575872" once per star would cost
|
||||||
|
* 25 MB per million stars — more than the whole rest of the catalogue — to store a string that
|
||||||
|
* is already implied by two fields next to it. An empty entry costs three bytes.
|
||||||
|
*
|
||||||
|
* Dense with holes rather than a list of pairs, because which one is smaller depends entirely
|
||||||
|
* on the catalogue: every HYG star has a designation worth keeping, and paying an index per
|
||||||
|
* entry to say so would be 60% larger than just writing them in order.
|
||||||
|
*/
|
||||||
names: string[];
|
names: string[];
|
||||||
/** Distinct spectral classifications; the meta column holds indices into this. */
|
/** Distinct spectral classifications; the meta column holds indices into this. */
|
||||||
spectralTypes: string[];
|
spectralTypes: string[];
|
||||||
|
/**
|
||||||
|
* Distinct source ids, in the same dictionary form as the spectral types, plus the designation
|
||||||
|
* prefix each one names its unnamed stars with.
|
||||||
|
*/
|
||||||
|
sources: { id: string; designationPrefix: string }[];
|
||||||
|
/**
|
||||||
|
* One per star: an index into `sources`. Empty when every star came from the same place, which
|
||||||
|
* would otherwise cost a couple of hundred kilobytes to say nothing.
|
||||||
|
*/
|
||||||
|
sourceIndices: number[];
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* How a source names a star that has no name of its own. `Gaia DR3 <id>` for Gaia; HYG's own
|
||||||
|
* fallbacks already produce real designations, so it never needs this.
|
||||||
|
*/
|
||||||
|
const DESIGNATION_PREFIXES: Readonly<Record<string, string>> = {
|
||||||
|
gaia: 'Gaia DR3'
|
||||||
|
};
|
||||||
|
|
||||||
interface StarMetaColumns {
|
interface StarMetaColumns {
|
||||||
ids: Int32Array;
|
ids: Int32Array;
|
||||||
magnitudes: Float32Array;
|
magnitudes: Float32Array;
|
||||||
@@ -85,13 +115,29 @@ export function encodeStarCatalog(stars: readonly StarRecord[]): {
|
|||||||
const spectralTypes: string[] = [];
|
const spectralTypes: string[] = [];
|
||||||
const spectralTypeIds = new Map<string, number>();
|
const spectralTypeIds = new Map<string, number>();
|
||||||
const names: string[] = [];
|
const names: string[] = [];
|
||||||
|
const sources: { id: string; designationPrefix: string }[] = [];
|
||||||
|
const sourceIds = new Map<string, number>();
|
||||||
|
const sourceIndices: number[] = [];
|
||||||
|
|
||||||
stars.forEach((star, index) => {
|
stars.forEach((star, index) => {
|
||||||
positions[index * 3] = star.x;
|
positions[index * 3] = star.x;
|
||||||
positions[index * 3 + 1] = star.y;
|
positions[index * 3 + 1] = star.y;
|
||||||
positions[index * 3 + 2] = star.z;
|
positions[index * 3 + 2] = star.z;
|
||||||
|
|
||||||
names.push(star.name);
|
let sourceIndex = -1;
|
||||||
|
if (star.source !== undefined) {
|
||||||
|
const known = sourceIds.get(star.source);
|
||||||
|
if (known === undefined) {
|
||||||
|
sourceIndex = sources.push({ id: star.source, designationPrefix: DESIGNATION_PREFIXES[star.source] ?? star.source }) - 1;
|
||||||
|
sourceIds.set(star.source, sourceIndex);
|
||||||
|
} else {
|
||||||
|
sourceIndex = known;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
sourceIndices.push(sourceIndex);
|
||||||
|
|
||||||
|
// Left empty when it is simply the designation the source would generate anyway.
|
||||||
|
names.push(star.name === designationFor(sources[sourceIndex]?.designationPrefix, star.id) ? '' : star.name);
|
||||||
|
|
||||||
let spectralTypeId = spectralTypeIds.get(star.spectralType);
|
let spectralTypeId = spectralTypeIds.get(star.spectralType);
|
||||||
if (spectralTypeId === undefined) {
|
if (spectralTypeId === undefined) {
|
||||||
@@ -105,7 +151,14 @@ export function encodeStarCatalog(stars: readonly StarRecord[]): {
|
|||||||
columns.spectralTypeIndices[index] = spectralTypeId;
|
columns.spectralTypeIndices[index] = spectralTypeId;
|
||||||
});
|
});
|
||||||
|
|
||||||
return { index: { count, names, spectralTypes }, positions, meta };
|
// A per-star column is only worth writing when the stars actually differ.
|
||||||
|
const mixedSources = sources.length > 1;
|
||||||
|
return { index: { count, names, spectralTypes, sources, sourceIndices: mixedSources ? sourceIndices : [] }, positions, meta };
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The name a source gives a star it has no other name for. */
|
||||||
|
function designationFor(prefix: string | undefined, id: number): string | undefined {
|
||||||
|
return prefix === undefined ? undefined : `${prefix} ${id}`;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Rebuilds the star records the app works with from the three loaded assets. */
|
/** Rebuilds the star records the app works with from the three loaded assets. */
|
||||||
@@ -115,15 +168,20 @@ export function decodeStarCatalog(index: StarCatalogIndex, positions: Float32Arr
|
|||||||
|
|
||||||
for (let i = 0; i < index.count; i++) {
|
for (let i = 0; i < index.count; i++) {
|
||||||
const colorIndex = columns.colorIndices[i];
|
const colorIndex = columns.colorIndices[i];
|
||||||
|
const id = columns.ids[i];
|
||||||
|
const sourceIndex = index.sourceIndices.length > 0 ? index.sourceIndices[i] : index.sources.length === 1 ? 0 : -1;
|
||||||
|
const source = index.sources[sourceIndex];
|
||||||
|
|
||||||
stars[i] = {
|
stars[i] = {
|
||||||
id: columns.ids[i],
|
id,
|
||||||
name: index.names[i],
|
name: index.names[i] || designationFor(source?.designationPrefix, id) || `HYG ${id}`,
|
||||||
x: positions[i * 3],
|
x: positions[i * 3],
|
||||||
y: positions[i * 3 + 1],
|
y: positions[i * 3 + 1],
|
||||||
z: positions[i * 3 + 2],
|
z: positions[i * 3 + 2],
|
||||||
magnitude: columns.magnitudes[i],
|
magnitude: columns.magnitudes[i],
|
||||||
spectralType: index.spectralTypes[columns.spectralTypeIndices[i]],
|
spectralType: index.spectralTypes[columns.spectralTypeIndices[i]],
|
||||||
colorIndex: Number.isNaN(colorIndex) ? null : colorIndex
|
colorIndex: Number.isNaN(colorIndex) ? null : colorIndex,
|
||||||
|
...(source ? { source: source.id } : {})
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -20,6 +20,12 @@ export interface StarRecord {
|
|||||||
* `colorIndexToRgb`.
|
* `colorIndexToRgb`.
|
||||||
*/
|
*/
|
||||||
colorIndex: number | null;
|
colorIndex: number | null;
|
||||||
|
/**
|
||||||
|
* Which catalogue this star's position came from, once more than one contributes. Absent for a
|
||||||
|
* single-source build; see `star-merge.ts`, where overlapping catalogues are reconciled and
|
||||||
|
* the better-measured parallax wins.
|
||||||
|
*/
|
||||||
|
source?: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** HYG id used for the Sun itself, so solar-system bodies can reference their host star. */
|
/** HYG id used for the Sun itself, so solar-system bodies can reference their host star. */
|
||||||
|
|||||||
File diff suppressed because one or more lines are too long
@@ -9,6 +9,7 @@ import { fetchExoplanets } from './fetchExoplanets';
|
|||||||
import { fetchSolarSystem } from './fetchSolarSystem';
|
import { fetchSolarSystem } from './fetchSolarSystem';
|
||||||
import { BYTES_PER_STAR_META, BYTES_PER_STAR_POSITION, decodeStarCatalog, encodeStarCatalog } from '../../src/app/shared/models/star-catalog';
|
import { BYTES_PER_STAR_META, BYTES_PER_STAR_POSITION, decodeStarCatalog, encodeStarCatalog } from '../../src/app/shared/models/star-catalog';
|
||||||
import { fetchStars } from './fetchStars';
|
import { fetchStars } from './fetchStars';
|
||||||
|
import { describeSources } from './sources/registry';
|
||||||
import { rematchHostStars } from '../../src/app/shared/astro/host-star-matching';
|
import { rematchHostStars } from '../../src/app/shared/astro/host-star-matching';
|
||||||
import { dataPath } from './lib/paths';
|
import { dataPath } from './lib/paths';
|
||||||
|
|
||||||
@@ -147,6 +148,9 @@ function validateDeepSky(objects: DeepSkyRecord[]): void {
|
|||||||
*/
|
*/
|
||||||
async function build(): Promise<void> {
|
async function build(): Promise<void> {
|
||||||
console.log('=== NASA star map ETL ===\n');
|
console.log('=== NASA star map ETL ===\n');
|
||||||
|
console.log('Catalogues:');
|
||||||
|
console.log(describeSources());
|
||||||
|
console.log();
|
||||||
|
|
||||||
const stars = await fetchStars();
|
const stars = await fetchStars();
|
||||||
console.log();
|
console.log();
|
||||||
|
|||||||
+46
-3
@@ -1,8 +1,11 @@
|
|||||||
import { writeFileSync } from 'node:fs';
|
import { writeFileSync } from 'node:fs';
|
||||||
|
|
||||||
import { raDecDistanceToXyz } from '../../src/app/shared/astro/coordinates';
|
import { raDecDistanceToXyz } from '../../src/app/shared/astro/coordinates';
|
||||||
|
import { mergeStarCatalogues } from '../../src/app/shared/astro/star-merge';
|
||||||
import { encodeStarCatalog } from '../../src/app/shared/models/star-catalog';
|
import { encodeStarCatalog } from '../../src/app/shared/models/star-catalog';
|
||||||
import { StarRecord, SUN_STAR_ID } from '../../src/app/shared/models/star.model';
|
import { StarRecord, SUN_STAR_ID } from '../../src/app/shared/models/star.model';
|
||||||
|
import { positionalSources } from './sources/registry';
|
||||||
|
import { PARALLAX_PRECISION_MAS } from './sources/star-sources';
|
||||||
import { parseCsvObjects, parseOptionalNumber } from './lib/csv';
|
import { parseCsvObjects, parseOptionalNumber } from './lib/csv';
|
||||||
import { fetchTextCached } from './lib/http';
|
import { fetchTextCached } from './lib/http';
|
||||||
import { dataPath, ensureDataDir } from './lib/paths';
|
import { dataPath, ensureDataDir } from './lib/paths';
|
||||||
@@ -100,10 +103,50 @@ export async function fetchStars(): Promise<StarRecord[]> {
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
stars.sort((a, b) => a.id - b.id);
|
|
||||||
writeStarAssets(stars);
|
|
||||||
|
|
||||||
console.log(` kept ${stars.length} stars (of ${rows.length} in the catalog).`);
|
console.log(` kept ${stars.length} stars (of ${rows.length} in the catalog).`);
|
||||||
|
|
||||||
|
const merged = await mergeWithOtherSources(stars);
|
||||||
|
merged.sort((a, b) => a.id - b.id);
|
||||||
|
writeStarAssets(merged);
|
||||||
|
return merged;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Unions HYG with every other positional source that is wired in and reachable.
|
||||||
|
*
|
||||||
|
* A source that cannot be reached is reported and skipped rather than failing the run. That is
|
||||||
|
* not defensive padding: the archives this would draw on are frequently unavailable, and a build
|
||||||
|
* that produces a smaller catalogue is far better than one that produces none.
|
||||||
|
*/
|
||||||
|
async function mergeWithOtherSources(hygStars: StarRecord[]): Promise<StarRecord[]> {
|
||||||
|
const others = positionalSources().filter((source) => source.id !== 'hyg');
|
||||||
|
if (others.length === 0) {
|
||||||
|
return hygStars;
|
||||||
|
}
|
||||||
|
|
||||||
|
const candidates = [{ sourceId: 'hyg', parallaxPrecisionMas: PARALLAX_PRECISION_MAS['hyg'], stars: hygStars }];
|
||||||
|
|
||||||
|
for (const source of others) {
|
||||||
|
try {
|
||||||
|
candidates.push({
|
||||||
|
sourceId: source.id,
|
||||||
|
parallaxPrecisionMas: PARALLAX_PRECISION_MAS[source.id] ?? 1,
|
||||||
|
stars: await source.fetch!()
|
||||||
|
});
|
||||||
|
} catch (error) {
|
||||||
|
console.log(` skipping ${source.name}: ${error instanceof Error ? error.message : error}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (candidates.length === 1) {
|
||||||
|
return hygStars;
|
||||||
|
}
|
||||||
|
|
||||||
|
const { stars, summary } = mergeStarCatalogues(candidates);
|
||||||
|
console.log(` merged ${summary.total} stars from ${candidates.length} catalogues (${summary.duplicates} duplicates resolved to the better parallax):`);
|
||||||
|
for (const [sourceId, count] of Object.entries(summary.bySource)) {
|
||||||
|
console.log(` ${sourceId}: ${count}`);
|
||||||
|
}
|
||||||
return stars;
|
return stars;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,108 @@
|
|||||||
|
import { raDegDecDistanceToXyz } from '../../../src/app/shared/astro/coordinates';
|
||||||
|
import { StarRecord } from '../../../src/app/shared/models/star.model';
|
||||||
|
import { parseCsvObjects, parseOptionalNumber } from '../lib/csv';
|
||||||
|
import { fetchTextCached } from '../lib/http';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Gaia DR3, via the ESA archive's TAP service.
|
||||||
|
*
|
||||||
|
* The only one of the large modern surveys that can add stars to a *3D* map, because it is the
|
||||||
|
* only one that measures parallaxes for them. Its 1.8 billion sources are roughly 1% of the
|
||||||
|
* Galaxy — no catalogue is close to the rest — but within a few hundred parsecs it is complete
|
||||||
|
* in a way Hipparcos never was, and its parallaxes are fifty times more precise.
|
||||||
|
*
|
||||||
|
* **This has never been run.** Every ESA, NOIRLab, SDSS and Euclid endpoint is unreachable from
|
||||||
|
* the environment this was written in, so the query below is written against the published DR3
|
||||||
|
* schema and has not been executed against it. Treat the column names as the first thing to
|
||||||
|
* check if a real run misbehaves.
|
||||||
|
*/
|
||||||
|
|
||||||
|
const GAIA_TAP_URL = 'https://gea.esac.esa.int/tap-server/tap/sync';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* How far out to take Gaia, in parsecs, and the faintest star to keep.
|
||||||
|
*
|
||||||
|
* Both exist to bound the download rather than the science. Gaia's parallaxes stay useful far
|
||||||
|
* past anything this map draws, so the limit here is a payload decision: the catalogue is baked
|
||||||
|
* into a static asset that a browser downloads before the first frame.
|
||||||
|
*/
|
||||||
|
const DISTANCE_CUTOFF_PC = Number(process.env['ETL_GAIA_DISTANCE_PC'] ?? 250);
|
||||||
|
const MAGNITUDE_LIMIT = Number(process.env['ETL_GAIA_MAGNITUDE_LIMIT'] ?? 12);
|
||||||
|
const ROW_LIMIT = Number(process.env['ETL_GAIA_ROW_LIMIT'] ?? 500000);
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Relative parallax error above which a star is dropped: a parallax measured to worse than 20%
|
||||||
|
* gives a distance that is not worth plotting, and inverting a noisy parallax biases it badly.
|
||||||
|
*/
|
||||||
|
const MAX_PARALLAX_ERROR_RATIO = 0.2;
|
||||||
|
|
||||||
|
/** Parallax in milliarcseconds for a given distance — the query's cutoff, expressed as Gaia has it. */
|
||||||
|
function parallaxFloorMas(distancePc: number): number {
|
||||||
|
return 1000 / distancePc;
|
||||||
|
}
|
||||||
|
|
||||||
|
function buildQuery(): string {
|
||||||
|
return [
|
||||||
|
`select top ${ROW_LIMIT}`,
|
||||||
|
'source_id, ra, dec, parallax, parallax_error, phot_g_mean_mag, bp_rp',
|
||||||
|
'from gaiadr3.gaia_source',
|
||||||
|
`where parallax > ${parallaxFloorMas(DISTANCE_CUTOFF_PC).toFixed(6)}`,
|
||||||
|
`and parallax_over_error > ${(1 / MAX_PARALLAX_ERROR_RATIO).toFixed(1)}`,
|
||||||
|
`and phot_g_mean_mag < ${MAGNITUDE_LIMIT}`,
|
||||||
|
'order by phot_g_mean_mag asc'
|
||||||
|
].join(' ');
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Gaia publishes no spectral classifications, but `bp_rp` is a colour index on the same footing
|
||||||
|
* as HYG's `ci` — so the app's existing colour and spectral-class handling works unchanged, and
|
||||||
|
* the spectral type is left as unknown rather than invented from the colour.
|
||||||
|
*/
|
||||||
|
const UNKNOWN_SPECTRAL_TYPE = 'Unknown';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Gaia source ids are 19 digits and there are no proper names, so a star's identity here is its
|
||||||
|
* catalogue designation. The app's star ids are 32-bit, which a Gaia source id overflows, so the
|
||||||
|
* two are kept apart: `id` is assigned within this run's own range and the designation carries
|
||||||
|
* the real identifier in the name.
|
||||||
|
*/
|
||||||
|
const GAIA_ID_BASE = 1_000_000_000;
|
||||||
|
|
||||||
|
export async function fetchGaiaStars(): Promise<StarRecord[]> {
|
||||||
|
const url = `${GAIA_TAP_URL}?REQUEST=doQuery&LANG=ADQL&FORMAT=csv&QUERY=${encodeURIComponent(buildQuery())}`;
|
||||||
|
console.log(`Fetching Gaia DR3 (within ${DISTANCE_CUTOFF_PC} pc, G < ${MAGNITUDE_LIMIT}, at most ${ROW_LIMIT} rows)...`);
|
||||||
|
|
||||||
|
const csv = await fetchTextCached(url, 'gaia-dr3.csv');
|
||||||
|
const rows = parseCsvObjects(csv);
|
||||||
|
const stars: StarRecord[] = [];
|
||||||
|
|
||||||
|
rows.forEach((row, index) => {
|
||||||
|
const parallaxMas = parseOptionalNumber(row['parallax']);
|
||||||
|
const raDeg = parseOptionalNumber(row['ra']);
|
||||||
|
const decDeg = parseOptionalNumber(row['dec']);
|
||||||
|
if (!parallaxMas || parallaxMas <= 0 || raDeg === undefined || decDeg === undefined) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const distancePc = 1000 / parallaxMas;
|
||||||
|
if (distancePc > DISTANCE_CUTOFF_PC) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const { x, y, z } = raDegDecDistanceToXyz(raDeg, decDeg, distancePc);
|
||||||
|
stars.push({
|
||||||
|
id: GAIA_ID_BASE + index,
|
||||||
|
name: `Gaia DR3 ${row['source_id']}`,
|
||||||
|
x,
|
||||||
|
y,
|
||||||
|
z,
|
||||||
|
magnitude: parseOptionalNumber(row['phot_g_mean_mag']) ?? MAGNITUDE_LIMIT,
|
||||||
|
spectralType: UNKNOWN_SPECTRAL_TYPE,
|
||||||
|
colorIndex: parseOptionalNumber(row['bp_rp']) ?? null,
|
||||||
|
source: 'gaia'
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
console.log(` kept ${stars.length} Gaia stars (of ${rows.length} rows).`);
|
||||||
|
return stars;
|
||||||
|
}
|
||||||
@@ -0,0 +1,80 @@
|
|||||||
|
import { fetchGaiaStars } from './gaia';
|
||||||
|
import { StarSource } from './star-sources';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Every catalogue this pipeline knows about, wired in or not, with an honest note on each.
|
||||||
|
*
|
||||||
|
* Kept as data so `npm run etl` can print what it actually used rather than what a README claims
|
||||||
|
* it uses. The four unimplemented entries are not placeholders for missing work: three of them
|
||||||
|
* cannot contribute stars to a 3D map at all, for the reason recorded against each.
|
||||||
|
*/
|
||||||
|
export const STAR_SOURCES: readonly StarSource[] = [
|
||||||
|
{
|
||||||
|
id: 'hyg',
|
||||||
|
name: 'HYG database (Hipparcos, Yale Bright Star, Gliese)',
|
||||||
|
role: 'positional',
|
||||||
|
endpoint: 'https://raw.githubusercontent.com/astronexus/HYG-Database',
|
||||||
|
contributes: 'A complete, named, spectrally classified bright-star catalogue with parallaxes — 68388 stars within 250 pc.',
|
||||||
|
unimplementedBecause: null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: 'gaia',
|
||||||
|
name: 'Gaia DR3',
|
||||||
|
role: 'positional',
|
||||||
|
endpoint: 'https://gea.esac.esa.int/tap-server/tap/sync',
|
||||||
|
contributes:
|
||||||
|
'Parallaxes fifty times more precise than Hipparcos, for 1.8 billion sources — the only survey that can add stars to a 3D map, because it is the only one that measures how far away they are.',
|
||||||
|
unimplementedBecause: null,
|
||||||
|
fetch: fetchGaiaStars
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: 'decaps2',
|
||||||
|
name: 'DECaPS2 (Dark Energy Camera Plane Survey)',
|
||||||
|
role: 'backdrop',
|
||||||
|
endpoint: 'https://datalab.noirlab.edu/tap',
|
||||||
|
contributes:
|
||||||
|
'The deepest optical census of the southern galactic plane: 3.32 billion objects across 130 degrees, in the dust-obscured region every other catalogue thins out in.',
|
||||||
|
unimplementedBecause:
|
||||||
|
'Photometric only — no parallaxes, so not one of its 3.32 billion objects can be placed in depth. Fifty times Gaia’s object count and zero stars this map can position. It would enter as a direction-only backdrop layer, alongside the deep-sky shell.'
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: 'sdss5-mwm',
|
||||||
|
name: 'SDSS-V Milky Way Mapper',
|
||||||
|
role: 'enrichment',
|
||||||
|
endpoint: 'https://api.sdss.org',
|
||||||
|
contributes:
|
||||||
|
'All-sky optical and infrared spectroscopy: effective temperatures, surface gravities, metallicities and radial velocities, which is real spectral classification rather than a colour index standing in for one.',
|
||||||
|
unimplementedBecause:
|
||||||
|
'Adds no positions. It is keyed to targets Gaia already places, so it joins onto an existing catalogue rather than extending it — worth having once Gaia is in, and worth nothing before.'
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: 'euclid-q2-bulge',
|
||||||
|
name: 'Euclid Galactic Bulge Survey (Q2)',
|
||||||
|
role: 'backdrop',
|
||||||
|
endpoint: 'https://easidr.esac.esa.int/sas',
|
||||||
|
contributes: 'High-resolution imagery and astrometry of the crowded inner bulge, around 8 kpc away.',
|
||||||
|
unimplementedBecause:
|
||||||
|
'At 8 kpc a parallax is a few microarcseconds, so this is astrometry without usable distances for a map of this kind. Its natural use here is imagery — a real photograph of the bulge on the galactic view, in place of part of the procedural model.'
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: 'saga',
|
||||||
|
name: 'SAGA (Stellar Abundances for Galactic Archaeology)',
|
||||||
|
role: 'enrichment',
|
||||||
|
endpoint: 'https://sagadatabase.jp',
|
||||||
|
contributes: 'Compiled elemental abundances for metal-poor stars — the chemical record of how the Galaxy assembled.',
|
||||||
|
unimplementedBecause:
|
||||||
|
'A compilation keyed to stars other catalogues place, covering tens of thousands of objects rather than millions. Like SDSS-V it enriches; it cannot extend the map on its own.'
|
||||||
|
}
|
||||||
|
];
|
||||||
|
|
||||||
|
export function positionalSources(): readonly StarSource[] {
|
||||||
|
return STAR_SOURCES.filter((source) => source.role === 'positional' && source.fetch !== undefined);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** A one-line-per-source report of what the pipeline can and cannot draw on. */
|
||||||
|
export function describeSources(): string {
|
||||||
|
return STAR_SOURCES.map((source) => {
|
||||||
|
const status = source.unimplementedBecause === null ? (source.fetch ? 'wired in' : 'wired in (built separately)') : 'declared, not fetched';
|
||||||
|
return ` [${source.role}] ${source.name} — ${status}`;
|
||||||
|
}).join('\n');
|
||||||
|
}
|
||||||
@@ -0,0 +1,45 @@
|
|||||||
|
import { StarRecord } from '../../../src/app/shared/models/star.model';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The catalogues this map can draw on, and what each of them can actually contribute.
|
||||||
|
*
|
||||||
|
* The distinction that matters is not size. It is whether a catalogue knows how *far away* its
|
||||||
|
* objects are, because a 3D map cannot place a star it only has a direction for. That splits the
|
||||||
|
* available surveys into three roles which are not interchangeable, and a survey being enormous
|
||||||
|
* says nothing about which role it fills — DECaPS2 has fifty times Gaia's object count and
|
||||||
|
* cannot place a single one of them in depth.
|
||||||
|
*/
|
||||||
|
export type StarSourceRole =
|
||||||
|
/** Has a direction *and* a distance, usually from parallax. Can put a star in the scene. */
|
||||||
|
| 'positional'
|
||||||
|
/** Has stellar parameters keyed to an identifier. Adds knowledge about stars already placed. */
|
||||||
|
| 'enrichment'
|
||||||
|
/** Has directions but no usable distance. Can only be painted on the backdrop shell. */
|
||||||
|
| 'backdrop';
|
||||||
|
|
||||||
|
export interface StarSource {
|
||||||
|
readonly id: string;
|
||||||
|
readonly name: string;
|
||||||
|
readonly role: StarSourceRole;
|
||||||
|
/** Where the data comes from, for the credits and for anyone re-running the pipeline. */
|
||||||
|
readonly endpoint: string;
|
||||||
|
/** What this source adds that the others do not. */
|
||||||
|
readonly contributes: string;
|
||||||
|
/**
|
||||||
|
* Why it is not yet wired in, or `null` when it is. Kept as data rather than as a comment so
|
||||||
|
* the ETL can print an honest summary of what actually ran.
|
||||||
|
*/
|
||||||
|
readonly unimplementedBecause: string | null;
|
||||||
|
/** Fetches this source's stars. Absent for sources that are declared but not implemented. */
|
||||||
|
readonly fetch?: () => Promise<StarRecord[]>;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Precision of the parallax each positional source measures with, in milliarcseconds — which is
|
||||||
|
* what decides how far out its distances stay meaningful, and which of two catalogues to believe
|
||||||
|
* when both contain the same star.
|
||||||
|
*/
|
||||||
|
export const PARALLAX_PRECISION_MAS: Readonly<Record<string, number>> = {
|
||||||
|
hyg: 1,
|
||||||
|
gaia: 0.02
|
||||||
|
};
|
||||||
Reference in New Issue
Block a user