Draw the whole catalogue, and build the aggregation the rest would need

Two things, one verified and one that cannot be.

The render budget is now the whole catalogue: 68388 stars, one instanced
draw call, which is what a GPU should be asked to do. The budget itself
stays, because the catalogue is meant to grow past what any machine should
draw at once — Gaia alone could contribute a million — and at that point
the selection is what keeps the field legible rather than a grey wash. A
`?stars=` override handles the machines that cannot, including the
software rasterizer the end-to-end suite runs against, whose frame rate is
two orders of magnitude below a real GPU's and which was measuring the
rasterizer rather than the app.

The aggregation is the second thing, and none of it has run. Every ESA,
NOIRLab, SDSS and Euclid endpoint is unreachable from here — only GitHub
raw is, which is why HYG and OpenNGC are the current sources. So this is
infrastructure and a Gaia query written against the published DR3 schema,
not data.

What the framework encodes is that these surveys are not interchangeable.
The distinction is not size but whether a catalogue knows how far away its
objects are, because a 3D map cannot place a star it only has a direction
for. Gaia is the only one of the five that can add stars here, because it
is the only one that measures parallaxes. DECaPS2 has fifty times Gaia's
object count and photometry alone — not one of its 3.32 billion objects
can be placed in depth. Euclid's bulge is 8 kpc away, where a parallax is
microarcseconds; its contribution would be imagery. SDSS-V and SAGA are
keyed to stars something else already places, so they enrich rather than
extend. Those roles are recorded as data the ETL prints, not as prose that
can drift.

Overlapping catalogues are reconciled on direction rather than on 3D
proximity, which is the one non-obvious part. Two surveys agree on a
star's direction to within an arcsecond and disagree on its distance by
tens of per cent, so a star at 200 pc is 50 pc from itself between
catalogues while being unmistakably the same object. Matching in 3D would
need a tolerance so loose it swallowed real neighbours. The better
parallax wins where both reach; where only one does, the star stays.

Names become dense-with-holes with a source dictionary, because a survey
catalogue has no proper names — writing "Gaia DR3 4472832130942575872"
once per star would cost 25 MB per million to repeat what two adjacent
fields already say. An empty entry costs three bytes and is regenerated on
load. The Sun needed its own case in the merge: it sits at the origin, has
no direction to compare, and appears in every catalogue.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WaySiNst4HhDXBHnMy8p5G
This commit is contained in:
Claude
2026-08-05 08:51:54 +00:00
parent 29fd92d118
commit efa9e4084a
18 changed files with 772 additions and 25 deletions
+148
View File
@@ -0,0 +1,148 @@
import { describe, expect, it } from 'vitest';
import { raDegDecDistanceToXyz } from './coordinates';
import { StarRecord } from '../models/star.model';
import { directionCosine, isSameStar, mergeStarCatalogues } from './star-merge';
/** A star at a given sky position and distance, which is how catalogues actually report them. */
function at(id: number, raDeg: number, decDeg: number, distancePc: number, overrides: Partial<StarRecord> = {}): StarRecord {
const { x, y, z } = raDegDecDistanceToXyz(raDeg, decDeg, distancePc);
return { id, name: `star-${id}`, x, y, z, magnitude: 5, spectralType: 'G2V', colorIndex: 0.6, ...overrides };
}
const HIPPARCOS = { sourceId: 'hyg', parallaxPrecisionMas: 1 };
const GAIA = { sourceId: 'gaia', parallaxPrecisionMas: 0.02 };
describe('isSameStar', () => {
it('matches two catalogues reporting the same star', () => {
expect(isSameStar(at(1, 101.28, -16.71, 2.64), at(2, 101.28, -16.71, 2.63))).toBe(true);
});
it('tolerates the distance disagreement two parallaxes actually have', () => {
// Hipparcos and Gaia routinely differ by tens of per cent at a few hundred parsecs. That
// disagreement is the reason to prefer one of them, not evidence they are different stars.
expect(isSameStar(at(1, 200, 10, 200), at(2, 200, 10, 260))).toBe(true);
});
it('does not match two different stars that happen to be at the same distance', () => {
expect(isSameStar(at(1, 200, 10, 200), at(2, 200.5, 10, 200))).toBe(false);
});
it('does not match along a line of sight when the distances genuinely conflict', () => {
// Same direction, one three times further away: a background star, not the same object.
expect(isSameStar(at(1, 200, 10, 100), at(2, 200, 10, 300))).toBe(false);
});
it('matches on direction rather than on 3D proximity', () => {
// The distinction the merge rests on. These two are 60 pc apart in space and are the same
// star; a 3D-proximity test would have to be so loose it swallowed real neighbours.
const a = at(1, 45, 20, 200);
const b = at(2, 45, 20, 260);
expect(Math.hypot(a.x - b.x, a.y - b.y, a.z - b.z)).toBeGreaterThan(50);
expect(isSameStar(a, b)).toBe(true);
});
it('treats two stars at the origin as the same, and one at the origin as unlike any other', () => {
const origin: StarRecord = { id: 0, name: 'Sol', x: 0, y: 0, z: 0, magnitude: -26.7, spectralType: 'G2V', colorIndex: 0.65 };
expect(isSameStar(origin, { ...origin, id: 1 })).toBe(true);
expect(isSameStar(origin, at(2, 45, 20, 10))).toBe(false);
});
});
describe('directionCosine', () => {
it('is one for the same direction and stays inside the domain of acos', () => {
expect(directionCosine(at(1, 45, 20, 5), at(2, 45, 20, 500))).toBeCloseTo(1, 12);
expect(Math.abs(directionCosine(at(1, 45, 20, 5), at(2, 225, -20, 5)))).toBeLessThanOrEqual(1);
});
});
describe('mergeStarCatalogues', () => {
it('keeps the better-measured catalogue where two overlap', () => {
// Gaia's parallax is fifty times more precise, so where both have a star, its position is
// Gaia's — regardless of which catalogue was passed first.
const shared = { raDeg: 101.28, decDeg: -16.71 };
const { stars, summary } = mergeStarCatalogues([
{ ...HIPPARCOS, stars: [at(1, shared.raDeg, shared.decDeg, 2.7)] },
{ ...GAIA, stars: [at(2, shared.raDeg, shared.decDeg, 2.64)] }
]);
expect(stars).toHaveLength(1);
expect(stars[0].id).toBe(2);
expect(stars[0].source).toBe('gaia');
expect(summary.duplicates).toBe(1);
});
it('keeps a star the better catalogue does not reach', () => {
// The point of merging rather than replacing: Gaia is more precise but not a superset of
// everything, and a bright star it omits should not vanish from the map.
const { stars } = mergeStarCatalogues([
{ ...HIPPARCOS, stars: [at(1, 10, 10, 100)] },
{ ...GAIA, stars: [at(2, 200, -30, 50)] }
]);
expect(stars.map((star) => star.id).sort()).toEqual([1, 2]);
expect(stars.find((star) => star.id === 1)?.source).toBe('hyg');
});
it('records where every star came from', () => {
const { stars, summary } = mergeStarCatalogues([
{ ...HIPPARCOS, stars: [at(1, 10, 10, 100), at(3, 20, 10, 100)] },
{ ...GAIA, stars: [at(2, 200, -30, 50)] }
]);
expect(summary.bySource).toEqual({ hyg: 2, gaia: 1 });
expect(new Set(stars.map((star) => star.source))).toEqual(new Set(['hyg', 'gaia']));
});
it('does not depend on the order the catalogues were given in', () => {
const shared = [at(1, 30, 5, 80)];
const better = [at(2, 30, 5, 79)];
const forwards = mergeStarCatalogues([{ ...HIPPARCOS, stars: shared }, { ...GAIA, stars: better }]);
const backwards = mergeStarCatalogues([{ ...GAIA, stars: better }, { ...HIPPARCOS, stars: shared }]);
expect(forwards.stars.map((s) => s.id)).toEqual(backwards.stars.map((s) => s.id));
});
it('leaves a star that already names its source alone', () => {
const { stars } = mergeStarCatalogues([{ ...GAIA, stars: [at(1, 10, 10, 100, { source: 'gaia-dr4' })] }]);
expect(stars[0].source).toBe('gaia-dr4');
});
it('finds duplicates that straddle a sky-grid boundary', () => {
// The bucketing is an optimisation, and an optimisation that changes the answer is a bug.
// Every one of these sits on or beside a cell edge.
for (const [raDeg, decDeg] of [
[0, 0],
[0.5, 0.5],
[359.999, -0.0001],
[180, 89.9]
]) {
const { stars } = mergeStarCatalogues([
{ ...HIPPARCOS, stars: [at(1, raDeg, decDeg, 100)] },
{ ...GAIA, stars: [at(2, raDeg, decDeg, 100)] }
]);
expect(stars).toHaveLength(1);
}
});
it('handles a single catalogue as a plain pass-through', () => {
const { stars, summary } = mergeStarCatalogues([{ ...HIPPARCOS, stars: [at(1, 10, 10, 100), at(2, 20, 20, 100)] }]);
expect(stars).toHaveLength(2);
expect(summary.duplicates).toBe(0);
});
it('handles no catalogues at all', () => {
expect(mergeStarCatalogues([]).stars).toEqual([]);
});
it('scales to catalogues large enough to matter', () => {
// The reason for the sky grid: the naive pairwise merge is quadratic, and these surveys are
// the size where that stops being an academic point.
const many = Array.from({ length: 20000 }, (_, i) => at(i, (i * 0.017) % 360, ((i * 0.031) % 160) - 80, 100));
const started = Date.now();
const { stars } = mergeStarCatalogues([{ ...HIPPARCOS, stars: many }, { ...GAIA, stars: many.map((s) => ({ ...s, id: s.id + 100000 })) }]);
expect(stars).toHaveLength(20000);
expect(Date.now() - started).toBeLessThan(10000);
});
});
+149
View File
@@ -0,0 +1,149 @@
import { StarRecord } from '../models/star.model';
/**
* Merges star catalogues that overlap.
*
* Every all-sky survey contains the bright stars, so unioning two catalogues without matching
* them first would draw Sirius twice — in slightly different places, since two instruments never
* agree exactly. The merge therefore has to decide when two rows are the same object, and which
* of them to believe.
*
* Identity is decided on the sky rather than in space. Two catalogues agree closely on a star's
* *direction* — it is an angle, measured directly — and disagree much more on its *distance*,
* which comes from a parallax with real error bars. Matching on 3D proximity would therefore
* fail exactly where the catalogues are most useful: a star at 200 pc with a 25% distance
* disagreement is 50 pc from itself, while its direction is identical to within an arcsecond.
*/
const DEG_TO_RAD = Math.PI / 180;
/** Angular separation, in degrees, below which two entries are taken to be the same star. */
export const MERGE_ANGULAR_TOLERANCE_DEG = 1 / 3600;
/**
* How far two distances may disagree, as a ratio, and still describe the same star. Generous on
* purpose: Hipparcos and Gaia routinely differ by tens of per cent at a few hundred parsecs, and
* that disagreement is the *reason* to prefer one, not evidence they are different objects.
*/
export const MERGE_DISTANCE_RATIO_TOLERANCE = 0.5;
export interface MergeCandidate {
readonly sourceId: string;
/** Lower is better — the parallax precision this source measures with, in milliarcseconds. */
readonly parallaxPrecisionMas: number;
readonly stars: readonly StarRecord[];
}
export interface MergeSummary {
readonly total: number;
/** Entries dropped because a better-measured catalogue already had that star. */
readonly duplicates: number;
readonly bySource: Readonly<Record<string, number>>;
}
/** Unit direction of a star, which is the quantity catalogues actually agree on. */
function direction(star: StarRecord): [number, number, number] {
const length = Math.hypot(star.x, star.y, star.z);
return length === 0 ? [0, 0, 0] : [star.x / length, star.y / length, star.z / length];
}
function distanceOf(star: StarRecord): number {
return Math.hypot(star.x, star.y, star.z);
}
/**
* Buckets a direction onto a coarse sky grid, so a star only has to be compared against the
* handful of entries near it rather than against every star already merged.
*
* The cell is much larger than the match tolerance, so a pair straddling a boundary would be
* missed — which is why {@link neighbouringCells} checks the adjacent cells too.
*/
const SKY_CELL_DEG = 0.5;
function cellKey(raDeg: number, decDeg: number): string {
return `${Math.floor(raDeg / SKY_CELL_DEG)}:${Math.floor(decDeg / SKY_CELL_DEG)}`;
}
function skyAngles(star: StarRecord): { raDeg: number; decDeg: number } {
const [x, y, z] = direction(star);
return { raDeg: (Math.atan2(y, x) / DEG_TO_RAD + 360) % 360, decDeg: Math.asin(Math.max(-1, Math.min(1, z))) / DEG_TO_RAD };
}
function neighbouringCells(raDeg: number, decDeg: number): string[] {
const keys: string[] = [];
for (let dRa = -1; dRa <= 1; dRa++) {
for (let dDec = -1; dDec <= 1; dDec++) {
keys.push(cellKey(raDeg + dRa * SKY_CELL_DEG, decDeg + dDec * SKY_CELL_DEG));
}
}
return keys;
}
/** Cosine of the angle between two stars' directions. */
export function directionCosine(a: StarRecord, b: StarRecord): number {
const [ax, ay, az] = direction(a);
const [bx, by, bz] = direction(b);
return Math.max(-1, Math.min(1, ax * bx + ay * by + az * bz));
}
/** Whether two entries describe the same star: same direction, and distances not in conflict. */
export function isSameStar(a: StarRecord, b: StarRecord): boolean {
const [near, far] = [distanceOf(a), distanceOf(b)].sort((p, q) => p - q);
// The Sun sits at the origin of this coordinate system and so has no direction at all, which
// the angular test below cannot speak about. Every catalogue contains it, so without this the
// merge would happily keep one Sun per source.
if (near === 0) {
return far === 0;
}
const separationDeg = Math.acos(directionCosine(a, b)) / DEG_TO_RAD;
if (separationDeg > MERGE_ANGULAR_TOLERANCE_DEG) {
return false;
}
return (far - near) / near <= MERGE_DISTANCE_RATIO_TOLERANCE;
}
/**
* Unions the given catalogues, keeping one entry per star.
*
* Sources are taken in order of how precisely they measure parallax, best first, and a star is
* only added if no better-measured catalogue already has it. So where Gaia and Hipparcos
* overlap, the position is Gaia's; where only Hipparcos reaches, the star is still there.
*/
export function mergeStarCatalogues(candidates: readonly MergeCandidate[]): { stars: StarRecord[]; summary: MergeSummary } {
const ordered = [...candidates].sort((a, b) => a.parallaxPrecisionMas - b.parallaxPrecisionMas);
const merged: StarRecord[] = [];
const grid = new Map<string, StarRecord[]>();
const bySource: Record<string, number> = {};
let duplicates = 0;
for (const candidate of ordered) {
bySource[candidate.sourceId] = 0;
for (const star of candidate.stars) {
const { raDeg, decDeg } = skyAngles(star);
const alreadyPresent = neighbouringCells(raDeg, decDeg).some((key) => (grid.get(key) ?? []).some((existing) => isSameStar(existing, star)));
if (alreadyPresent) {
duplicates++;
continue;
}
const withSource: StarRecord = { ...star, source: star.source ?? candidate.sourceId };
merged.push(withSource);
bySource[candidate.sourceId]++;
const key = cellKey(raDeg, decDeg);
const cell = grid.get(key);
if (cell) {
cell.push(withSource);
} else {
grid.set(key, [withSource]);
}
}
}
return { stars: merged, summary: { total: merged.length, duplicates, bySource } };
}
+42 -1
View File
@@ -54,11 +54,15 @@ describe('encodeStarCatalog / decodeStarCatalog', () => {
});
it('keeps the index free of anything that is not a string, since the numbers are elsewhere', () => {
expect(Object.keys(encoded.index).sort()).toEqual(['count', 'names', 'spectralTypes']);
expect(encoded.index.count).toBe(STARS.length);
expect(encoded.index.names).toEqual(STARS.map((star) => star.name));
});
it('writes no per-star source column when every star came from the same place', () => {
// It would be a couple of hundred kilobytes to say nothing.
expect(encoded.index.sourceIndices).toEqual([]);
});
it('is smaller than the array of objects it replaced', () => {
// The whole reason for the format: the old encoding repeated eight key names per star.
const asObjects = JSON.stringify(STARS).length;
@@ -91,3 +95,40 @@ describe('encodeStarCatalog / decodeStarCatalog', () => {
expect(round[4999].id).toBe(4999);
});
});
describe('star catalogue provenance and derived names', () => {
const MIXED: StarRecord[] = [
{ id: 5, name: 'Sirius', x: 1, y: 0, z: 0, magnitude: -1.4, spectralType: 'A0', colorIndex: 0.0, source: 'hyg' },
// A survey star with no name of its own: what it is called is its catalogue designation.
{ id: 900, name: 'Gaia DR3 900', x: 0, y: 2, z: 0, magnitude: 11, spectralType: 'Unknown', colorIndex: 1.2, source: 'gaia' },
{ id: 901, name: 'Gaia DR3 901', x: 0, y: 0, z: 3, magnitude: 11.5, spectralType: 'Unknown', colorIndex: 1.3, source: 'gaia' }
];
const encoded = encodeStarCatalog(MIXED);
const decoded = decodeStarCatalog(encoded.index, encoded.positions, encoded.meta);
it('stores nothing for a name that is just the catalogue designation', () => {
// 25 bytes per star, per million stars, to repeat what two adjacent fields already say.
expect(encoded.index.names).toEqual(['Sirius', '', '']);
});
it('regenerates those names exactly on the way back', () => {
expect(decoded.map((star) => star.name)).toEqual(['Sirius', 'Gaia DR3 900', 'Gaia DR3 901']);
});
it('carries each star provenance through', () => {
expect(decoded.map((star) => star.source)).toEqual(['hyg', 'gaia', 'gaia']);
});
it('writes the source column only once the stars differ', () => {
expect(encoded.index.sources.map((source) => source.id)).toEqual(['hyg', 'gaia']);
expect(encoded.index.sourceIndices).toEqual([0, 1, 1]);
});
it('keeps a real name even when the star has a source that could generate one', () => {
const named = encodeStarCatalog([{ ...MIXED[1], name: 'Some Proper Name' }]);
expect(named.index.names).toEqual(['Some Proper Name']);
expect(decodeStarCatalog(named.index, named.positions, named.meta)[0].name).toBe('Some Proper Name');
});
});
+64 -6
View File
@@ -37,12 +37,42 @@ export const BYTES_PER_STAR_META =
/** `stars-index.json`: everything that is a string, plus the count the columns are sized by. */
export interface StarCatalogIndex {
count: number;
/** One per star, in catalogue order. */
/**
* One per star, in catalogue order — but empty where the name is simply the star's catalogue
* designation, which is regenerated on load from the source and the id.
*
* A survey-scale catalogue has no proper names to speak of. Gaia's designations are its
* 19-digit source ids, so writing "Gaia DR3 4472832130942575872" once per star would cost
* 25 MB per million stars — more than the whole rest of the catalogue — to store a string that
* is already implied by two fields next to it. An empty entry costs three bytes.
*
* Dense with holes rather than a list of pairs, because which one is smaller depends entirely
* on the catalogue: every HYG star has a designation worth keeping, and paying an index per
* entry to say so would be 60% larger than just writing them in order.
*/
names: string[];
/** Distinct spectral classifications; the meta column holds indices into this. */
spectralTypes: string[];
/**
* Distinct source ids, in the same dictionary form as the spectral types, plus the designation
* prefix each one names its unnamed stars with.
*/
sources: { id: string; designationPrefix: string }[];
/**
* One per star: an index into `sources`. Empty when every star came from the same place, which
* would otherwise cost a couple of hundred kilobytes to say nothing.
*/
sourceIndices: number[];
}
/**
* How a source names a star that has no name of its own. `Gaia DR3 <id>` for Gaia; HYG's own
* fallbacks already produce real designations, so it never needs this.
*/
const DESIGNATION_PREFIXES: Readonly<Record<string, string>> = {
gaia: 'Gaia DR3'
};
interface StarMetaColumns {
ids: Int32Array;
magnitudes: Float32Array;
@@ -85,13 +115,29 @@ export function encodeStarCatalog(stars: readonly StarRecord[]): {
const spectralTypes: string[] = [];
const spectralTypeIds = new Map<string, number>();
const names: string[] = [];
const sources: { id: string; designationPrefix: string }[] = [];
const sourceIds = new Map<string, number>();
const sourceIndices: number[] = [];
stars.forEach((star, index) => {
positions[index * 3] = star.x;
positions[index * 3 + 1] = star.y;
positions[index * 3 + 2] = star.z;
names.push(star.name);
let sourceIndex = -1;
if (star.source !== undefined) {
const known = sourceIds.get(star.source);
if (known === undefined) {
sourceIndex = sources.push({ id: star.source, designationPrefix: DESIGNATION_PREFIXES[star.source] ?? star.source }) - 1;
sourceIds.set(star.source, sourceIndex);
} else {
sourceIndex = known;
}
}
sourceIndices.push(sourceIndex);
// Left empty when it is simply the designation the source would generate anyway.
names.push(star.name === designationFor(sources[sourceIndex]?.designationPrefix, star.id) ? '' : star.name);
let spectralTypeId = spectralTypeIds.get(star.spectralType);
if (spectralTypeId === undefined) {
@@ -105,7 +151,14 @@ export function encodeStarCatalog(stars: readonly StarRecord[]): {
columns.spectralTypeIndices[index] = spectralTypeId;
});
return { index: { count, names, spectralTypes }, positions, meta };
// A per-star column is only worth writing when the stars actually differ.
const mixedSources = sources.length > 1;
return { index: { count, names, spectralTypes, sources, sourceIndices: mixedSources ? sourceIndices : [] }, positions, meta };
}
/** The name a source gives a star it has no other name for. */
function designationFor(prefix: string | undefined, id: number): string | undefined {
return prefix === undefined ? undefined : `${prefix} ${id}`;
}
/** Rebuilds the star records the app works with from the three loaded assets. */
@@ -115,15 +168,20 @@ export function decodeStarCatalog(index: StarCatalogIndex, positions: Float32Arr
for (let i = 0; i < index.count; i++) {
const colorIndex = columns.colorIndices[i];
const id = columns.ids[i];
const sourceIndex = index.sourceIndices.length > 0 ? index.sourceIndices[i] : index.sources.length === 1 ? 0 : -1;
const source = index.sources[sourceIndex];
stars[i] = {
id: columns.ids[i],
name: index.names[i],
id,
name: index.names[i] || designationFor(source?.designationPrefix, id) || `HYG ${id}`,
x: positions[i * 3],
y: positions[i * 3 + 1],
z: positions[i * 3 + 2],
magnitude: columns.magnitudes[i],
spectralType: index.spectralTypes[columns.spectralTypeIndices[i]],
colorIndex: Number.isNaN(colorIndex) ? null : colorIndex
colorIndex: Number.isNaN(colorIndex) ? null : colorIndex,
...(source ? { source: source.id } : {})
};
}
+6
View File
@@ -20,6 +20,12 @@ export interface StarRecord {
* `colorIndexToRgb`.
*/
colorIndex: number | null;
/**
* Which catalogue this star's position came from, once more than one contributes. Absent for a
* single-source build; see `star-merge.ts`, where overlapping catalogues are reconciled and
* the better-measured parallax wins.
*/
source?: string;
}
/** HYG id used for the Sun itself, so solar-system bodies can reference their host star. */