Say which band each star was measured in, whose catalogue it is, and how sure its distance is
The readout named Hipparcos, Yale and Gliese while 378 775 of the 455 608 stars are described by Gaia DR3, printed one "Magnitude" for G and V alike, and gave every distance to the parsec. The band cannot be read off the star's source, which records whose position it has: 62 002 stars Gaia places keep HYG's V and B-V, and the 575 hosts renamed after their planets are Gaia's, in G. stars-meta.bin gains two byte columns, 14 to 16 bytes a star (6 378 512 to 7 289 728 bytes; gzip -9 2 804 049 to 3 110 991). One holds the magnitude's band (V 76 555 stars, G 378 744, none 309, whose magnitude is a stand-in), which colour the colour index is (B-V 75 203, BP-RP 376 703), and whether the distance is Gaia's parallax. The other holds the distance's relative error as its square root in 255ths: a step is 0.08 % of distance at 1 %, 0.35 % at 20 %, and 100 % is the top. encode, decode, BYTES_PER_STAR_META and the build.ts round trip cover both; no workflow reads the format. Where the errors come from: - Gaia rows keep the parallax_error their query already fetched: median 0.3 %, 90th percentile 1.2 %, at most 20 %, the query's own cut. - A HYG star at Gaia's distance takes the cross-match's parallax_over_error, and a star merged into a Gaia entry keeps that entry's error with its position. - The 3 067 Hipparcos stars that keep their Hipparcos distance, Rigel, Deneb and Alnilam among them, take e_plx from van Leeuwen's 2007 reduction: a new cached query of public.hipparcos_newreduction on the ESA archive, whose 117 955 rows HYG's distances invert. - The archive's stars take sy_disterr1/2 from pscomppars, in a query and cache file of their own so the composite rows already cached were not refetched. - 439 distances have no published error: 357 Gliese rows and 82 archive hosts. Of the errors, 392 786 are 1 % or less and are not printed; 61 156 print as "117 ± 12 pc" to the distance's own digits; 1 196 between 20 and 100 %, and 30 past it, print as the range the parallax gives, since a symmetric error in parallax is a lopsided one in distance. The star card (measured on the dev server) now reads, for example: - Rigel: 265 ± 23 pc, V 0.18, B-V -0.03, source HYG. - Deneb: 433 ± 60 pc. - Alnilam: "476 pc to 833 pc" (Hipparcos 1.65 ± 0.45 mas). - Gaia DR3 5612323414549657984: 111 ± 2 pc, G 4.63, BP-RP -0.15, source Gaia DR3. - Proxima Centauri: 1.30 pc, V 11.01, source "HYG, Gaia DR3 distance". - TRAPPIST-1: G 15.62, BP-RP 4.90, source Gaia DR3. - Kepler-186: V 15.14, source NASA Exoplanet Archive. The neighbourhood's subtitle reads "Gaia DR3 378,775 · HYG 73,556 · NASA Exoplanet Archive 3,277", counted by the catalogue describing each star. build.ts validateStars now fails a catalogue with more than 1 000 stars without a band (309 today) or without a distance error (439). Dropping G from the Gaia rows gave 379 040 without a band, and dropping their parallax_error gave 441 216 without an error; both runs failed. Decoding the catalogue in Node took a median 29 ms before and 24 ms after (nine runs each, within noise). In the app, five cold boots gave a 654-786 ms long task after the data landed and the HUD at 1.83-2.07 s. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5.5
@@ -97,6 +97,37 @@ describe('encodeStarCatalog / decodeStarCatalog', () => {
|
||||
});
|
||||
|
||||
|
||||
describe('the photometry and distance error columns', () => {
|
||||
const MEASURED: StarRecord[] = [
|
||||
{ id: 1, name: 'Sirius', x: 1, y: 0, z: 0, magnitude: -1.44, magnitudeBand: 'V', spectralType: 'A0m', colorIndex: 0.009, colorSystem: 'B-V', distanceError: 0.0036, source: 'hyg' },
|
||||
{ id: 2, name: 'Gaia DR3 2', x: 0, y: 117, z: 0, magnitude: 11.2, magnitudeBand: 'G', spectralType: 'Unknown', colorIndex: 1.43, colorSystem: 'BP-RP', distanceError: 0.199, distanceFromGaia: true, source: 'gaia' },
|
||||
// A HYG star at Gaia's distance, and one no survey gave a magnitude, a colour or an error.
|
||||
{ id: 3, name: 'HD 3', x: 0, y: 0, z: 300, magnitude: 7, magnitudeBand: 'V', spectralType: 'K0', colorIndex: 1.0, colorSystem: 'B-V', distanceError: 0.000001, distanceFromGaia: true, source: 'hyg' },
|
||||
{ id: 4, name: 'Gaia DR3 4', x: 5, y: 5, z: 0, magnitude: 12, spectralType: 'Unknown', colorIndex: null, distanceFromGaia: true, source: 'gaia' },
|
||||
// A Hipparcos parallax smaller than its own error.
|
||||
{ id: 5, name: 'HIP 5', x: 0, y: 200, z: 0, magnitude: 6, magnitudeBand: 'V', spectralType: 'B8', colorIndex: -0.1, colorSystem: 'B-V', distanceError: 1.4, source: 'hyg' }
|
||||
];
|
||||
const encoded = encodeStarCatalog(MEASURED);
|
||||
const decoded = decodeStarCatalog(encoded.index, encoded.positions, encoded.meta);
|
||||
|
||||
it('carries the band, which colour the colour index is, and whose parallax the distance is', () => {
|
||||
expect(decoded.map((star) => star.magnitudeBand)).toEqual(['V', 'G', 'V', undefined, 'V']);
|
||||
expect(decoded.map((star) => star.colorSystem)).toEqual(['B-V', 'BP-RP', 'B-V', undefined, 'B-V']);
|
||||
expect(decoded.map((star) => star.distanceFromGaia)).toEqual([false, true, true, true, false]);
|
||||
});
|
||||
|
||||
it('keeps a distance error to within a step at both ends of its range, and none as none', () => {
|
||||
// A step is 0.05 % of distance at Sirius's 0.36 %, and 0.35 % at the 20 % Gaia's cut allows.
|
||||
expect(Math.abs(decoded[0].distanceError! - 0.0036)).toBeLessThan(0.0003);
|
||||
expect(Math.abs(decoded[1].distanceError! - 0.199)).toBeLessThan(0.002);
|
||||
// Too small to round to a step, but published, so not read back as unpublished.
|
||||
expect(decoded[2].distanceError).toBeGreaterThan(0);
|
||||
expect(decoded[3].distanceError).toBeUndefined();
|
||||
// Past the parallax itself there is no upper bound on the distance, which is what 100 % says.
|
||||
expect(decoded[4].distanceError).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('star catalogue provenance and derived names', () => {
|
||||
const MIXED: StarRecord[] = [
|
||||
{ id: 5, name: 'Sirius', x: 1, y: 0, z: 0, magnitude: -1.4, spectralType: 'A0', colorIndex: 0.0, source: 'hyg' },
|
||||
|
||||
@@ -26,13 +26,38 @@ export const STAR_POSITION_COMPONENTS = 3;
|
||||
export const BYTES_PER_STAR_POSITION = STAR_POSITION_COMPONENTS * Float32Array.BYTES_PER_ELEMENT;
|
||||
|
||||
/**
|
||||
* Columns in `stars-meta.bin`, in order: catalogue id, apparent magnitude, colour index, and an
|
||||
* index into the spectral-type dictionary. Stored column by column rather than record by record
|
||||
* so each one is a single typed-array view over the buffer, with no per-record stride or
|
||||
* alignment padding.
|
||||
* Columns in `stars-meta.bin`, in order: catalogue id, apparent magnitude, colour index, an
|
||||
* index into the spectral-type dictionary, what those were measured in (see {@link PHOTOMETRY}),
|
||||
* and the distance's relative error (see {@link DISTANCE_ERROR_STEPS}). Stored column by column
|
||||
* rather than record by record so each one is a single typed-array view over the buffer, with no
|
||||
* per-record stride or alignment padding.
|
||||
*/
|
||||
export const BYTES_PER_STAR_META =
|
||||
Int32Array.BYTES_PER_ELEMENT + Float32Array.BYTES_PER_ELEMENT + Float32Array.BYTES_PER_ELEMENT + Uint16Array.BYTES_PER_ELEMENT;
|
||||
Int32Array.BYTES_PER_ELEMENT +
|
||||
Float32Array.BYTES_PER_ELEMENT +
|
||||
Float32Array.BYTES_PER_ELEMENT +
|
||||
Uint16Array.BYTES_PER_ELEMENT +
|
||||
Uint8Array.BYTES_PER_ELEMENT +
|
||||
Uint8Array.BYTES_PER_ELEMENT;
|
||||
|
||||
/**
|
||||
* Bits of the photometry column. The band takes two: none (a stand-in magnitude), V or G. None
|
||||
* of it follows from the source, which records where the *position* came from: 62 002 stars
|
||||
* Gaia places keep HYG's V and B−V, and the archive's stars in G have a B−V from their
|
||||
* temperature. The last bit says whose parallax the distance is, which for a HYG star Gaia did
|
||||
* not place can still be Gaia's.
|
||||
*/
|
||||
const PHOTOMETRY = { bandV: 1, bandG: 2, bandMask: 3, colorBpRp: 4, distanceFromGaia: 8 } as const;
|
||||
|
||||
/**
|
||||
* The distance error column holds the square root of the relative error, in 255ths, and 0 where
|
||||
* none was published. The errors span three orders of magnitude — Gaia's are a median 0.3 % and
|
||||
* at most 20 %, the cut its queries make, while a Hipparcos parallax the map keeps for a bright
|
||||
* star can be as large as itself — and the square root keeps a step small at each end: 0.08 % of
|
||||
* distance at 1 %, 0.35 % at 20 %, under 1 % at 100 %. Anything past 100 % is stored as that,
|
||||
* where it no longer bounds the distance from above.
|
||||
*/
|
||||
const DISTANCE_ERROR_STEPS = 255;
|
||||
|
||||
/** `stars-index.json`: everything that is a string, plus the count the columns are sized by. */
|
||||
export interface StarCatalogIndex {
|
||||
@@ -81,6 +106,8 @@ interface StarMetaColumns {
|
||||
magnitudes: Float32Array;
|
||||
colorIndices: Float32Array;
|
||||
spectralTypeIndices: Uint16Array;
|
||||
photometry: Uint8Array;
|
||||
distanceErrors: Uint8Array;
|
||||
}
|
||||
|
||||
/** Lays typed-array views over the meta buffer at the offsets the format defines. */
|
||||
@@ -93,8 +120,12 @@ function metaColumns(buffer: ArrayBuffer, count: number): StarMetaColumns {
|
||||
const colorIndices = new Float32Array(buffer, offset, count);
|
||||
offset += count * Float32Array.BYTES_PER_ELEMENT;
|
||||
const spectralTypeIndices = new Uint16Array(buffer, offset, count);
|
||||
offset += count * Uint16Array.BYTES_PER_ELEMENT;
|
||||
const photometry = new Uint8Array(buffer, offset, count);
|
||||
offset += count * Uint8Array.BYTES_PER_ELEMENT;
|
||||
const distanceErrors = new Uint8Array(buffer, offset, count);
|
||||
|
||||
return { ids, magnitudes, colorIndices, spectralTypeIndices };
|
||||
return { ids, magnitudes, colorIndices, spectralTypeIndices, photometry, distanceErrors };
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -152,6 +183,13 @@ export function encodeStarCatalog(stars: readonly StarRecord[]): {
|
||||
columns.magnitudes[index] = star.magnitude;
|
||||
columns.colorIndices[index] = star.colorIndex ?? Number.NaN;
|
||||
columns.spectralTypeIndices[index] = spectralTypeId;
|
||||
columns.photometry[index] =
|
||||
(star.magnitudeBand === 'V' ? PHOTOMETRY.bandV : star.magnitudeBand === 'G' ? PHOTOMETRY.bandG : 0) |
|
||||
(star.colorSystem === 'BP-RP' ? PHOTOMETRY.colorBpRp : 0) |
|
||||
(star.distanceFromGaia ? PHOTOMETRY.distanceFromGaia : 0);
|
||||
// At least one step, so an error too small to round to one is not read back as none published.
|
||||
columns.distanceErrors[index] =
|
||||
star.distanceError === undefined ? 0 : Math.max(1, Math.round(Math.sqrt(Math.min(1, star.distanceError)) * DISTANCE_ERROR_STEPS));
|
||||
});
|
||||
|
||||
// A per-star column is only worth writing when the stars actually differ.
|
||||
@@ -186,6 +224,9 @@ export function decodeStarCatalog(index: StarCatalogIndex, positions: Float32Arr
|
||||
const id = columns.ids[i];
|
||||
const sourceIndex = index.sourceIndices.length > 0 ? index.sourceIndices[i] : index.sources.length === 1 ? 0 : -1;
|
||||
const source = index.sources[sourceIndex];
|
||||
const photometry = columns.photometry[i];
|
||||
const band = photometry & PHOTOMETRY.bandMask;
|
||||
const distanceError = columns.distanceErrors[i];
|
||||
|
||||
stars[i] = {
|
||||
id,
|
||||
@@ -196,6 +237,11 @@ export function decodeStarCatalog(index: StarCatalogIndex, positions: Float32Arr
|
||||
magnitude: columns.magnitudes[i],
|
||||
spectralType: index.spectralTypes[columns.spectralTypeIndices[i]],
|
||||
colorIndex: Number.isNaN(colorIndex) ? null : colorIndex,
|
||||
// Set on every record, if only to undefined, so that all of them have the one shape.
|
||||
magnitudeBand: band === PHOTOMETRY.bandV ? 'V' : band === PHOTOMETRY.bandG ? 'G' : undefined,
|
||||
colorSystem: Number.isNaN(colorIndex) ? undefined : photometry & PHOTOMETRY.colorBpRp ? 'BP-RP' : 'B-V',
|
||||
distanceError: distanceError === 0 ? undefined : (distanceError / DISTANCE_ERROR_STEPS) ** 2,
|
||||
distanceFromGaia: (photometry & PHOTOMETRY.distanceFromGaia) !== 0,
|
||||
...(source ? { source: source.id } : {})
|
||||
};
|
||||
}
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
/**
|
||||
* A single star from the HYG (Hipparcos/Yale/Gliese) catalog, positioned relative to the
|
||||
* A single star from Gaia DR3, HYG (Hipparcos/Yale/Gliese) or the NASA Exoplanet Archive — see
|
||||
* `source` — positioned relative to the
|
||||
* Sun in the galaxy-scale coordinate system (parsecs). The same positions are also packed
|
||||
* into a compact binary buffer (`stars.bin`, in index order) for fast bulk rendering; this
|
||||
* record format (`stars-index.json`) is used for search, labels, and lookups by id/name.
|
||||
@@ -13,13 +14,32 @@ export interface StarRecord {
|
||||
magnitude: number;
|
||||
spectralType: string;
|
||||
/**
|
||||
* B-V colour index, or `null` where the catalog has no photometry — about 10% of stars
|
||||
* within the distance cutoff. Deliberately nullable rather than defaulted: `0` is a real,
|
||||
* Colour index — B-V, or Gaia's BP-RP where `colorSystem` says so — or `null` where the
|
||||
* catalog has no photometry. Deliberately nullable rather than defaulted: `0` is a real,
|
||||
* meaningful colour index (a hot blue-white A-type star), so using it to stand for "unknown"
|
||||
* silently mis-colours those stars. Consumers resolve the gap from `spectralType`; see
|
||||
* `colorIndexToRgb`.
|
||||
*/
|
||||
colorIndex: number | null;
|
||||
/**
|
||||
* The band `magnitude` was measured in: Johnson V (HYG, and the archive where it has one) or
|
||||
* Gaia's G, which for a red dwarf reads up to three magnitudes brighter than V. Absent where no
|
||||
* survey measured the star and `magnitude` is the ETL's stand-in.
|
||||
*/
|
||||
magnitudeBand?: 'V' | 'G';
|
||||
/**
|
||||
* Which colour `colorIndex` is: Johnson B−V, or Gaia's BP−RP, which is larger for the same star
|
||||
* — 0.82 against 0.65 for a G2 dwarf like the Sun, 3.35 against 1.83 for an M5 dwarf (Pecaut &
|
||||
* Mamajek). Absent with it.
|
||||
*/
|
||||
colorSystem?: 'B-V' | 'BP-RP';
|
||||
/**
|
||||
* Relative uncertainty of the distance, σd/d: the relative error of the parallax it was
|
||||
* inverted from, which to first order is the same. Absent where none was published.
|
||||
*/
|
||||
distanceError?: number;
|
||||
/** Whether the distance is Gaia DR3's parallax, whichever catalogue describes the star. */
|
||||
distanceFromGaia?: boolean;
|
||||
/**
|
||||
* Which catalogue this star's position came from, once more than one contributes. Absent for a
|
||||
* single-source build; see `star-merge.ts`, where overlapping catalogues are reconciled and
|
||||
|
||||
Reference in New Issue
Block a user