avalon-vanguard/star-map
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Keep the row cap live for the queries that can reach it

Browse Source 
Gating it on `jobsQuery` switched it off for every override that *widens* the
query — which is the only way to fill `select top N` at all. `ETL_GAIA_MAGNITUDE_LIMIT=14`
asks for 500 000 rows, the sky holds more, and the answer is the limit rather
than the filters: exactly what the tripwire is for, and it no longer fired. It
now reads the row limit itself, so only a deliberately smaller slice is silent.

Measured with a synthetic answer of exactly 500 000 rows in the cache, under the
key the widened query hashes to: refused. With the `jobsQuery` gate back, the
same run keeps 500 000 Gaia stars and goes on to publish them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016jxMkwA2rbicdGxHosecYi
This commit is contained in:
Senrokai
2026-09-18 15:46:42 +02:00
co-authored by Claude Opus 5
parent b7f277ea04
commit 1a5785fb63
1 changed files with 4 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tools/etl/sources/gaia.ts
+4 -1
View File
@@ -132,7 +132,10 @@ export async function fetchGaiaStars(): Promise<StarRecord[]> {
`served with a 200, or the query was edited without updating DEFAULT_QUERY_ROWS; delete tools/etl/.cache/${cacheKey} once the archive answers properly` `served with a 200, or the query was edited without updating DEFAULT_QUERY_ROWS; delete tools/etl/.cache/${cacheKey} once the archive answers properly`
); );
} }
if (jobsQuery && rows.length >= ROW_LIMIT) { // Not gated on `jobsQuery` like the floor above it: the only ways to reach this cap are the
// overrides that *widen* the query, and they are exactly when it is worth saying. What it must
// not fire on is a deliberately smaller slice, where filling the limit is the whole point.
if (ROW_LIMIT >= DEFAULT_ROW_LIMIT && rows.length >= ROW_LIMIT) {
throw new GaiaAnswerError(`Gaia returned the query's own ${ROW_LIMIT}-row limit, so it is the limit deciding what the map holds; raise ETL_GAIA_ROW_LIMIT.`); throw new GaiaAnswerError(`Gaia returned the query's own ${ROW_LIMIT}-row limit, so it is the limit deciding what the map holds; raise ETL_GAIA_ROW_LIMIT.`);
} }
const stars: StarRecord[] = []; const stars: StarRecord[] = [];