Night Sky Digest
New astrophysics papers every morning, summarised with citations that are checked, searchable in four languages. A retrieval system I evaluated before trusting it, and run like a production service on my own cluster.
- abstracts indexed
- 51,282
- nDCG@10, hybrid, 80 queries
- 0.72
- human audit vs labels, κ
- 0.81
- warm search, end to end
- ~0.65 s
How it works
Three nightly jobs and one API, all on k3s, deployed by Argo CD from a separate GitOps repository. The GPU work runs on my desktop over Tailscale.
Evaluated before trusted
80 queries, 60 English and 20 German, each with an intent that defines relevance. 1,971 pooled results were graded by two local models from different families; a third broke ties. I audited a random 30: 90% exact agreement, κ 0.81. A CI gate fails any change that drops recall@10 or nDCG@10 by more than two points.
The finding I didn't expect: full-text search scores zero on German and on contrarian questions, and drags hybrid search down there. That's the first experiment for M6.
| Queries | Keyword | Meaning | Hybrid |
|---|---|---|---|
| English (60) | 0.45 | 0.69 | 0.74 |
| German (20) | 0.00 | 0.67 | 0.67 |
| Contrarian (8) | 0.02 | 0.49 | 0.47 |
| All (80) | 0.34 | 0.69 | 0.72 |
Run like a service
Alerts fire on what a reader would notice, not on every hiccup. Multiwindow burn-rate rules page my phone through Telegram.
Measured from the database, so short-lived jobs need no push gateway.
28-day windows; a minimum-traffic guard so two slow requests can't page.
CloudNativePG base backups plus WAL, recovered into a throwaway cluster.
Digest-pinned bases, Trivy, Syft, keyless cosign, verified before deploys.
What broke, and what it taught me
- The first digest looked right and wasn't. 23 of 79 "new" papers were revisions from 2024: arXiv's created date belongs to the latest version. New now means announced that day with a fresh ID.
- LaTeX ate letters. The model copied \textit into its JSON, where \t is a tab. Abstracts are stripped of LaTeX before the model sees them.
- The fix was in Git, the pod kept crashing. Langfuse's web pod ran out of JavaScript heap, and the first sync was stuck retrying, so Argo CD never picked up the fix until I ended it.
- The first nightly backup died at part 19. The S3 store hit its memory limit mid-upload. WAL archiving was fine, which is why the restore drill still worked.
- An alert caught my own process. A commit pushed after its PR was merged never reached main; the "metrics missing" alert fired. No more pushes to a PR once it's ready.