By July the index had stopped being small, and the problem moved. Finding matching listings was no longer hard. Ordering them was, and I had been treating ordering as a detail of the query.
Then I measured it. In the shape I shipped first, 27.4% of results came back with none of the user's search words anywhere in the job title.
Not wrong rows exactly. Rows that mentioned the query somewhere in a 4,000-word description while the title said something else entirely. To a reader scanning a list of titles, that is indistinguishable from a broken search.
Ranking belongs in the database, not the query builder
The ordering now lives in a Postgres function. The full-text index weights the title above the description, which is the whole fix, and it comes with a trap worth naming because it cost me an afternoon.
Those weights are only ever read by ts_rank_cd. Set the weights and keep ordering by something else, and nothing happens. Rank with ts_rank_cd over an unweighted index, and nothing happens either. The weighting and the ranking function are one change spread across two files, and each half is inert on its own - which is exactly the kind of change that looks applied, passes review, and does nothing.
One thing that is deliberately absent from that function: sponsorship. There was a paid-placement term in an early version and a migration took it out. A ranking that can be bought is not a ranking, and it is the one thing a job seeker has no way to detect from the outside.
Streaming, because the honest wait is not zero
A live search across twenty boards takes as long as the slowest board you are still willing to wait for. You can hide that behind a spinner or you can show it, so search streams over SSE and reports per-provider progress as each board answers.
SSE rather than a WebSocket: progress is a one-way push, SSE is a single HTTP request that reconnects on its own, and the upgrade dance buys nothing when nothing travels upstream.
The cache and the live fan-out are also two different products, and only one of them is free. Anyone can search what the index already holds, with no account. The live fan-out needs one. Two details of that gate turned out to matter more than the gate itself: it sits on the pipeline step rather than on one entry branch, because there are three ways into a search and a check per branch is a check you will forget to add to the fourth. And every result event carries a capability flag, so a frontend deployed ahead of the backend degrades instead of guessing.
Paging that does not re-sort the world
A few cheap decisions, each of which was a bug first:
- Singleflight on identical concurrent queries. Two people searching the same thing at the same second run it once.
- Keyset paging rather than OFFSET, so page 40 costs what page 2 costs.
- Cross-scrape deduplication from page 2 onward. Page 1 is the one the reader is waiting on, and it is already deduplicated at write time.
- A 60-day age ceiling that does not delete undated rows, because dropping everything with no stated date would silently discard about 40% of the corpus.
A count that lies about itself
There are eleven filter dimensions, and every option carries a count. The subtle part is which rows a count is allowed to see.
A facet count has to apply every filter except the one it is counting. Otherwise the number beside an unselected option is zero by construction, and zero answers a question nobody asked. What a reader wants to know is what they will be left with if they click, and that is only computable by leaving that dimension out of its own filter: