Jobzyl went live in June searching six job boards. It searches twenty today, plus company careers boards through seven applicant tracking systems. The interesting part of that sentence is not the number. It is that nothing about the ingest changed shape to get there.
This is the post I would have wanted before starting: what the fan-out looks like, and which four decisions did the load-bearing work.
One interface, twenty-three implementations
Every source sits behind a single search() method. A board is a file, not a migration and not a branch in a scheduler. Indeed, Adzuna and Reed are the global core; Careerjet and Jooble are the workhorses; then the government APIs, which are the underrated tier - Bundesagentur fuer Arbeit, France Travail, JobTechDev in Sweden, MyCareersFuture in Singapore, USAJOBS - and then the remote-only boards. Twenty-three providers are implemented, twenty are disclosed publicly.
The fan-out is parallel with a per-provider timeout, because twenty boards in sequence is a minutes-long wait and one slow board should never be able to hold the response. That part is obvious. The part that was not obvious is the failure mode it creates.
A provider that swallows its own errors reports 100% success forever
The natural way to write a provider is defensively:
async def search(self, query):
try:
return await self._fetch(query)
except httpx.HTTPError:
return [] # looks careful. is a lie.That code never fails. It also never reports, and an aggregator that cannot tell an empty board from a broken one has no idea what it is serving. A board that has been down for a week looks exactly like a board with no matching jobs.
So swallowing is banned at the base class. A provider that raises is recorded as failed and the response says so; the only thing allowed to return an empty list is a board that genuinely answered with nothing. It is a two-line rule and it is the single most valuable decision in the ingest, because every coverage number downstream depends on it being true.
If a provider cannot fail, you are not measuring providers. You are measuring your own optimism.
Nine providers do not know what a country is
Nine of the sources ignore location entirely: they hand you the whole feed and expect you to filter it yourself. That is fine once. The scrape queue works through keyword-by-country pairs, so without a cache those nine refetched byte-identical feeds once per pair. Arbeitnow's feed alone is 1.98 MB.
The fix is a request-level HTTP cache keyed on url, params and headers, with a TTL and a 256-entry LRU. One deliberate hole in it: failures are not cached. Caching a 503 turns a thirty-second outage into a TTL-long one, and the whole point of the previous section is that a failure has to stay visible.
Company careers boards, by guessing
The seven ATS integrations - Greenhouse, Lever, Ashby, SmartRecruiters, Workable, Recruitee, Teamtailor - read a company's own careers board rather than an aggregator's copy of it. The listings are fresher and the descriptions are intact.
The catch is that no ATS publishes an index of its customers. There is no endpoint that says which companies are on Greenhouse. So discovery is slug guessing against a name list: 68.6% of 140 company names resolved in 29.8 seconds. Guessing does not scale by itself, so it prunes itself - names go out in bands of 500, and the run stops once three consecutive bands come in under a 15% yield floor. The floor is what keeps a long tail of misses from eating a budget that has better uses.
The write path failed in the least interesting way possible
PostgREST connects with an 8-second statement timeout. One upsert of 10,350 rows hit it, died, and took an entire sweep with it. Rows go through a chunked RPC now, and it chunks on two bounds rather than one: a row count and a byte count. Row count alone is not enough, because job descriptions vary enough in size that 2,000 short rows and 2,000 long ones are not the same request.
Retention has a similar shape of bug in it. Rows rotate into an archive table at 30 days, on coalesce(date_posted, created_at) rather than date_posted - about 40% of listings state no date at all, and on the naive column every one of them dodges retention forever. Nothing a user saved is ever archived, whatever its age.
Deduplication is part of the write, not a cleanup job
The same role sits on four boards under four URLs, and the reader wants one row. That needs an identity key that is stable across sources: accent-folded, punctuation-stripped, company legal suffixes dropped. It runs at write time and again on every read path, because a write-time pass can only see one batch and a duplicate that arrives tomorrow is still a duplicate.
The key is deliberately conservative. It merges nothing it cannot prove, which means it happily leaves duplicates on the table rather than merge two jobs that differ: