Senior Index Team Engineer

About Kagi

Kagi is your companion for a better web. We're building a friendly internet where you can explore, learn, and interact with confidence - free from ads, tracking, and distractions. Kagi builds advanced, user-centric technology to empower you, not replace you. Our products include Kagi Search, Orion browser, Kagi Assistant, Kagi News, Kagi Summarizer, and Kagi Translate. Every product reflects our commitment to a surveillance-free internet, designed around the user.

About the Index Team

The Index team builds and maintains Kagi's own search index - the layer that makes our results meaningfully different from everyone else's. While most search engines stop at aggregating results from the same sources, we go further: we build, maintain, and continuously improve our own index to give Kagi users results that nobody else can offer. It's a hard technical problem involving distributed job scheduling, large-scale data pipelines, index quality engineering, and real-time monitoring - and the work directly shapes what millions of searches return.

The Role

We're looking for a Senior Engineer to work on the systems that power Kagi's own index. You'll own pieces of the full pipeline - from URL frontier and job scheduling through document ingestion, processing, and indexed retrieval - and help us make that pipeline faster, fresher, and more reliable. You'll also work on index quality: understanding what's in the index, what should be, and what shouldn't be. This is a role for someone who finds deep satisfaction in the unglamorous work that makes a search index actually good: the scheduling logic that's wrong 0.1% of the time, the deduplication that keeps coverage clean, the monitoring that catches quality regressions before users do.

What You'll Do

Embed within the Search team to ensure index improvements translate into better results for user.
Design and improve the URL frontier and crawl job scheduling system - how we decide what to index, when, and in what order
Own and evolve the indexing strategy (what we index post-crawl) and ranking implementation.
Investigate and fix index quality issues - spam, stale content, deduplication, coverage gaps

Must-Haves

5+ years of backend engineering experience in production information retrieval/search systems
Prior experience with search index infrastructure, URL frontier systems, or crawl scheduling
Deep expertise across the information retrieval stack: index design, relevance scoring and ranking optimization, near-duplicate detection, query engine architectures, and graph-based signal extraction.
Strong database skills - you're comfortable designing and optimizing complex PostgreSQL schemas and queries, including large job/task tables that need to scale
Proficiency in Python and/or Rust - our primary languages for this work
Ability to reason about large-scale data pipelines: throughput, correctness, idempotency, and operational reliability

Nice to have:

Experience with observability tooling (Prometheus, Grafana or similar)
Comfort with Redis for distributed state and queue management
Experience with distributed job scheduling or work queue systems - you understand the failure modes and edge cases
Experience operating systems where correctness and data quality are the primary constraints

If you've ever spent a week tracking down a subtle bug in a distributed job scheduler and felt proud rather than defeated at the end of it, this is the role.

Originally posted on Himalayas