Why I'm Obsessed With Search

I've been building search systems for a few years now — building search over 420M+ software repositories, building distributed systems to power billion scale vector search at Qdrant, and reverse-engineering how Exa does web search.

This post will tell you why I care so much about search. My hope is that it either convinces you to work on search or helps you find your own thing to be obsessed about.

What do I mean by search here?

The word "search" gets used loosely, so let me narrow it down. Three things often get lumped together under this label:

Optimization: Any kind of optimization can be called "search," and it's related to what the Bitter Lesson talks about. Interesting, but not what I mean here.
Querying: "which rows match these exact filters?" What databases are built for.
Retrieval: "for this fuzzy intent, what results are most relevant?" This is what I mean.

Querying returns everything that matches; retrieval decides what matters most. That requires understanding intent and separating signal from noise.

Search is everywhere and shapes our lives

Search is the most embedded technology in modern life: finding information, ordering food, hunting houses, discovering people and products, writing code. Think Google, YouTube, Amazon, LinkedIn, etc.

It's everywhere, and its quality shapes what we know and what we believe is possible. And yet useful information, great products, people worth knowing are often not discovered because search isn't perfect (yet?).

Search is how curiosity becomes knowledge

I'm naturally very curious and curiosity generates questions. Search provides answers to those questions. Answers lead to knowledge when absorbed well. And eventually that knowledge leads to more questions. Search is the engine for compounding knowledge. The better the feedback loop between question and answer, the faster you learn.

Search is a tool that amplifies our cognitive capability

Humanity has flourished by building tools to overcome its biological limits: planes because we couldn't fly, submarines because we couldn't breathe underwater, photos because memory is fallible. When someone builds a great tool and gives it to the rest of society, the impact compounds.

The human brain is remarkable at pattern recognition (also a kind of search), storage, and compression — but no human can store and rank billions of documents within milliseconds. Search is a tool for the mind that makes this possible. It narrows an incomprehensibly large space down to the handful of things actually worth our attention.

How search shaped my own path

I grew up in a small town in India. When we finally got internet on our computer, Google was the first thing the computer guy showed me — and it became my gateway to a whole new world.

Without any books, courses, or mentors, I learnt programming at the age of 13 — entirely by searching. I'd type half-formed questions into Google — "how to add button in VB.net" — and land on forums like SourceForge, CodeProject, or W3Schools. I'd download source code of applications, do Ctrl+F, and figure out how things worked.

When Chrome launched, it felt like a leap from Internet Explorer — fast, clean, and search was built right into the address bar. Then came voice search, and saying "OK Google" felt like having my own Jarvis. I remember spending hours exploring everything it could do.

Looking back, search is the reason I could teach myself to code at an early age. It took me years to realize that "how does Google search actually work?" was one of my earliest curiosities — and looks like I've been thinking about the same recently.

Search is a hard problem worth solving

Search sits at the intersection of data structures, distributed systems, performance, and machine learning — some of the hardest areas in computer science. Mastering search forces you to go deep on all of them.

What makes search even more unique is the tension between signal and noise. You're not just finding matches — you're ranking them. That means modeling relevance, which means understanding intent, which means wrestling with ambiguity. "Apple" could mean the fruit, the company, or the record label — and the system has to figure it out. And ranking isn't purely technical — business decisions like sponsored results and content policies add more complexity.

Search also sits at the top of the user funnel with some of the highest query volumes in most products — so scale magnifies every mistake. This combination builds deep intuition and taste that generalizes broadly.

Search queries are a window into collective intent

Aggregate search queries are a real-time map of what humanity is curious about. What people search is what they're thinking, feeling, or worried about at any given moment. Understanding those query patterns, search journeys, and how needs evolve over time — and then zooming out to community, country, and global scale — is genuinely one of the most amazing things for me.

I love checking out trends.google.com and marveling at the insight that emerges from aggregate search signals. For example, during COVID-19, searches for "loss of smell" spiked across countries before it was even recognized as a symptom. Aggregate search data has also predicted election outcomes, box office hits, and economic downturns before official indicators caught up. Google/Perplexity, Amazon/Flipkart, Opentable/Zomato, Zillow/NoBroker, Pinterest/Instagram — every search team sits on a window into how the world thinks within their verticals.

Tony Stark Hologram Dashboard AI Tech

The Duality of Search & Generation

This is more about nature of the universe and information than just humanity. I always had the abstract thought but this blog by Han Xiao (Co-founder, Jina AI) gave me the right words.

Search is Overfitted Generation; Generation is Underfitted Search

At one extreme, search is like an overfitted generator: constrained to what already exists in an index, optimizing for fidelity to known items. At the other extreme, generation is like an underfitted search system: it can improvise and compose new outputs but may drift from grounded truth.

I find this extremely fascinating. Here's a diagram from Jina.ai that explains what I mean. You can swap ??? with either search or generation:

You can also try out lexica.art which has an input box and buttons to both search or generate!

Future of search

Search has three things to trade off: speed, quality, and cost. Quality is the hardest to measure — but it's the one that matters most. High-speed, low-cost search that returns the wrong thing just wastes the user's time more efficiently. Yet many companies optimize for cheap and deliver poor experiences as a result.

But one thing is shifting: latency tolerance. Users are increasingly willing to wait a few seconds longer for results that are actually right. That tolerance is quietly unlocking a different design space.

Here are the directions I think will define the next era of search quality:

Agentic search: Iteratively decomposing queries into sub-searches, evaluating intermediate results, and deciding what to search next based on findings. The real unlock is the number of iterations: it makes quality, cost, and latency tunable tradeoffs on a query level. This is already happening with Perplexity, Cursor, Exa, Parallel, etc and will go beyond web/code search.
Personalization at depth: Search that knows your history, mental model, and current context will outperform search that treats every query as cold-start. AI agents like Claude and OpenAI now have memory which allows them to perform better search. The gap between "search for anyone" and "search for you" is enormous.
Unified search and generation: Retrieval systems pull knowledge from external sources, while generative systems pull knowledge compressed into parameters. They have already gotten very close with the rise of RAG and agents that use search as a tool. But with continual learning, the boundary will dissolve and the system will decide where a given piece of knowledge sits across this spectrum and how to surface it.

Closing

A kid in a small town typed "how to add button in VB.net" and taught himself to code. I want to make sure the next kid's search works even better. Few problems give you that kind of impact and that's why I wanna keep building the future of search!