What you'll learn
Quick Answer
Search engines find candidate pages fast using an inverted index (word to list of documents), score how relevant each candidate is using term-frequency-based formulas like TF-IDF or its successor BM25, and, for the open web, add link-based authority signals like PageRank on top. Modern engines blend in hundreds of additional signals, but this retrieve-then-score pipeline is still the foundation.
Step One: Find Every Document That Could Match
If a search engine stored documents the obvious way — each document listing the words it contains — answering “which documents mention 'python'?” would mean scanning every single document, every single time. That's called a forward index, and it doesn't scale past a tiny collection. Every real search system instead builds an inverted index: a mapping from each word to the list of documents that contain it (often together with the positions where it appears, so phrase queries like “python list” can check that the words are adjacent, not just both present somewhere).
Building one means tokenizing each document (splitting it into words), normalizing those tokens (lowercasing, and often stemming so “running” and “ran” both index under “run”), and appending the document's ID to each token's posting list. A query then becomes a lookup against a handful of posting lists instead of a scan of the whole collection — intersect the lists for an AND query, union them for an OR query. This is the foundational structure behind essentially every full-text search system in production, from Elasticsearch and Solr down to SQLite's FTS extension.
Step Two: Score How Relevant Each Match Actually Is
The inverted index tells you which documents contain a query's words at all. It says nothing about which of those documents are actually about the query, as opposed to mentioning a word once in passing. That's a scoring problem, and the classic starting point is TF-IDF (term frequency – inverse document frequency):
tf(t, d) = (count of term t in document d) / (total terms in d)
idf(t) = ln(N / df(t)) where N = total documents, df(t) = documents containing t
tfidf(t,d) = tf(t, d) * idf(t)The intuition: a term that appears often in a specific document (high tf) is probably important to that document, but only if it isn't a term that appears in nearly every document anyway (low idf for common words like “the” or “is”). Multiplying the two together rewards words that are both frequent locally and rare globally — exactly the words that distinguish one document from the rest of the collection.
Here's that formula computed for real, over a tiny 5-document corpus, scored against the query “python data science”:
docs = {
"doc1": "python is a popular programming language for beginners",
"doc2": "java is a popular programming language for enterprise software",
"doc3": "python has simple syntax and is great for data science",
"doc4": "javascript runs in the browser and is popular for web development",
"doc5": "data science uses python and statistics extensively",
}
df = Counter()
for tokens in tokenized.values():
for term in set(tokens):
df[term] += 1
def idf(term):
return math.log(N / df[term]) if df[term] else 0.0
def tf(term, tokens):
return tokens.count(term) / len(tokens)Running it produces these document frequencies and idf values for the three query terms:
df('python') = 3 / 5 docs idf('python') = ln(5/3) = 0.5108
df('data') = 2 / 5 docs idf('data') = ln(5/2) = 0.9163
df('science') = 2 / 5 docs idf('science') = ln(5/2) = 0.9163...and this final ranking, produced by summing each document's tf-idf score across all three query terms:
#1: doc5 score=0.3348 -- 'data science uses python and statistics extensively'
#2: doc3 score=0.2343 -- 'python has simple syntax and is great for data science'
#3: doc1 score=0.0639 -- 'python is a popular programming language for beginners'
#4: doc2 score=0.0000 -- 'java is a popular programming language for enterprise software'
#5: doc4 score=0.0000 -- 'javascript runs in the browser and is popular for web development'doc5 comes out on top because “data”, “science”, and “python” together make up a larger share of its short 7-word text than they do of doc3's longer sentence, even though doc3 also contains all three terms. Before trusting that number, it's worth re-deriving doc5's score by hand rather than just trusting the script: tf for each of the three query terms in doc5 is 1/7 = 0.142857, so the total is 0.142857 × (0.5108 + 0.9163 + 0.9163) = 0.334772 — which matches the script's output to six decimal places, confirming the implementation isn't the thing that's wrong.
Why Rare Words Move the Score More Than Common Ones
Look at the idf values above: “python”, which appears in 3 of the 5 documents, gets idf = 0.51. “data” and “science”, each appearing in only 2 of the 5, get the noticeably higher idf = 0.92. The rarer a term is across the collection, the more it moves a document's score when that document does contain it — which is exactly the intuition you want. A search for “python data science” should reward documents that combine the less common terms “data” and “science” with “python”, not just documents that happen to mention the very common word “python” alone.
Push that idea to its extreme and it explains why TF-IDF handles stopwords gracefully even without a manual stopword list: a word like “is”, appearing in 4 of these 5 documents, would get idf = ln(5/4) = 0.223, and a word appearing in literally every document gets idf = ln(1) = 0, contributing nothing to any score at all. Most production systems still strip stopwords before indexing anyway, purely for efficiency — there's no point storing and scanning a posting list for “the” that touches nearly every document in the collection — but the scoring formula itself already suppresses their influence on ranking even if you don't.
TF-IDF's Modern Successor: BM25
TF-IDF has two practical weaknesses. Term frequency contributes close to linearly — a document that repeats “python” 50 times doesn't actually deserve a score anywhere near 50x higher than one that mentions it meaningfully once or twice, but plain TF-IDF doesn't cap that contribution. And it takes no direct account of document length — a long document naturally accumulates higher raw term counts than a short one on the same topic, which raw TF-IDF can reward for the wrong reason.
BM25 (Okapi BM25) is the refinement that fixes both, and it's not a niche alternative — it's the actual industry default. Lucene switched its default scoring function from TF-IDF-style scoring to BM25 in version 6 (2016), and Elasticsearch followed in version 5.0, which means it's what most Elasticsearch and Solr deployments have been ranking with by default for close to a decade. BM25 adds term frequency saturation, controlled by a parameter k1 (default 1.2), so additional occurrences of a term matter less and less rather than linearly; and document length normalization, controlled by a parameter b (default 0.75), so a document isn't penalized or rewarded purely for being longer or shorter than average. The core idea — reward terms that are frequent locally and rare globally — is unchanged from TF-IDF; BM25 just shapes that reward more realistically.
Step Three: Let the Web's Own Links Vote
TF-IDF and BM25 both score a document purely against a query's words — they have no concept of whether the document itself is any good. For the open web specifically, that's a real gap: two pages can use identical keywords with wildly different actual authority. PageRank, the algorithm Google was originally built around, tackles that by treating hyperlinks as votes: a page that's linked to by many other pages, especially other highly-ranked pages, is treated as more important.
The mental model that makes the math intuitive is the random surfer: imagine a user clicking links completely at random forever. Most of the time they follow a link on the current page, but with some probability at each step they instead “teleport” to a totally random page on the web. That probability of continuing to follow links, rather than teleporting, is the damping factor, and it's set to 0.85 in the original PageRank formulation — an 85% chance of following a link, a 15% chance of jumping elsewhere. The teleport step isn't a minor detail; without it, the random surfer can get trapped forever in a small loop of pages that only link to each other, which would make their rank artificially inflate with no way out. PageRank is computed iteratively: every page starts with an equal share of rank, each page distributes its current rank across the pages it links to, and this repeats until the numbers stop changing meaningfully.
Putting It Together: Retrieve Cheap, Rank Expensive
No real search engine runs a full, expensive scoring pass — BM25 plus link authority plus everything else — against its entire index for every query; that would be far too slow against a corpus of billions of pages. The standard pattern is retrieve-then-rerank: use the inverted index to cheaply narrow billions of documents down to a candidate set of a few thousand that actually contain the query's terms, then run progressively more expensive scoring only against that much smaller set — first something BM25-like, then, for the survivors, additional signals like link authority, freshness, and (in modern engines) machine-learned rerankers trained on click behavior.
Modern search engines layer in hundreds of additional signals beyond the ones in this post — page-load speed, mobile-friendliness, freshness, personalization, and increasingly, model-based relevance scoring that goes well beyond hand-written formulas. But the two-stage shape — cheap lexical retrieval first, expensive ranking only on the shortlist — and the underlying intuition that both TF-IDF and PageRank share — reward what's specific and reward what's vouched for — are still the foundation everything else gets built on top of.