A SQL LIKE '%database%' query forces a full table scan — it cannot use a B-tree index because the pattern starts with a wildcard. For searching natural language text across millions of documents, we need a fundamentally different data structure: the inverted index.

The Inverted Index

An inverted index maps every term to the list of document IDs containing that term (called a posting list). To answer "which documents contain the word database?", you look up the term in the index and immediately get the answer — no scanning required.

Lucene's Analyzer Pipeline

Before text enters the inverted index, it passes through an analyzer with three stages:

1. Character filters — strip HTML tags, normalize unicode, replace patterns (e.g., convert "café" → "cafe").
2. Tokenizer — split text into individual tokens. The standard tokenizer splits on whitespace and punctuation: "full-text search" → ["full", "text", "search"].
3. Token filters — transform each token:
   • Lowercase: "Database" → "database"
   • Stemming: "running" → "run", "databases" → "databas" (Porter stemmer)
   • Stop word removal: remove "the", "is", "a", "an" — they add noise, not signal
   • Synonyms: "quick" → ["quick", "fast"] so searching for either finds both

The same analyzer must be applied at index time and query time — if you stem "running" to "run" at index time but query for the literal "running", you get zero results.

Segments and Near-Real-Time Search

Lucene does not update the inverted index in place. Instead:

1. New documents are buffered in memory (the in-memory buffer).
2. Periodically (or on refresh, default every 1 second in Elasticsearch), the buffer is flushed to a new immutable segment on disk.
3. Each segment is a self-contained mini inverted index with its own dictionary and posting lists.
4. A search fans out across all segments and merges results.

Because segments are immutable, deletes are handled by a separate "delete bitmap" — the document is marked as deleted and filtered out at query time. The actual data is only physically removed during segment merging.

Segment Merging

Over time, many small segments accumulate. Lucene runs a background merge policy (similar to LSM-tree compaction) that combines multiple small segments into fewer large ones. This reduces the number of segments a search must query, improves query latency, and reclaims space from deleted documents.

PostgreSQL Full-Text Search

PostgreSQL provides built-in full-text search using the tsvector type (a sorted list of lexemes with positions) and tsquery type (a boolean query). A GIN index on a tsvector column gives inverted-index performance without leaving PostgreSQL. Example: SELECT * FROM articles WHERE to_tsvector('english', body) @@ to_tsquery('database & indexing').

Elasticsearch = Distributed Lucene

Elasticsearch wraps Lucene with distribution: it shards the index across nodes, replicates for fault tolerance, and provides a REST API. Each Elasticsearch shard is a single Lucene index. When you search, the coordinating node fans out to all shards, each shard searches its local Lucene segments, and results are merged.