fivenines
20/25

Guided Problem

MiniPG Build 14: Change Schema Without Losing Identity

Time
25m
Level
intermediate
Artifacts
not specified
Progress0%

Build Your Own PostgreSQL

Catalogs as the database about the database

Time
6 min
Prerequisites
Checkpoints And Crash Recovery

What You Will Learn

  • Explain how transactional catalog rows make PostgreSQL self-describing
  • Distinguish human names from stable internal object identities and dependencies
  • Trace DDL through catalog updates, locking, cache invalidation, and visibility
system catalogobject identifierdependencytransactional DDLcatalog cacheinvalidationbootstrapsystem view

Metadata is data too

The bookstore can store orders only after PostgreSQL knows that `orders` is a relation, which columns it has, which types interpret their bytes, and which roles may access it. A configuration file could describe that structure, but then schema changes would live outside transaction, recovery, and query machinery. PostgreSQL instead stores its own description in system catalogs.

Catalogs are relations. Creating a table inserts catalog rows; dropping a function changes catalog state; analysis reads catalogs to bind names and types. The central invariant is transactional self-description: metadata observed by a statement must form a catalog state permitted by transaction visibility, rather than a half-applied mixture of old and new definitions.

Names are not enough

Catalog entries need stable identities. Names can change, schemas can shift, and objects can be referenced by many other objects. Internal object identifiers let the system tell apart two tables that once shared a name at different times, or overloaded functions with the same name and different argument types. Human-readable names are part of the interface. Internal identities are what the engine trusts.

Catalog map

Rendering diagram…

Trace one schema change

A session creates `store.shipments`. PostgreSQL acquires the required logical locks, allocates object identities, inserts catalog rows for the relation and columns, records dependencies, and creates any physical storage. Other sessions must not bind queries to a partial definition. If the transaction aborts, its catalog versions are not visible as committed metadata and the structural effects are cleaned up.

After commit, cached catalog facts in other backends cannot remain indefinitely stale. Invalidation messages tell them which entries must be reconsidered, while normal transaction rules still determine when the new definition is visible. Catalog caching accelerates repeated analysis; invalidation preserves the self-description invariant as DDL changes it.

The bootstrap knot

Hard-coded metadata works until the database needs to change while it is running. A tiny database could keep table definitions in configuration files or in memory structures loaded at startup. That may be fine for a toy, but it makes transactional schema changes, permissions, dependency tracking, and introspection difficult. If metadata lives in transactional tables, a schema change can commit or roll back like any other change. Users can query system views to understand the database. Tools can inspect structure without private APIs.

The bootstrap problem is the catch. If catalogs are tables, how does the database know the definition of the catalog tables before the catalogs exist? A real system needs a bootstrap path: a small amount of compiled-in knowledge or initialization data that creates the first catalog structures. After that, the database can use its normal mechanisms. The system starts itself, then relies more and more on its own metadata.

Dependencies make change safe

Dependency tracking keeps schema evolution safe. A view may depend on a table column. An index depends on a table and operator class. A function may depend on a type. Dropping an object without accounting for its dependents can leave broken catalog references. The database must either reject the operation or cascade it deliberately. This is more than user convenience, because the executor and planner rely on catalog consistency.

Catalogs also make extensibility possible. If operators, types, functions, casts, and access methods are catalog entries, new behavior can be registered instead of hard-coded into every layer. The analyzer can resolve a user-defined operator by consulting catalogs. The planner can learn properties from metadata. The executor can call registered functions through a common interface. Catalog design decides how open the database can become without losing coherence.

Privileges live in catalog state too. Roles, memberships, grants, default privileges, and ownership rules must be durable and transactional. A query's meaning includes not only which object a name resolves to, but whether the current user may perform the requested operation. Security is therefore catalog-driven. Changing a grant changes future analysis and execution without rewriting application code.

Caching catalog lookups is essential. Query analysis and planning read metadata constantly. Fetching catalog rows from storage for every reference would be expensive. PostgreSQL keeps system caches that map object identifiers and names to catalog tuples. Those caches must be invalidated when metadata changes: fast cached reads require a correct reaction to transactional DDL.

Schema changes are transactions with structural consequences. Adding a column, creating an index, or replacing a function modifies catalog rows and may trigger physical work. Other sessions may have plans that depend on old definitions. Locks, invalidation messages, and transaction visibility determine who sees which schema version and when. Catalogs are not static dictionaries. They are live data with concurrency rules.

Information schemas and system views present catalogs in user-friendly forms. Internal catalog tables may be optimized for engine needs rather than human readability. Views can translate them into portable or convenient shapes. This separation lets the engine keep precise metadata while tools and users get stable inspection surfaces.

Metadata deserves data-grade questions

For every schema fact, ask which catalog identity owns it, which transaction version made it visible, which dependents rely on it, and which caches must forget an older answer. Treating metadata as informal configuration skips those questions. Treating it as transactional data lets PostgreSQL reuse recovery, locking, visibility, and query machinery while still presenting stable system views to tools and users.

Into extensibility

Catalogs close a loop. The database uses tables to describe tables, transactions to change metadata safely, indexes to find metadata quickly, and permissions to control metadata access. The next lesson covers the extensibility this design opens up: functions, operators, and custom behavior as ordinary parts of the database.

Once behavior is described as metadata, the engine can grow without turning every new idea into a core rewrite.

Next step

See what actually stuck.

Take the practice scenarios now.