Github Designing Data-intensive Applications [Firefox]

First, they used to offload read queries. The main production database (the leader) handled all writes. A constellation of read-only replicas served SELECT queries for the web interface, API calls, and analytics. This follows Kleppmann’s principle of separating read paths from write paths. However, replication introduced its own classic problem: replication lag . A user might comment on an issue (write to leader) and then immediately refresh the page, only to read from a replica that hasn’t yet applied the change. GitHub solved this with application-level logic: for a short “critical consistency” window after a write, the application forced reads to go to the leader.

This is where gh-ost (GitHub Online Schema Tool) shines. Traditional ALTER TABLE locks the table, blocking writes for minutes or hours. gh-ost instead creates a shadow table with the new schema, copies data in small chunks, and replays the binary log of writes from the original table onto the shadow table—all while the application continues running. At the final moment, it performs a near-instantaneous atomic swap of table names. This is a direct implementation of Kleppmann’s discussion of and eventual consistency . The system is in a temporary, inconsistent state (rows exist in both tables), but the application logic hides this complexity. The maintainability payoff is immense: GitHub can deploy schema changes hundreds of times per day, a velocity unthinkable in a system that required scheduled maintenance windows. Conclusion: The Eternal Trade-Offs GitHub is not a perfect system. It has suffered outages, data inconsistencies, and scaling pains. But its evolution from a single MySQL database to a global, polyglot data platform exemplifies every major idea in Designing Data-Intensive Applications . It teaches us that there is no “one true way.” Reliable systems use replication, but fight lag. Scalable systems use sharding, but lose distributed transactions. Maintainable systems evolve online, but pay the complexity of dual-writes and temporary inconsistency. github designing data-intensive applications

Second, and more radically, GitHub implemented (horizontal partitioning) using a custom middleware layer called gh-ost (GitHub Online Schema Transfers) and later, their Vitess-inspired system. They split the massive issues and pull_requests tables by repository ID. This meant that data for a single repository always lived on one shard. This is a thoughtful choice: most queries (e.g., “list all issues in this repo”) are naturally local to a shard, avoiding costly distributed joins. The downside, as Kleppmann warns, is the loss of cross-shard transactional guarantees. For example, moving an issue from one repository to another becomes a complex distributed transaction, something GitHub handles with asynchronous workflows and idempotent retries. Reliability and the Chaos of Large Scale Designing a reliable system at GitHub’s scale means accepting that components will fail—and not just servers, but also network partitions, clock skews, and software bugs. Kleppmann emphasizes that reliability is not about preventing failure, but about building systems that tolerate it. First, they used to offload read queries

GitHub’s architecture reflects this through and reconciliation . Consider the git push operation. Network requests can time out, and clients will retry. If GitHub processes the same push twice, it must not duplicate commits or corrupt the repository. By leveraging Git’s own immutable, content-addressed nature (where the same data yields the same hash), pushes are naturally idempotent. However, metadata operations are harder. When a webhook delivers a “push” event to an integration, the integration might fail. GitHub therefore implements an outbox pattern : the event is written to a persistent queue (like Kafka or their internal Resque system) before being sent. If delivery fails, the queue retries with exponential backoff, guaranteeing at-least-once delivery. The consumer, in turn, must be written to handle duplicates gracefully. GitHub solved this with application-level logic: for a

Another fascinating reliability challenge is . Git uses SHA-1 hashes to detect corruption. But what about the relational metadata? GitHub relies on fsync and database replication logs (binlogs). A deep lesson from Kleppmann is that “fault-tolerant” does not mean “correct by default.” GitHub invests heavily in offline validation jobs that scan production data for anomalies—orphaned issue comments, missing pull request associations—and alerts engineers to repair them. This is an admission that even with ACID transactions, latent bugs or hardware bit-flips can corrupt data. Maintainability: Evolving the Schema Without Downtime Perhaps the most underappreciated aspect of designing data-intensive applications is maintainability . Kleppmann argues that systems are not static; they evolve as requirements change. GitHub’s greatest technical achievement is its ability to alter the schema of a multi-terabyte, highly active MySQL database without taking the site offline.