System Architecture Decision-Making: Trade-offs & ADRs

1. The Problem & Why It Matters

Imagine joining a 12-person fintech startup whose checkout flow takes $900\text{ ms}$ and fails twice a week. Inspecting the repository reveals 38 microservices, three message brokers, and an eventually consistent NoSQL store holding financial balances—copied from a hyperscaler conference talk built for $5,000$ engineers. Every schema tweak requires synchronized pull requests across six repositories, ledger reconciliations drift by thousands of dollars during network blips, and nobody remembers why relational ACID transactions were abandoned because zero decision records were written.

When architectural choices follow industry fashion instead of explicit constraint math, three pathologies emerge:

1. The Distributed Monolith: Splitting coupled domains across network RPCs multiplies latency ($10\text{ ns}$ function call $\to 1\text{–}5\text{ ms}$ RPC) and partial-failure modes while preserving lockstep deployment coupling.
2. Governance Inversion: Teams spend three weeks bikeshedding reversible Type 2 (Two-Way Door) tweaks (linter rules, cache TTLs) while casually committing irreversible Type 1 (One-Way Door) storage and public API choices in Slack threads.
3. Institutional Amnesia: Without recorded context and accepted risks, successors either treat legacy quirks as sacred untouchable lore or “simplify” away load-bearing invariants and trigger severe regressions.

2. Intuitive Mental Model

Choosing a system architecture is like selecting logistics vehicles under physical constraints:

• A Modular Monolith is a modern freight train running on dedicated steel rails: one locomotive crew, unified braking, high payload efficiency, and zero mid-air collision coordination—ideal until multiple cities need independent regional routes off the main track.
• A Microservice Mesh is a fleet of 50 autonomous cargo drones: each drone launches, scales, and crashes independently, provided you pay the upfront and ongoing tax of radar telemetry (distributed tracing), air-traffic deconfliction (service mesh, mTLS, retry budgets), and parcel reassembly (sagas and outbox deduplication).

Pair this with Conway’s Law (“organizations design systems that mirror their communication structures”): a single 10-engineer team operating 40 microservices pays the full air-traffic-control tax while gaining zero organizational parallelism.

SYSTEM ARCHITECTURE BLUEPRINTDISTRIBUTED SYSTEMS

⤢Expand

ARCHITECTURAL FLOW · STEP 1 OF 3

1. Monolith & Modular Monolith

Establishes the ingress and structural boundary across Monolith & Modular Monolith: Single deployable; In-memory calls; ACID transactions Low Op Complexity →.

Architectural Invariant: Zero network tax Sweet Spot for Most →.

ARCHITECTURAL FLOW · STEP 2 OF 3

2. Type 1: One-Way Door & Type 2: Two-Way Door

Coordinates core data-plane execution and state transitions through Type 1: One-Way Door & Type 2: Two-Way Door: Primary Database Selection; Multi-Region Data Topology High Deliberation Required; Caching Eviction Policy.

Architectural Invariant: Captures institutional memory and prevents repeated debates.

ARCHITECTURAL FLOW · STEP 3 OF 3

3. Type 1: One-Way Door & Type 2: Two-Way Door

Enforces durability, state convergence, and output verification across Type 1: One-Way Door & Type 2: Two-Way Door: Primary Database Selection; Multi-Region Data Topology High Deliberation Required; Caching Eviction Policy.

Architectural Invariant: Captures institutional memory and prevents repeated debates.

3. Architectural Mechanisms, Topologies & PACELC Trade-offs

3.1 The Service Evolution Spectrum & Comparative Trade-off Matrix

3.2 Beyond CAP: Mapping Data Decisions with PACELC

Brewer’s CAP Theorem proves that during a network Partition, a distributed datastore must trade Availability against linearizable Consistency. However, partitions are rare; Daniel Abadi’s PACELC formulation captures the everyday latency tax:

If there is a Partition, how does the system trade Availability vs. Consistency; Else (normal operation), how does it trade Latency vs. Consistency?

• PC/EC (CockroachDB, Google Spanner, etcd, TiDB): Rejects writes on minority partitions to prevent split-brain (PC), and even during healthy operation (E) pays synchronous Raft/Paxos quorum RTT ($1\text{–}5\text{ ms}$ multi-AZ, $50\text{–}150\text{ ms}$ multi-region) to preserve strong Consistency.
• PA/EL (Apache Cassandra / DynamoDB eventual reads): Accepts writes/reads on isolated replicas during partitions (PA) and returns fast local responses with minimal Latency during normal operation (EL), shifting conflict resolution (LWW / vector clocks / CRDTs) to the application.

4. Formal Mathematical Derivations

4.1 Constrained Multi-Attribute Utility Scoring (MADA)

When comparing $K$ candidate architectures across $M$ quality attributes (e.g., ACID Integrity, Operational Simplicity, Write Scalability, Tail Latency), assign normalized weights $w_j\in [0,1]$ satisfying ${\sum }_{j=1}^M{w}_j=1$, and evaluate each candidate $i$ on a bounded rubric $s_{i,j}\in [1,5]$.

Because no amount of secondary convenience can compensate for violating a non-negotiable hard invariant (e.g., losing double-entry ledger ACID atomicity or exceeding a hard regulatory residency rule), multiply the linear utility sum by an indicator barrier $\Phi (C_i)\in \{0,1\}$:

Worked Verification & Crossover Sensitivity: Suppose a payment ledger requires $5,000\text{ peak write TPS}$ with strict ACID serializability ($\Phi =1$ requires native ACID), weighted as $w_{\text{acid}}=0.35,w_{\text{ops}}=0.25,w_{\text{scale}}=0.20,w_{\text{latency}}=0.20$:

1. Modular Monolith + Partitioned PostgreSQL ($\Phi =1$, scores $\{5.0,4.5,3.0,4.0\}$): $$Q_{\text{PG}}=1\cdot (0.35\times 5.0+0.25\times 4.5+0.20\times 3.0+0.20\times 4.0)=1.750+1.125+0.600+0.800=4.275$$
2. Distributed SQL — CockroachDB ($\Phi =1$, scores $\{5.0,3.0,4.5,3.0\}$): $$Q_{\text{CRDB}}=1\cdot (0.35\times 5.0+0.25\times 3.0+0.20\times 4.5+0.20\times 3.0)=1.750+0.750+0.900+0.600=4.000$$
3. Microservices + Apache Cassandra ($\Phi =0$ due to lacking multi-key ACID transactions; raw sum $=2.725$): $$Q_{\text{Cass}}=0\times 2.725=0.000\text{ (Disqualified)}$$

Sensitivity Crossover: At $5,000\text{ TPS}$, Partitioned PostgreSQL wins by $+0.275$ because lower operational burden and single-hop commit latency outweigh CockroachDB’s horizontal write headroom. Once business growth projects sustained writes $>30,000\text{ TPS}$ (or active-active multi-region writes), single-primary PostgreSQL fails the hard throughput constraint (${\Phi }_{\text{PG}}\to 0$), cleanly triggering a planned migration to sharded Postgres or Distributed SQL!

4.2 Serial RPC Latency & Failure Compounding Tax

Decomposing an in-process call chain into $H$ synchronous microservice hops imposes both an additive serialization/network latency tax and a multiplicative failure risk:

At $1.5\text{ ms}$ RPC overhead per hop and $A_h=99.95\%$, a $12\text{-hop}$ synchronous chain adds $18\text{ ms}$ of pure network/serialization floor before touching a database and burns $1-0.9995^{12}\approx 0.60\%$ unavailability ($52.5\text{ hours/year}$)!

5. Architecture Decision Record (ADR) Standard Specification

Commit every Type 1 architectural decision alongside source code in docs/adr/NNNN-short-imperative-title.md (Michael Nygard format + explicit Re-evaluation Triggers):

# ADR-0004: Modular Monolith with Partitioned PostgreSQL 16 for Core Ledger

## Status
Accepted (2026-11-01) — Supersedes draft RFC-019

## Context & Problem Statement
Our payment clearing ledger must sustain 5,000 peak write TPS (p99 < 40 ms) with zero balance
anomalies under concurrent transfers. Our backend team has 8 engineers operating on-call.

## Decision Drivers (Hard Invariants & Weights)
- [Hard Invariant] Strict ACID serializable double-entry transfers (RPO = 0).
- [Hard Invariant] Sustain >= 5,000 peak write TPS with Multi-AZ < 30 s RTO.
- Minimize 24/7 operational paging complexity for an 8-engineer team.

## Considered Options
1. Modular Monolith + PostgreSQL 16 (hash-partitioned ledger tables, sync Multi-AZ standby).
2. 14 Microservices + Apache Kafka Saga Choreography + Apache Cassandra.
3. Modular Services + 5-node Multi-AZ CockroachDB cluster.

## Decision Outcome
Chosen Option: **Option 1 (Modular Monolith + PostgreSQL 16)**.

### Positive Consequences
- Single-node serializable transactions eliminate cross-service saga drift and compensation bugs.
- Zero inter-module RPC serialization tax; single CI/CD artifact and unified stack traces.

### Negative Consequences, Accepted Risks & Re-evaluation Triggers
- Single-primary write ceiling (~20k-25k TPS on 64-vCPU NVMe instance).
- Automated Patroni failover incurs a 10-25 second write pause during primary crash.
- **Quantitative Re-evaluation Trigger**: Re-open ADR for functional sharding or CockroachDB
  when peak primary write CPU exceeds 55% or sustained write throughput reaches 12,000 TPS.

6. Real-World Architectural Studies & Failure Post-Mortems

6.1 Shopify: Enforcing the Modular Monolith with Packwerk

Shopify processes tens of millions of requests per minute and multi-billion-dollar Black Friday flash sales through a core Ruby on Rails modular monolith. Instead of premature network decomposition, Shopify built Packwerk to enforce static package privacy and explicit dependency graphs at CI time—retaining local ACID semantics and zero-RPC latency while scaling horizontal stateless workers and sharded MySQL pods independently.

6.2 Amazon: Two-Pizza Teams & Mandate-Driven Service Boundaries

Amazon’s migration from its monolithic Obidos C++ binary to Service-Oriented Architecture succeeded because it solved an organizational coordination bottleneck across hundreds of teams: every service boundary corresponded to an autonomous “Two-Pizza Team” owning its private datastore, versioned external contract, and 24/7 error budget.

6.3 Failure Mode War Story: The 45-Service Distributed Monolith Blackout

• Root Cause: A dispatch startup split 18 engineers across 45 microservices that all synchronously chained Rider -> Pricing -> Driver -> Geofence and ultimately queried one shared CoreDB. An unindexed query in Geofence saturated CoreDB connections, stalling upstream HTTP worker threads across all 45 services simultaneously for 4 hours.
• Recovery Pattern: Consolidated the 6 chatty dispatch services into a single Dispatch Modular Monolith with isolated DB connection bulkheads, cutting $p99$ dispatch latency from $185\text{ ms}$ to $22\text{ ms}$ and eliminating cascading cross-service retry storms.

7. Production-Grade TypeScript Simulation: Multi-Attribute ADR Scoring Engine

/**
 * Deterministic Multi-Attribute Architecture Decision Engine (MADA + Hard Invariant Barrier)
 */

export interface EvaluationCriterion {
  id: string;
  name: string;
  weight: number; // Must sum to 1.0 across all criteria
}

export interface CandidateArchitecture {
  id: string;
  name: string;
  scores: Record<string, number>; // 1.0 (Worst) to 5.0 (Best)
  hardConstraintsPassed: boolean;
  disqualificationReason?: string;
}

export interface EvaluationResult {
  rank: number;
  candidateId: string;
  candidateName: string;
  rawUtility: number;
  qualifiedScore: number;
  qualified: boolean;
  disqualificationReason?: string;
}

export class ArchitectureDecisionEngine {
  private readonly criteria: EvaluationCriterion[];

  constructor(criteria: EvaluationCriterion[]) {
    const totalWeight = criteria.reduce((sum, c) => sum + c.weight, 0);
    if (Math.abs(totalWeight - 1.0) > 1e-6) {
      throw new Error(`Criteria weights must sum to 1.0, received ${totalWeight}`);
    }
    this.criteria = criteria;
  }

  public evaluate(candidates: CandidateArchitecture[]): EvaluationResult[] {
    const evaluated = candidates.map((candidate) => {
      let weightedSum = 0;
      for (const criterion of this.criteria) {
        const score = candidate.scores[criterion.id] ?? 1;
        weightedSum += score * criterion.weight;
      }

      const rawUtility = Number(weightedSum.toFixed(3));
      const qualifiedScore = candidate.hardConstraintsPassed ? rawUtility : 0;

      return {
        rank: 0,
        candidateId: candidate.id,
        candidateName: candidate.name,
        rawUtility,
        qualifiedScore,
        qualified: candidate.hardConstraintsPassed,
        disqualificationReason: candidate.disqualificationReason,
      };
    });

    evaluated.sort((a, b) => b.qualifiedScore - a.qualifiedScore || b.rawUtility - a.rawUtility);
    return evaluated.map((item, idx) => ({ ...item, rank: idx + 1 }));
  }
}

export function runArchitectureEvaluation(): void {
  const criteria: EvaluationCriterion[] = [
    { id: 'acid', name: 'Strict ACID Invariants', weight: 0.35 },
    { id: 'ops', name: 'Low Operational Overhead', weight: 0.25 },
    { id: 'scale', name: 'Horizontal Write Headroom', weight: 0.20 },
    { id: 'latency', name: 'Commit Tail Latency', weight: 0.20 },
  ];

  const engine = new ArchitectureDecisionEngine(criteria);

  const candidates: CandidateArchitecture[] = [
    {
      id: 'modular-pg',
      name: 'Modular Monolith + Partitioned PostgreSQL',
      scores: { acid: 5.0, ops: 4.5, scale: 3.0, latency: 4.0 },
      hardConstraintsPassed: true,
    },
    {
      id: 'cockroach-sql',
      name: 'Modular Services + CockroachDB (PC/EC)',
      scores: { acid: 5.0, ops: 3.0, scale: 4.5, latency: 3.0 },
      hardConstraintsPassed: true,
    },
    {
      id: 'micro-cassandra',
      name: 'Microservices + Apache Cassandra (PA/EL)',
      scores: { acid: 1.5, ops: 2.0, scale: 5.0, latency: 3.5 },
      hardConstraintsPassed: false,
      disqualificationReason: 'Fails multi-account serializable ACID ledger invariant',
    },
  ];

  const ranked = engine.evaluate(candidates);
  console.log('=== ADR Multi-Attribute Decision Matrix (5,000 TPS Payment Ledger) ===');
  for (const r of ranked) {
    const status = r.qualified ? 'QUALIFIED' : `DISQUALIFIED (${r.disqualificationReason})`;
    console.log(
      `#${r.rank} ${r.candidateName} | Final Score: ${r.qualifiedScore.toFixed(3)} (Raw: ${r.rawUtility.toFixed(3)}) | ${status}`
    );
  }
}

runArchitectureEvaluation();

Simulation Execution Output

=== ADR Multi-Attribute Decision Matrix (5,000 TPS Payment Ledger) ===
#1 Modular Monolith + Partitioned PostgreSQL | Final Score: 4.275 (Raw: 4.275) | QUALIFIED
#2 Modular Services + CockroachDB (PC/EC) | Final Score: 4.000 (Raw: 4.000) | QUALIFIED
#3 Microservices + Apache Cassandra (PA/EL) | Final Score: 0.000 (Raw: 2.725) | DISQUALIFIED (Fails multi-account serializable ACID ledger invariant)

8. Guided Applied System Design Exercises & Debugging Drills

Drill 1: The Premature Microservice Rewrite Pushback

Leadership proposes rewriting a profitable 150,000-DAU B2B SaaS monolith ($450\text{ peak QPS}$, 10 engineers) into 25 Kubernetes microservices “to prepare for enterprise scale.”

1. Draft a concise ADR evaluating Modular Monolith refactoring vs. 25-service decomposition using the Type 1/Type 2 framework and Conway’s Law.
2. Quantify the serial availability and RPC latency penalty of splitting a 1-hop database join into an 8-hop synchronous RPC chain where each service runs at $99.9\%$ availability.
3. Define three concrete, measurable Re-evaluation Triggers (e.g., independent domain team count, CPU/IO workload divergence, primary write TPS) that justify extracting a specific bounded context later.

Drill 2: Reversing a One-Way Storage Door Safely

A legacy team stored relational customer orders and profiles in a schema-less KV store lacking secondary indexes and ACID joins, causing daily billing discrepancies.

1. Compare an online Expand-Contract / Dual-Write + Backfill + Shadow-Read Verification migration into PostgreSQL against an asynchronous Change-Data-Capture (Debezium + Apache Kafka) analytical projection.
2. Specify the exact rollback switch that turns a risky cutover into a controlled Two-Way Door until parity reaches $100.000\%$ over 7 days.

9. Summary & Operational Decision Checklist

1. Classify the Door First: Fast-track reversible Type 2 choices locally; require peer-reviewed ADRs with quantitative capacity bounds for irreversible Type 1 data, protocol, and topology commitments.
2. Respect Conway’s Law: Never create more independently deployed services than you have autonomous teams capable of owning private data stores and 24/7 on-call SLAs.
3. Map Storage onto PACELC: Explicitly state whether a domain requires PC/EC consensus safety (paying quorum RTT on every write) or PA/EL availability and low latency (accepting stale reads and conflict resolution).
4. Embed Quantitative Re-evaluation Triggers: Every ADR’s Negative Consequences section must record the exact throughput, latency, or team-size threshold at which the chosen architecture should be revisited.

10. Authoritative References & Version Context

• Architecture & Database Context: Storage and modularity comparisons reflect PostgreSQL 16/17 declarative partitioning, CockroachDB 24.x Raft/MVCC serializable isolation, Apache Cassandra 5.0 (PA/EL tunable quorums + Accord transactions roadmap), and Shopify Packwerk modular boundary enforcement.
• Michael Nygard, “Documenting Architecture Decisions”, Cognitect Engineering, 2011 — defines the canonical lightweight Markdown ADR format (Context, Decision, Status, Consequences).
• Melvin E. Conway, “How Do Committees Invent?”, Datamation 14(4), pp. 28–31, 1968 — primary formulation of Conway’s Law linking organizational communication topology to system interfaces.
• Daniel J. Abadi, “Consistency Tradeoffs in Modern Distributed Database System Design: CAP is Only Part of the Story”, IEEE Computer 45(2), pp. 37–42, 2012 — introduces the PACELC theorem extending CAP to normal-operation latency vs. consistency trade-offs.
• Kelly Sutton and Kirstin Jarchow (Shopify Engineering), “Deconstructing the Monolith: Designing Software that Maximizes Developer Productivity” — production case study on enforcing modular boundaries inside a high-throughput monolith.