Architecture Study: Apache HDFS & HBase — Distributed Storage & Wide-Column Engines

When distributed data volumes scale into dozens of petabytes across thousands of commodity servers, relational databases and traditional shared-disk storage arrays break down. In 2003 and 2006, Google published seminal research papers on The Google File System (GFS) and Bigtable: A Distributed Storage System for Structured Data. The open-source distributed systems community implemented these blueprints as Apache Hadoop HDFS and Apache HBase.

Together, HDFS and HBase established the foundation of big-data distributed storage:

• HDFS: A scalable, fault-tolerant, append-only distributed filesystem providing high streaming throughput across petabytes of data on commodity hardware.
• HBase: A distributed, scalable, sorted, wide-column NoSQL database running on top of HDFS, providing random, real-time read/write access to billions of rows with billions of columns.

This study examines their internal architecture, storage layouts, metadata management, replication pipelines, and LSM engine mechanics.

1. Context & Architecture Setting

Project Identity, Version & License Baseline

HDFS and HBase were designed under distinct engineering assumptions that diverge fundamentally from POSIX filesystems and RDBMS databases:

1. Hardware failure is the norm, not the exception: In a cluster of $5,000$ commodity servers with $60,000$ spinning disks or NVMe drives, multiple disks, memory modules, and network switches fail every single day. The storage layer must recover automatically without operator intervention.
2. Streaming reads over low-latency random writes: Big-data analytics (MapReduce, Spark, Flink) process terabytes in bulk. HDFS prioritizes high sustained sequential streaming throughput ($>100\text{ MB/s}$ per spindle) over sub-millisecond random file seeks.
3. Write-once, read-many access model: Files on HDFS are append-only. Modifying bytes in the middle of an existing file is disallowed, vastly simplifying concurrency and replication invariants.
4. HBase on HDFS layering: Rather than implementing its own distributed replication protocol (like Cassandra’s Dynamo quorums or Raft consensus), HBase delegates raw replication and block durability entirely to HDFS, focusing its own architecture on LSM indexing, MemStore caching, and Region splitting.

2. Architectural Mental Model

SYSTEM ARCHITECTURE BLUEPRINTDISTRIBUTED SYSTEMS

⤢Expand

ARCHITECTURAL FLOW · STEP 1 OF 3

1. Apache HDFS Distributed File System & NameNode Metadata Management (In-Memory FSImage and EditLog)

Establishes the ingress and structural boundary across Apache HDFS Distributed File System & NameNode Metadata Management (In-Memory FSImage and EditLog): Active NameNode: In-memory namespace inode tree (~150 bytes per block); Standby NameNode: Checkpoints FSImage via JournalNodes (QJM Paxos); ZooKeeper Failover Controller (ZKFC): Automated split-brain fencing.

Architectural Invariant: EditLog transactions batched to Quorum Journal Manager (2F + 1 nodes).

ARCHITECTURAL FLOW · STEP 2 OF 3

2. HDFS 3-Way Rack-Aware Pipelined Writes & Rack 1 (Local)

Coordinates core data-plane execution and state transitions through HDFS 3-Way Rack-Aware Pipelined Writes & Rack 1 (Local): DataNode 1 (R1) Receives block from client stream (64KB pkts) Pipelines to Rack 2 →; Off-rack replica Pipelines to DN3 →; Same off-rack Acks back pipeline.

Architectural Invariant: Tolerates full rack power failure with zero data loss.

ARCHITECTURAL FLOW · STEP 3 OF 3

3. RegionServer Storage Anatomy (LSM on HDFS) & Write-Ahead Log

Enforces durability, state convergence, and output verification across RegionServer Storage Anatomy (LSM on HDFS) & Write-Ahead Log: HLog on HDFS Sequential append Guarantees durability Crash replay log; ConcurrentSkipList In-memory sorted tree Per Column Family Flushed at 128MB →; Immutable B-Tree Data Blocks (64KB) Bloom Filter block Multi-level Index.

Architectural Invariant: BlockCache (L1/L2): On-heap LRU + Off-heap BucketCache for zero GC pause.

+--------------------------------------------------------------------------------+
|                           HBASE & HDFS SYSTEM LAYOUT                           |
+--------------------------------------------------------------------------------+
| Client Applications (Put / Get / Scan / Analytics)                             |
+---------------------------------------+----------------------------------------+
                                        |
                 +----------------------+----------------------+
                 |                                             |
                 v (Metadata resolution)                       v (Direct I/O)
        +------------------+                         +------------------+
        | ZooKeeper Quorum |                         |   HBase Master   |
        | /hbase/meta-rs   |                         | (Region Splits & |
        +--------+---------+                         |  Rebalancing)    |
                 |                                   +------------------+
                 v
        +------------------------------------------------------+
        | RegionServer (Hosts Regions [startKey, endKey))      |
        | +--------------------+    +------------------------+ |
        | | WAL (HLog on HDFS) |    | MemStore (SkipListMap) | |
        | +---------+----------+    +-----------+------------+ |
        |           |                           | Flush        |
        +-----------|---------------------------|--------------+
                    v                           v
+--------------------------------------------------------------------------------+
| HDFS Storage Layer (Blocks: 128 MB default)                                    |
|                                                                                |
| Active NameNode (RAM Inodes + QJM JournalNodes) <-> Standby NameNode           |
|                                                                                |
| DataNode 1 (Rack 1)       DataNode 2 (Rack 2)       DataNode 3 (Rack 2)        |
| [HFile Block R1] -------> [HFile Block R2] -------> [HFile Block R3]           |
| (64KB Pipelined Stream)   (Cross-Rack Transit)      (Intra-Rack Transit)       |
+--------------------------------------------------------------------------------+

3. Core Mechanisms & Protocol Internals

HDFS NameNode Metadata Architecture

The NameNode is the master of the HDFS filesystem namespace.

• In-Memory Namespace: The NameNode maintains the entire directory tree, file ownership, permissions, and block IDs in RAM. This allows directory traversals and file open operations to complete in microseconds.
• FSImage & EditLog:
  • FSImage: A serialized point-in-time snapshot of the filesystem namespace.
  • EditLog: A write-ahead append log recording every namespace transaction (e.g., file creation, block allocation, deletion).
• High Availability via Quorum Journal Manager (QJM):
  • In modern HDFS, the Active NameNode logs transactions to an odd-numbered quorum of JournalNodes (typically 3 or 5).
  • A transaction is committed when a majority ($\lfloor N/2\rfloor +1$) acknowledge disk durability.
  • The Standby NameNode continuously reads edits from JournalNodes, keeping its in-memory namespace synchronized.
  • ZooKeeper Failover Controllers (ZKFC) use heartbeats and ephemeral z-nodes to perform automated, split-brain-fenced failover if the Active NameNode stops responding.

Rack-Aware Block Replication Pipeline

When an HDFS client writes a 128 MB block:

1. The client requests block allocation from the NameNode.
2. The NameNode returns a list of 3 target DataNodes based on rack topology:
   • $D{N}_1$: On the client’s local rack (or local node if client is a DataNode).
   • $D{N}_2$: On a distinct, remote rack.
   • $D{N}_3$: On a different node in that same remote rack.
3. The client connects to $D{N}_1$ and streams data in small $64\text{ KB}$ packets.
4. $D{N}_1$ writes the packet to local disk and simultaneously forwards it over TCP to $D{N}_2$.
5. $D{N}_2$ writes the packet and forwards it to $D{N}_3$.
6. Acknowledgment flows in reverse: $D{N}_3\to D{N}_2\to D{N}_1\to \text{Client}$.
7. Cross-rack efficiency: Only one stream traverses the expensive inter-rack switch core. If either rack loses power or networking, two replicas remain safe.

4. HBase Storage Engine Anatomy: The LSM Tree on HDFS

HBase tables are sparse, distributed, persistent, multidimensional sorted maps indexed by: $$\left(\text{RowKey},\text{ColumnFamily},\text{ColumnQualifier},\text{Timestamp}\right)⟶\text{Value}$$

+--------------------------------------------------------------------+
|               HBASE REGIONSERVER WRITE & FLUSH LIFECYCLE           |
+--------------------------------------------------------------------+
| 1. Client Put(rowKey, CF:qualifier, value)                         |
|    |                                                               |
|    +--> [WAL: Write-Ahead Log] (Append-only on HDFS)               |
|    |    (Ensures durability in case of RegionServer crash)         |
|    |                                                               |
|    +--> [MemStore: ConcurrentSkipListMap] (Sorted in RAM)          |
|         (Provides O(log N) lock-free read and write)               |
|                                                                    |
| 2. MemStore Flushes when size >= 128 MB                            |
|    |                                                               |
|    v                                                               |
| [HFile 1 (HDFS)]  [HFile 2 (HDFS)]  [HFile 3 (HDFS)]               |
| (Immutable sorted B-Tree format with Block Index & Bloom Filters)  |
|                                                                    |
| 3. LSM Compaction Engine                                           |
|    - Minor Compaction: Merges small HFiles into larger HFiles      |
|    - Major Compaction: Merges ALL HFiles, discards Tombstones      |
+--------------------------------------------------------------------+

The HFile Format

An HFile is an immutable on-disk file stored on HDFS:

• Data Blocks: Default $64\text{ KB}$ chunks of sorted KeyValue pairs compressed with Snappy or ZSTD.
• Index Blocks: Multi-level B-tree index loaded into RegionServer RAM to enable random seeks within an HFile in $O(\log B)$ disk block reads.
• Bloom Filters:
  • ROW: Hashing the row key enables HBase to skip an HFile entirely if the row key does not exist in that file.
  • ROWCOL: Hashes RowKey + ColumnQualifier, skipping files during targeted column queries.

5. Mathematical & Performance Analysis

HDFS Replication Reliability & MTBF Math

The Mean Time Between Failures ($MTBF$) of an HDFS storage cluster can be modeled probabilistically. Let:

• $N$ = Number of disks in the cluster ($N=10,000$).
• $AFR$ = Annualized Failure Rate of commodity disk drives ($AFR\approx 3\%=0.03$).
• $\lambda =\frac{AFR}{8760\text{ hours}}\approx 3.42\times 10^{-6}\text{ failures/hour per disk}$.
• $T_{repair}$ = Time required to reconstruct an under-replicated 128 MB block across the network ($T_{repair}\approx 15\text{ minutes}=0.25\text{ hours}$).

For a 3-way replicated block, data loss occurs only if all 3 replicas fail before replication completes: $$P(\text{triple failure during repair})\approx \left(\genfrac{}{}{0ex}{}{3}{3}\right)\times {\left(\lambda \times T_{repair}\right)}^2$$ $${\left(3.42\times 10^{-6}\times 0.25\right)}^2\approx {\left(8.55\times 10^{-7}\right)}^2\approx 7.3\times 10^{-13}$$

Across $100\text{ million}$ blocks, the annual probability of permanent block loss with 3-way rack-aware replication is less than $0.001\%$, providing 11 9s ($99.999999999\%$) of annual durability.

HBase Write Amplification Math

In an LSM engine, every written byte is rewritten multiple times during compactions. $$WA=\frac{\text{Bytes Written to Underlying Storage}}{\text{Bytes Ingested by Application}}$$ In HBase:

1. Write 1: Ingested to WAL on HDFS ($3\times$ replication) $\to W{A}_1=3$.
2. Write 2: Flushed from MemStore to HFile ($3\times$ replication) $\to W{A}_2=3$.
3. Write 3: Minor and Major compactions re-reading and rewriting HFiles ($3\times$ replication). $$\text{Total Write Amplification }W{A}_{total}\approx 6+3\times \sum \text{compaction tiers}\approx 15\text{ to }30$$

Understanding this write amplification is essential when sizing NVMe endurance and inter-rack network bandwidth.

6. Concrete Failure Modes & Edge Cases

1. The Small Files Heap Exhaustion Crisis

• Failure Mode: An ingestion pipeline outputs one small file (e.g., $5\text{ KB}$) every second per sensor for $10,000$ IoT sensors.
• System Impact: Over 6 months, the cluster accumulates $150\text{ million}$ files. While disk usage is only $750\text{ GB}$, the NameNode heap consumes: $$150,000,000\times 150\text{ bytes}\approx 22.5\text{ GB RAM}$$ Stop-the-world JVM GC pauses stretch past 60 seconds. ZooKeeper considers the NameNode dead, triggering false failovers and cluster-wide crash loops.
• Remediation: Use Hadoop Archives (HAR), sequence files, or compact small files in Kafka/Flink before writing to HDFS.

2. RegionServer Long JVM GC Pauses & ZK Session Expiration

• Failure Mode: RegionServers allocate heavy on-heap BlockCaches. During heavy analytical scan workloads, the JVM triggers a multi-second Full GC pause.
• System Impact: The RegionServer cannot send its ZooKeeper heartbeat within zookeeper.session.timeout (e.g., 30s). ZooKeeper deletes the ephemeral node. HMaster declares the RegionServer dead and triggers Region reassignment. When the GC pause finishes, the old RegionServer attempts to process writes, causing split-brain write conflicts until fenced.
• Remediation: Configure off-heap memory for HBase caching (BucketCache in offheap mode). Tune G1GC with -XX:MaxGCPauseMillis=50.

3. RowKey Hotspotting

• Failure Mode: Row keys are formed using timestamp_sensorId.
• System Impact: Because timestamps increase monotonically, all writes target the single Region holding the highest key range. One RegionServer runs at 100% CPU while 99 other RegionServers remain completely idle.
• Remediation: Salt row keys with a fixed hash prefix: (hash(sensorId) % 16) + "_" + timestamp + "_" + sensorId.

7. Production Trade-Offs & Anti-Patterns

Trade-Off Matrix

8. Practical Engineering Simulation: HDFS Block Placement & HBase MemStore

This TypeScript simulation models HDFS rack-aware block placement and an in-memory HBase LSM storage engine with WAL append, MemStore flushing, and Bloom filter checks.

/**
 * Systems Notebook - HDFS Block Placement & HBase LSM Engine Simulation
 */

interface DataNodeInfo {
  id: string;
  rack: string;
  availableBytes: number;
}

interface HDFSBlock {
  blockId: string;
  sizeBytes: number;
  replicas: string[]; // DataNode IDs
}

class HDFSBlockCoordinator {
  constructor(private readonly nodes: DataNodeInfo[]) {}

  /**
   * 3-Way Rack-Aware Placement:
   * R1: Local node or local rack
   * R2: Remote rack
   * R3: Same remote rack as R2 (different node)
   */
  public allocateBlock(blockId: string, sizeBytes: number, preferredRack: string): HDFSBlock {
    const localRackNodes = this.nodes.filter((n) => n.rack === preferredRack);
    const remoteRackNodes = this.nodes.filter((n) => n.rack !== preferredRack);

    if (localRackNodes.length === 0 || remoteRackNodes.length < 2) {
      throw new Error('Insufficient rack distribution for 3-way fault-tolerant placement');
    }

    const r1 = localRackNodes[0];
    const remoteRackName = remoteRackNodes[0].rack;
    const sameRemoteNodes = remoteRackNodes.filter((n) => n.rack === remoteRackName);

    const r2 = sameRemoteNodes[0];
    const r3 = sameRemoteNodes[1] ?? remoteRackNodes[1];

    return {
      blockId,
      sizeBytes,
      replicas: [r1.id, r2.id, r3.id]
    };
  }
}

interface KeyValueMutation {
  rowKey: string;
  family: string;
  qualifier: string;
  timestamp: number;
  value: string;
  isDelete?: boolean;
}

class HBaseLSMRegion {
  private walLog: KeyValueMutation[] = [];
  private memStore: Map<string, KeyValueMutation> = new Map();
  private hfiles: Array<{
    fileName: string;
    bloomFilter: Set<string>;
    data: KeyValueMutation[];
  }> = [];

  constructor(
    public readonly regionName: string,
    private readonly flushThresholdCount: number = 3
  ) {}

  public put(mutation: KeyValueMutation): void {
    // 1. Append to WAL (HDFS durability simulation)
    this.walLog.push(mutation);

    // 2. Insert into sorted MemStore
    const compositeKey = `${mutation.rowKey}:${mutation.family}:${mutation.qualifier}`;
    this.memStore.set(compositeKey, mutation);

    // 3. Flush if threshold reached
    if (this.memStore.size >= this.flushThresholdCount) {
      this.flushMemStore();
    }
  }

  private flushMemStore(): void {
    const hfileName = `hfile-${Date.now()}-${this.hfiles.length + 1}`;
    const sortedEntries = Array.from(this.memStore.values()).sort((a, b) =>
      a.rowKey.localeCompare(b.rowKey)
    );

    const bloom = new Set<string>();
    sortedEntries.forEach((e) => bloom.add(e.rowKey));

    this.hfiles.push({
      fileName: hfileName,
      bloomFilter: bloom,
      data: sortedEntries
    });

    console.log(`[MemStore Flush] Wrote ${sortedEntries.length} rows to ${hfileName}`);
    this.memStore.clear();
  }

  public get(rowKey: string, family: string, qualifier: string): KeyValueMutation | null {
    const compositeKey = `${rowKey}:${family}:${qualifier}`;

    // 1. Check MemStore first (newest data)
    if (this.memStore.has(compositeKey)) {
      const cell = this.memStore.get(compositeKey)!;
      return cell.isDelete ? null : cell;
    }

    // 2. Search HFiles in reverse chronological order (newest to oldest)
    for (let i = this.hfiles.length - 1; i >= 0; i--) {
      const hfile = this.hfiles[i];

      // Bloom filter probe skips unnecessary disk block reads
      if (!hfile.bloomFilter.has(rowKey)) {
        continue;
      }

      const match = hfile.data.find(
        (kv) => kv.rowKey === rowKey && kv.family === family && kv.qualifier === qualifier
      );

      if (match) {
        return match.isDelete ? null : match;
      }
    }

    return null;
  }
}

// -------------------------------------------------------------
// Verification Execution
// -------------------------------------------------------------
function runVerification(): void {
  console.log('=== Step 1: HDFS Rack-Aware Placement ===');
  const clusterNodes: DataNodeInfo[] = [
    { id: 'dn-rack1-node1', rack: '/rack1', availableBytes: 1e12 },
    { id: 'dn-rack1-node2', rack: '/rack1', availableBytes: 1e12 },
    { id: 'dn-rack2-node1', rack: '/rack2', availableBytes: 1e12 },
    { id: 'dn-rack2-node2', rack: '/rack2', availableBytes: 1e12 }
  ];

  const coordinator = new HDFSBlockCoordinator(clusterNodes);
  const block = coordinator.allocateBlock('blk_1073741824', 134217728, '/rack1');
  console.log('Allocated Block:', block);

  console.log('\n=== Step 2: HBase LSM Engine Put & Flush ===');
  const region = new HBaseLSMRegion('user_table,region_001', 2);

  region.put({ rowKey: 'user:101', family: 'cf1', qualifier: 'email', timestamp: 1, value: 'alice@corp.com' });
  region.put({ rowKey: 'user:102', family: 'cf1', qualifier: 'email', timestamp: 1, value: 'bob@corp.com' });
  // Trigger flush
  region.put({ rowKey: 'user:103', family: 'cf1', qualifier: 'email', timestamp: 2, value: 'charlie@corp.com' });

  console.log('\n=== Step 3: HBase Point Read with Bloom Filter ===');
  const result = region.get('user:101', 'cf1', 'email');
  console.log('Read Result user:101:', result?.value);

  const missing = region.get('user:999', 'cf1', 'email');
  console.log('Read Result user:999 (Bloom Filter skipped):', missing);
}

runVerification();

9. Observability & SRE Metrics

+-------------------------------------------------------------------------------+
|                      HDFS & HBASE SRE TELEMETRY DASHBOARD                     |
+-------------------------------------------------------------------------------+
| Metric Indicator       | JMX / Prometheus Name | Alert Condition & Remediation|
+------------------------+-----------------------+------------------------------+
| Under-Replicated Blocks| UnderReplicatedBlocks | Alert if > 0 for > 30m;      |
|                        | (Hadoop NameNode)     | DataNode failure in progress |
| NameNode RPC Queue Time| RpcQueueTime          | Alert if p99 > 100ms;        |
|                        | (Hadoop NameNode)     | metadata lock contention     |
| HBase Region Splits    | splitQueueLength      | Alert if queue > 10;         |
|                        | (HBase RegionServer)  | storage write stall imminent |
| StoreFile Compaction   | compactionQueueLength | Alert if queue > 20;         |
| Queue                  | (HBase RegionServer)  | I/O bandwidth exhausted      |
| RegionServer GC Pause  | JVM.PauseTimeMS       | Alert if pause > 5000ms;     |
|                        |                       | impending ZK session timeout |
+------------------------+-----------------------+------------------------------+

10. System Design Interview & Production Rubric

+-------------------------------------------------------------------------------+
| CRITERIA            | LEVEL 3 (Senior)          | LEVEL 4 (Staff / Principal) |
+---------------------+---------------------------+-----------------------------+
| Distributed Storage | Understands HDFS 3x       | Deconstructs rack-aware     |
| Topologies          | replication factor        | pipeline TCP streaming      |
| Storage Engines     | Mentions HBase as NoSQL   | Explains LSM MemStore flush,|
|                     | database                  | HFile B-Tree, & Bloom filter|
| Bottlenecks & Scale | Points to disk space as   | Identifies NameNode heap    |
|                     | primary limit             | exhaustion (small files)    |
| Data Partitioning   | Recommends auto-increment | Explains salting/hashing to |
|                     | IDs for primary keys      | avoid Region hotspotting    |
+---------------------+---------------------------+-----------------------------+

11. Exercises & Deliverables

1. Calculate NameNode RAM Requirements:
   • Write a sizing model in Python or TypeScript for a cluster storing $500\text{ million}$ files with an average of 1.2 blocks per file. Determine the NameNode heap size and garbage collection parameters required.
2. Simulate Region Split and Compaction:
   • Extend the TypeScript simulation above to split a Region when its HFiles exceed a target size threshold ($10\text{ GB}$). Implement the reference-file split mechanism that avoids physical data copying during split.
3. Design a Salting Row Key Scheme:
   • Design an HBase row key for a global telemetry platform ingesting $100,000$ events/second across $50,000$ vehicles. Ensure range queries by time within a vehicle remain efficient while writes distribute evenly across 64 RegionServers.

export interface HFileReference {
  refName: string;
  parentHFileName: string;
  splitRowKey: string;
  half: 'BOTTOM' | 'TOP'; // BOTTOM: < splitRowKey, TOP: >= splitRowKey
}

export function splitRegionZeroCopy(
  parentRegion: HBaseLSMRegion,
  allRowsSorted: KeyValueMutation[]
): { splitKey: string; bottomRefs: HFileReference[]; topRefs: HFileReference[] } {
  if (allRowsSorted.length < 2) {
    throw new Error('Region has insufficient rows to split');
  }
  const midIdx = Math.floor(allRowsSorted.length / 2);
  const splitKey = allRowsSorted[midIdx].rowKey;

  // Create lightweight metadata Reference Files instead of copying 10 GB HFiles over HDFS
  const bottomRefs: HFileReference[] = [
    { refName: `ref-bot-${Date.now()}`, parentHFileName: 'parent-hfile-01', splitRowKey: splitKey, half: 'BOTTOM' }
  ];
  const topRefs: HFileReference[] = [
    { refName: `ref-top-${Date.now()}`, parentHFileName: 'parent-hfile-01', splitRowKey: splitKey, half: 'TOP' }
  ];

  return { splitKey, bottomRefs, topRefs };
}

12. Comprehensive Architectural Summary

Apache HDFS and HBase represent an enduring masterclass in distributed systems separation of concerns:

• HDFS solves large-scale physical reliability through immutable append-only files, rack-aware block pipelining, and in-memory metadata management.
• HBase layers low-latency random queries on top of an append-only filesystem by utilizing Log-Structured Merge Trees (LSM), sequential WAL journaling, and RAM-cached MemStores.
• Decoupling distributed consensus and replication (handled by HDFS DataNodes) from indexing and query processing (handled by HBase RegionServers) enables petabyte-scale linear scaling across thousands of nodes.

13. Authoritative References & Version Context

• Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung (2003), “The Google File System”, ACM SOSP ‘03 — doi.org/10.1145/945445.945450
• Fay Chang et al. (2006), “Bigtable: A Distributed Storage System for Structured Data”, USENIX OSDI ‘06 — usenix.org
• Apache Hadoop 3.3/3.4 Architecture Guide — HDFS Architecture & Quorum Journal Manager — hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html
• Version Context: Architecture reflects Apache Hadoop 3.3+ / 3.4 HDFS (Erasure Coding 6+3, QJM HA) and Apache HBase 2.5+ storage engines, concluding Part 5’s Open-Source Architecture Studies and transitioning to Part 6: Production Engineering — Operate, Debug & Evolve (CORE-M26-001) in the Master Spine.