1.Introduction and Installation

Distributed systems require reliable coordination mechanisms to manage configuration, service discovery, and leader election across multiple nodes. etcd serves as the backbone for such coordination, providing a consistent and highly available key-value store designed specifically for distributed environments. This chapter introduces the fundamental nature of etcd, compares it with alternative solutions, outlines hardware requirements, and guides through various installation methods to establish a working environment.

What is etcd and Use Cases

etcd is a consistent distributed key-value store that manages small amounts of data critical to distributed system coordination. The name derives from the Unix /etc directory—traditionally used for single-system configuration—combined with "d" for distributed systems, creating essentially a "distributed /etc directory." Pronounced as "/ˈɛtsiːdiː/", this system stores configuration data and metadata that must remain consistent across clusters, even during network partitions or node failures.

The primary design philosophy centers on strong consistency and partition tolerance rather than availability. When network partitions occur, etcd prefers to become unavailable rather than risk inconsistent data—a critical property for configuration stores where stale data could lead to split-brain scenarios or conflicting system states. This makes etcd particularly suitable for scenarios requiring strict consistency guarantees.

Modern cloud-native infrastructure relies heavily on etcd for several critical functions. Kubernetes, the container orchestration platform, uses etcd as its primary data store for all cluster state, configuration, and metadata. The API server persists every object—pods, services, deployments—into etcd, utilizing its watch capabilities to monitor changes and trigger reconciliations across the cluster. Container Linux (formerly CoreOS) employed etcd through tools like locksmith to coordinate automatic operating system updates, ensuring only a subset of cluster nodes rebooted simultaneously to maintain service availability.

Beyond container orchestration, etcd supports various distributed coordination patterns including leader election, distributed locking, and service discovery. Applications leverage these primitives to coordinate work distribution, ensure singleton services across clusters, and maintain registry information for microservices architectures. The system handles small data volumes efficiently—typically configurations rather than application data—keeping entire datasets in memory for rapid access while maintaining durability through replicated write-ahead logs.

Comparison with ZooKeeper and Consul

When selecting a coordination service, understanding the distinctions between etcd, ZooKeeper, and Consul proves essential for architectural decisions. Each solution addresses similar problems but with different trade-offs regarding consistency models, operational complexity, and ecosystem integration.

etcd versus ZooKeeper

ZooKeeper has served as the traditional solution for distributed coordination for many years, but etcd represents an evolution informed by operational experience with ZooKeeper's limitations. Both systems provide consistent key-value storage and distributed primitives, yet etcd introduces several architectural improvements.

Dynamic cluster membership reconfiguration stands as a significant advantage. ZooKeeper requires static configuration or complex procedures to change cluster membership, while etcd supports runtime reconfiguration without downtime. This flexibility simplifies scaling operations and maintenance windows in production environments.

Data model differences also favor etcd. ZooKeeper maintains a hierarchical znode structure without multi-version concurrency control (MVCC), whereas etcd implements MVCC allowing access to historical versions of keys. This capability enables time-travel queries and consistent snapshots without blocking writes. Additionally, etcd's lease primitives decouple connections from sessions, providing more robust handling of temporary network interruptions compared to ZooKeeper's session-based ephemeral nodes.

Protocol modernization distinguishes the two systems significantly. ZooKeeper utilizes the custom Jute RPC protocol, limiting language binding availability primarily to Java ecosystems through client libraries like Curator. Conversely, etcd builds upon gRPC, offering broad language support including Go, C++, Java, and others. Furthermore, etcd exposes HTTP/JSON endpoints, enabling interaction using standard tools like curl without specialized client libraries.

Operational stability under load represents another critical difference. etcd maintains stable read and write performance under high throughput scenarios, while ZooKeeper clusters often experience performance degradation under similar loads. The watch mechanism in etcd provides reliable event streaming without silent event drops, whereas ZooKeeper's watch system requires careful client-side handling to prevent missed updates.

etcd versus Consul

Consul operates as a comprehensive service discovery framework rather than a pure key-value store. While both systems offer key-value functionality, their primary use cases diverge substantially. Consul provides built-in health checking, failure detection, and DNS services alongside its storage capabilities, making it suitable for organizations requiring an all-in-one service mesh solution.

However, for pure key-value storage requirements, etcd demonstrates superior scalability and consistency characteristics. Consul's storage system exhibits higher latencies and memory pressure when managing millions of keys, lacking etcd's MVCC capabilities, conditional transactions, and reliable streaming watches. Consul supports atomic operations but does not provide the same granular revision-based consistency guarantees as etcd.

The architectural positioning differs: etcd functions as a substrate for building higher-level systems (like Kubernetes), while Consul presents as an end-to-end solution. When applications require only distributed configuration storage and coordination primitives without service mesh features, etcd offers better performance characteristics and simpler operational overhead. Conversely, environments needing comprehensive service discovery with health checking may find Consul's integrated approach preferable, though they sacrifice some storage performance and consistency granularity.

etcd versus NewSQL Databases

NewSQL systems such as CockroachDB, TiDB, and Google Spanner provide strong consistency across distributed data stores but target fundamentally different use cases. These databases partition data across multiple consistent replication groups (shards), enabling horizontal scaling to terabyte and petabyte datasets with SQL query capabilities.

etcd maintains all data within a single consistent replication group, optimizing for low-latency metadata operations rather than bulk data storage. While NewSQL systems sacrifice some latency for horizontal scalability and rich query semantics, etcd prioritizes strict ordering and rapid consensus for small datasets. The revision mechanism in etcd assigns a global monotonic ID to every modification across the entire keyspace—a property difficult to maintain in sharded NewSQL systems without cross-shard coordination overhead.

For metadata storage, service discovery, and distributed coordination where data volumes remain below several gigabytes, etcd provides superior performance and simpler operational characteristics. NewSQL databases suit applications requiring complex queries across large datasets distributed across data centers, whereas etcd excels at high-throughput coordination tasks with minimal data volumes.

System Requirements and Hardware

Running etcd effectively requires careful consideration of hardware resources, particularly regarding disk performance and network reliability. While etcd functions adequately on modest hardware for development purposes, production deployments demand specific configurations to ensure stability and performance.

Operating System and Architecture

Linux operating systems with amd64 architecture provide the most stable production environment for etcd deployments. While other platforms support etcd, the Linux kernel offers optimal performance characteristics for the synchronous I/O operations critical to etcd's consensus protocol. ARM architectures and other Unix variants work but receive less extensive production testing.

CPU Requirements

Typical etcd deployments require modest CPU resources. Clusters serving fewer than 1000 requests per second generally operate comfortably with two to four dedicated cores. High-throughput scenarios involving thousands of concurrent clients or tens of thousands of requests per second may require eight to sixteen cores, as CPU-bound operations increase with the number of watchers and concurrent connections. The Raft consensus algorithm and gRPC processing consume CPU cycles proportionally to request volume and cluster size.

Memory Considerations

Memory requirements depend primarily on the dataset size and watcher count. Since etcd caches the entire keyspace in memory and maintains watcher tracking structures, adequate RAM proves essential. Standard deployments typically function well with 8GB of memory. Heavy deployments featuring thousands of watchers and millions of keys require 16GB to 64GB to prevent out-of-memory conditions or excessive garbage collection pauses. The memory footprint scales with the number of keys, active leases, and connected clients.

Disk Performance

Disk performance represents the most critical hardware factor for etcd stability. The consensus protocol requires writing every request to persistent storage before acknowledgment, making disk latency directly impact cluster availability. Slow disks increase request latency and risk triggering leader elections due to heartbeat timeouts.

Minimum requirements specify 50 sequential IOPS (approximately 7200 RPM disk performance), while production workloads benefit from 500 sequential IOPS provided by local SSDs or high-performance virtualized block devices. etcd exhibits particular sensitivity to disk write latency variance; SSDs generally outperform spinning disks due to lower latency variability. When using mechanical drives, select 15,000 RPM models and consider RAID 0 configurations to increase throughput, though mirroring remains unnecessary given etcd's built-in replication across three or more nodes.

Disk bandwidth requirements remain modest—10MB/s suffices for recovering 100MB of data within 15 seconds. Larger clusters handling gigabyte-scale datasets should provision 100MB/s or higher for rapid member recovery.

Network Infrastructure

Network reliability significantly impacts cluster stability. etcd clusters require low-latency, high-bandwidth connections between members, preferably within a single data center to minimize partition risks. 1GbE networking satisfies most deployments, while large clusters benefit from 10GbE to reduce recovery times when members fail and must catch up on replication.

Cross-data-center deployments introduce latency overheads that may trigger false-positive failure detections. When multi-region redundancy proves necessary, select geographically close data centers and adjust heartbeat timeouts accordingly.

Installing Pre-built Binaries

The most straightforward installation method involves downloading official pre-built binaries from the etcd release repository. This approach ensures obtaining a tested, stable version without compilation dependencies.

Begin by downloading the compressed archive corresponding to your platform and architecture from the official releases page. Extract the archive contents to reveal the binary directory containing etcd, etcdctl, and etcdutl executables. Move these binaries to a location within your system PATH, such as /usr/local/bin/, or add the extraction directory to your PATH environment variable.

Verification confirms successful installation:

etcd --version

This command should display the etcd version, API version, and associated Git SHA, confirming the binary executes correctly and is accessible from your shell.

Building from Source

Building etcd from source requires Go version 1.21 or later. This method suits developers requiring specific modifications, bleeding-edge features, or custom optimizations not available in release binaries.

First, clone the etcd repository using Git, specifying the desired version tag:

git clone -b v3.7.0 https://github.com/etcd-io/etcd.git

Navigate to the repository directory and execute the build script:

cd etcd
./scripts/build.sh

The compilation process generates binaries within the bin directory. Add this directory to your PATH to make the commands available system-wide:

export PATH="$PATH:`pwd`/bin"

Verify the build succeeded by checking the version output. Building from source allows customization of build tags and integration of experimental features, though production deployments typically prefer the stability of release binaries.

Package Manager Installation

Package managers offer convenient installation for development environments, though production deployments should verify version currency as distribution repositories often lag behind official releases.

macOS via Homebrew

Homebrew provides the simplest macOS installation path. Update the package index and install etcd:

brew update
brew install etcd

Verify installation by querying the version. Homebrew typically maintains recent versions, making this suitable for development workflows on macOS.

Linux via Homebrew

Linux users can utilize Homebrew for recent software versions rather than relying on potentially outdated distribution repositories. After installing Homebrew on Linux, update and install:

brew update
brew install etcd

This method bypasses the often significantly outdated versions found in official Linux distribution repositories. While convenient for development, verify version compatibility with your target deployment environment, as package manager versions may not match the latest stable release.

Caution Regarding OS Packages

Installing etcd through standard Linux distribution repositories (apt, yum, dnf) often yields outdated versions. These packages lack automatic maintenance by the etcd project, potentially missing critical bug fixes or security patches. For production systems, prefer pre-built binaries or source builds over OS package manager installations unless specifically maintaining curated internal repositories.

Installation Verification

After installation, verify functionality by starting a local single-node cluster and executing basic operations. This validation ensures the etcd server and client tools function correctly and can communicate.

Launch the etcd server:

etcd

The server starts listening on localhost:2379 for client requests and localhost:2380 for peer communication. The output displays info-level logs indicating server initialization and listening ports.

From a separate terminal, test write operations using etcdctl:

etcdctl put greeting "Hello, etcd"

A successful write returns "OK". Retrieve the value to confirm read operations:

etcdctl get greeting

The output displays the key "greeting" followed by its value "Hello, etcd". This confirms the installation functions correctly and the cluster accepts client connections.

For a more comprehensive verification, check cluster health and version information:

etcdctl endpoint status --cluster
etcdctl version

These commands verify API compatibility and cluster connectivity. With a working installation established, the next chapter explores etcd's core architectural concepts including the key-value data model, revision mechanism, and Raft consensus algorithm that underpin these operations.