Chapter 26: Service Discovery & Configuration

In dynamic distributed systems, service instances are constantly being created, destroyed, scaled, and moved. Service discovery provides a mechanism for services to find and communicate with each other without hardcoded network locations. This chapter covers service discovery patterns, popular tools (Consul, etcd, ZooKeeper), DNS-based approaches, and configuration management strategies.

1. The Problem

Traditional static configuration breaks down in modern cloud-native environments:

Challenge Traditional Approach Problem Dynamic Need
Hardcoded IPs config.json with IP:PORT Fails when instance replaced Runtime service lookup
Autoscaling Manual load balancer updates New instances not discoverable Auto-registration of new instances
Container Orchestration Static hostnames Containers get random IPs Ephemeral endpoint tracking
Health Failures Manual intervention to remove Requests routed to dead instances Automatic health-based deregistration
Multi-Region Region-specific configs No failover discovery Cross-region service mesh
Zero-Downtime Deploys Blue-green DNS swap Long DNS TTL delays Immediate endpoint updates
Interview Question: Why can't we just use DNS for service discovery?
Answer: DNS has long TTL caching (minutes), no native health checking, limited load balancing (round-robin), and no metadata support. In dynamic environments, DNS can't react fast enough to instance failures or rapid scaling events.

Key Requirements for Service Discovery

2. Service Discovery Patterns

Client-Side Discovery

Client queries the service registry directly and selects an instance using a load-balancing algorithm.

sequenceDiagram participant C as Client participant R as Service Registry participant S1 as Service Instance 1 participant S2 as Service Instance 2 S1->>R: Register (IP: 10.0.1.5) S2->>R: Register (IP: 10.0.1.6) C->>R: Query "payment-service" R-->>C: [10.0.1.5, 10.0.1.6] Note over C: Client selects instance
using load balancing C->>S1: HTTP Request S1-->>C: Response
Aspect Client-Side Discovery
Pros Simple, no extra hop, client controls load balancing strategy
Cons Client must implement discovery logic, couples client to registry
Examples Netflix Eureka + Ribbon, Consul with client agent
Load Balancing In-process (round-robin, random, least-connections)

Server-Side Discovery

Client makes a request to a load balancer, which queries the service registry and forwards to an instance.

sequenceDiagram participant C as Client participant LB as Load Balancer participant R as Service Registry participant S1 as Service Instance 1 participant S2 as Service Instance 2 S1->>R: Register S2->>R: Register LB->>R: Subscribe to updates R-->>LB: Instance list C->>LB: Request to "payment-service" Note over LB: Select healthy instance LB->>S1: Forward request S1-->>LB: Response LB-->>C: Response
Aspect Server-Side Discovery
Pros Client is simple, load balancer abstracts discovery complexity
Cons Extra network hop, load balancer is potential bottleneck/SPOF
Examples AWS ELB, Kubernetes Services, Nginx + Consul Template
Load Balancing Centralized (hardware/software load balancer)
Interview Question: Compare client-side vs server-side discovery trade-offs.
Answer: Client-side is faster (no extra hop) but requires client library and couples clients to registry. Server-side is simpler for clients (just hit a stable endpoint) but adds latency and needs a highly available load balancer. In practice, server-side is more common in Kubernetes environments, while microservices frameworks often use client-side.

3. Service Registry

The service registry is a database of available service instances and their locations. Two registration approaches:

Self-Registration Pattern

Aspect Details
Mechanism Service instance registers itself with registry on startup
Heartbeat Instance sends periodic heartbeats to maintain registration
Deregistration Instance deregisters on graceful shutdown; registry times out on missed heartbeats
Pros Simple architecture, no external registration agent needed
Cons Couples service to registry, every service must implement registration logic
Examples Netflix Eureka, Spring Cloud, Consul HTTP API

Third-Party Registration Pattern

Aspect Details
Mechanism External registrar watches service instances and registers them
Health Checking Registrar performs health checks and updates registry
Deregistration Registrar removes instance when it detects failure
Pros Service is decoupled from registry, consistent registration logic
Cons More complex infrastructure, registrar is an additional component
Examples Kubernetes Kubelet + API Server, Registrator + Consul, AWS ECS Service Discovery

Health Check Strategies

Strategy Mechanism Pros Cons
Heartbeat Service sends periodic "I'm alive" signal Low overhead, simple Doesn't verify actual health, false positives
HTTP Health Endpoint Registry polls /health endpoint Can check dependencies, detailed status Extra load on service, polling overhead
TCP Check Verify port is open Very lightweight Only checks network, not application health
Script-Based Custom script execution Highly flexible Can be slow, security concerns
gRPC Health Protocol Standard gRPC health checking Standardized for gRPC services Only for gRPC
Interview Question: What happens if a service crashes without deregistering?
Answer: With heartbeat-based registration, the registry will time out the instance after missing N consecutive heartbeats (e.g., 30 seconds in Consul). With health checks, the registry actively detects failure within the check interval (e.g., 10 seconds). Until timeout/detection, clients may receive stale endpoints, so retry logic and circuit breakers are critical.

4. DNS-Based Discovery

DNS can provide basic service discovery, but has significant limitations in dynamic environments.

DNS Approaches

Approach Mechanism Example Limitations
A Records Map hostname to IP addresses api.example.com → [10.0.1.5, 10.0.1.6] No port info, no metadata, TTL caching
SRV Records Include port, priority, weight _http._tcp.api.example.com → 10.0.1.5:8080 Poor client support, still has TTL issues
Round-Robin DNS Multiple A records rotated Client gets different IP each query Doesn't account for load or health
Kubernetes DNS CoreDNS with service-to-ClusterIP payment-service.default.svc.cluster.local Works well within cluster only

DNS TTL Problem

TTL Setting Impact Use Case
High TTL (5-60 min) Reduces DNS load, but stale data persists Static infrastructure
Low TTL (1-5 sec) Faster updates, but heavy DNS query load Dynamic environments (but still inadequate)
Zero TTL Every request queries DNS, huge overhead Almost never practical

Why DNS Fails for Modern Service Discovery:

Interview Question: When is DNS-based discovery acceptable?
Answer: For relatively static workloads with low instance churn, or when paired with a load balancer (DNS points to LB, LB does real discovery). Kubernetes uses DNS internally because kube-proxy handles the actual endpoint routing, so DNS just resolves to a stable ClusterIP.

5. Consul

Consul is a distributed service mesh solution providing service discovery, health checking, key/value store, and multi-datacenter support.

Architecture

graph TB subgraph "Datacenter 1" CS1[Consul Server 1
Leader] CS2[Consul Server 2] CS3[Consul Server 3] CA1[Consul Agent
Service A] CA2[Consul Agent
Service B] CA3[Consul Agent
Service C] CS1 -.Raft Consensus.-> CS2 CS2 -.Raft Consensus.-> CS3 CS3 -.Raft Consensus.-> CS1 CA1 --Gossip Protocol--> CA2 CA2 --Gossip Protocol--> CA3 CA1 --RPC--> CS1 CA2 --RPC--> CS2 CA3 --RPC--> CS3 end subgraph "Datacenter 2" CS4[Consul Server 4] end CS1 -.WAN Gossip.-> CS4

Core Components

Component Role Count Details
Consul Server Maintain cluster state, participate in Raft consensus 3-5 per datacenter One is leader, others are followers
Consul Agent Run on every node, register services, health checks 1 per host/container Lightweight, communicates with servers
Gossip Protocol Serf for membership and failure detection All agents LAN gossip (within DC), WAN gossip (across DCs)
Raft Consensus Leader election, log replication among servers Server cluster Ensures consistency of service catalog

Service Registration

Method Example Use Case
Agent Config File /etc/consul.d/web.json Static services on VMs
HTTP API PUT /v1/agent/service/register Dynamic registration from application
Consul Template Auto-generate config from templates Update Nginx/HAProxy config
Kubernetes Sync consul-k8s controller Sync K8s services to Consul

Health Checks

Type Configuration When to Use
Script Execute script, check exit code Custom logic, legacy apps
HTTP GET /health, expect 2xx status RESTful services
TCP Connect to port Non-HTTP services (databases)
TTL Service must update within interval Application-driven health
gRPC gRPC health checking protocol gRPC services
Docker Execute command in container Docker-based services

Key Features

Feature Description Example Use Case
KV Store Distributed key-value store for config Feature flags, connection strings
DNS Interface Query services via DNS (*.service.consul) Legacy apps needing DNS-based discovery
HTTP API RESTful API for all operations Programmatic service discovery
Prepared Queries Complex queries with failover logic Multi-DC failover, geo-routing
Consul Connect Service mesh with mTLS and intentions Zero-trust security, encrypted service-to-service
Multi-Datacenter Federate multiple Consul clusters Global service catalog

Query Example

# DNS query
dig @127.0.0.1 -p 8600 web.service.consul

# HTTP API query
curl http://localhost:8500/v1/health/service/web?passing

# Returns only healthy instances:
[
  {
    "Node": "node1",
    "Address": "10.0.1.5",
    "ServiceID": "web-1",
    "ServiceAddress": "10.0.1.5",
    "ServicePort": 8080,
    "Checks": [
      {"Status": "passing"}
    ]
  }
]
Interview Question: How does Consul handle network partitions?
Answer: Consul uses Raft consensus among servers. If the leader is partitioned from the majority, a new leader is elected in the majority partition. The minority partition cannot commit writes (maintaining consistency). Agents use gossip to detect failures quickly. When the partition heals, servers reconcile state.

6. etcd

etcd is a distributed, consistent key-value store used for shared configuration and service discovery, most notably as Kubernetes' backing store.

Architecture

graph TB subgraph "etcd Cluster" E1[etcd Node 1
Leader] E2[etcd Node 2
Follower] E3[etcd Node 3
Follower] E1 -.Raft Logs.-> E2 E1 -.Raft Logs.-> E3 E2 -.Election/Heartbeat.-> E1 E3 -.Election/Heartbeat.-> E1 end C1[Client 1] -->|Write| E1 C2[Client 2] -->|Read| E2 C3[Client 3] -->|Watch| E3 K8S[Kubernetes API Server] -->|All K8s State| E1

Core Concepts

Concept Description Details
Raft Consensus Leader election, log replication Ensures strong consistency, requires quorum (N/2+1)
Key-Value Store Hierarchical key space (like filesystem) /services/api/instance1 → {"ip": "10.0.1.5"}
Watch API Subscribe to key changes in real-time Clients notified immediately when values change
Lease Time-to-live for keys (auto-delete on expiry) Heartbeat mechanism: renew lease to keep key alive
MVCC Multi-version concurrency control Each write creates new version, enables time-travel queries
Transactions Atomic compare-and-swap, if-then-else Enables distributed locks, leader election

Service Discovery with etcd

Step Operation etcd Command
1. Grant Lease Create lease with TTL etcdctl lease grant 10
2. Register Service Put key with lease etcdctl put /services/api/node1 "10.0.1.5:8080" --lease=
3. Heartbeat Keep lease alive etcdctl lease keep-alive
4. Discover Read all instances etcdctl get /services/api/ --prefix
5. Watch Subscribe to changes etcdctl watch /services/api/ --prefix
6. Deregister Delete key (or let lease expire) etcdctl del /services/api/node1

Kubernetes Use of etcd

Kubernetes stores all cluster state in etcd:

K8s Component etcd Interaction
API Server Only component that reads/writes etcd directly
Scheduler Watches for unscheduled pods, writes binding decisions
Controller Manager Watches resource state, reconciles to desired state
Kubelet Watches for pods assigned to node

etcd vs Consul

Aspect etcd Consul
Primary Use Case General distributed KV store Service discovery and service mesh
Consensus Raft Raft (servers) + Gossip (agents)
Health Checking Must implement via leases Built-in (HTTP, TCP, script, etc.)
Multi-DC Requires separate clusters + custom sync Native multi-datacenter federation
DNS Interface No Yes (*.service.consul)
Watch Mechanism Highly efficient, low latency Blocking queries or event stream
Interview Question: Why does Kubernetes use etcd instead of Consul?
Answer: etcd is simpler and more focused on consistency and watch performance. Kubernetes needs a strongly consistent store with efficient watch APIs for its control plane. Consul's additional service discovery features (DNS, health checks) are unnecessary since Kubernetes implements those at higher layers. etcd's simplicity makes it easier to operate and reason about.

7. ZooKeeper

Apache ZooKeeper is a distributed coordination service providing configuration, synchronization, and naming services. Predates Consul and etcd but still widely used.

Architecture

graph TB subgraph "ZooKeeper Ensemble" Z1[ZooKeeper Server 1
Leader] Z2[ZooKeeper Server 2
Follower] Z3[ZooKeeper Server 3
Follower] Z1 -.ZAB Protocol.-> Z2 Z1 -.ZAB Protocol.-> Z3 Z2 -.Election.-> Z1 Z3 -.Election.-> Z1 end C1[Client 1] -->|Reads| Z2 C2[Client 2] -->|Writes| Z1 C3[Client 3] -->|Watches| Z3 note[Zab: ZooKeeper Atomic Broadcast]

Core Concepts

Concept Description Example
ZNode Hierarchical tree node (like filesystem) /services/api/node1
Persistent ZNode Remains until explicitly deleted Configuration data
Ephemeral ZNode Deleted when client session ends Service instance registration
Sequential ZNode Auto-appends monotonic counter /locks/lock-0000000001, lock-0000000002
Watch One-time trigger on znode change Notified when /config/db updates
Session Client connection with heartbeat Session timeout triggers ephemeral node cleanup
ZAB Protocol ZooKeeper Atomic Broadcast (like Raft) Leader election and state replication

Service Discovery Pattern

Step Action ZooKeeper Operation
1. Create Service Path Create persistent parent node create /services/api
2. Register Instance Create ephemeral child node create -e /services/api/node1 "10.0.1.5:8080"
3. Discover Instances List children of service path ls /services/api → [node1, node2, node3]
4. Watch for Changes Set watch on service path ls /services/api watch
5. Get Instance Data Read node data get /services/api/node1 → "10.0.1.5:8080"
6. Auto-Deregister Session timeout deletes ephemeral node Automatic on crash/disconnect

Advanced Recipes

Recipe Implementation Use Case
Leader Election Create ephemeral sequential nodes, lowest sequence is leader Distributed job scheduler (only leader executes)
Distributed Lock Create ephemeral sequential node, watch previous node Ensure only one process modifies shared resource
Barrier Nodes wait until N clients reach barrier Coordinated start of distributed computation
Queue Sequential nodes form order-preserved queue Distributed work queue
Two-Phase Commit Coordinator creates transaction node, participants vote Distributed transaction coordination

Limitations

Limitation Impact Mitigation
Single-Shot Watches Watch triggers once, must re-register Curator framework provides persistent watches
No Built-In Health Checks Session timeout is only health indicator Application must implement health logic
1MB Data Limit Cannot store large values Store references to external storage
Java-Centric Best client libraries are Java (Curator) Other language bindings less mature
Aging Technology Less active development than Consul/etcd Still very stable, but consider modern alternatives
Interview Question: How does ZooKeeper's ephemeral node model enable service discovery?
Answer: When a service starts, it creates an ephemeral znode under the service path. ZooKeeper maintains a session with the client via heartbeats. If the service crashes or network partitions, the session times out (e.g., 40 seconds) and ZooKeeper automatically deletes the ephemeral node, effectively deregistering the service. Clients watching the service path are notified immediately.

8. Kubernetes Service Discovery

Kubernetes has built-in service discovery via Services, DNS, and the API server. No external registry required.

Service Types

Type Mechanism Access Pattern Use Case
ClusterIP Virtual IP, only reachable within cluster Internal service-to-service Default, most common for microservices
NodePort Opens port on every node External access via NodeIP:NodePort Dev/test, simple external exposure
LoadBalancer Provisions external load balancer (cloud provider) External access via LB IP Production external services
ExternalName CNAME to external service Alias to external DNS name Abstract external dependencies
Headless (ClusterIP: None) DNS returns pod IPs directly, no VIP Direct pod access for stateful sets Databases, Kafka, peer discovery

Service Discovery Flow

sequenceDiagram participant P as Pod A participant DNS as CoreDNS participant API as API Server participant EP as Endpoints Controller participant KP as kube-proxy participant P2 as Pod B Note over P2: Pod B starts, joins Service EP->>API: Watch Pod changes API-->>EP: Pod B ready EP->>API: Update Endpoints P->>DNS: Resolve "backend-service.default.svc.cluster.local" DNS-->>P: ClusterIP 10.96.0.10 P->>KP: Connect to 10.96.0.10:8080 Note over KP: iptables/IPVS rules
select Pod B IP KP->>P2: Forward to 10.244.1.5:8080 P2-->>P: Response

DNS Resolution in Kubernetes

DNS Format Resolves To Scope
service-name ClusterIP (same namespace) Within namespace
service-name.namespace ClusterIP Cross-namespace
service-name.namespace.svc.cluster.local ClusterIP (FQDN) Cluster-wide
pod-ip-address.namespace.pod.cluster.local Specific pod IP Direct pod access (rare)
headless-service All pod IPs (A records) Stateful workloads

Endpoints and EndpointSlices

Concept Description Scale
Endpoints (legacy) Single object listing all pod IPs for a service Up to ~5000 endpoints per service
EndpointSlices Sharded endpoint lists (100 endpoints per slice) Scales to 10,000+ endpoints
Updates Controller watches pods, updates endpoints on change Sub-second propagation
kube-proxy Watches endpoints, programs iptables/IPVS rules Translates ClusterIP to pod IPs

Headless Services for Stateful Workloads

apiVersion: v1
kind: Service
metadata:
  name: cassandra
spec:
  clusterIP: None  # Headless
  selector:
    app: cassandra
  ports:
  - port: 9042

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: cassandra
spec:
  serviceName: cassandra  # Headless service
  replicas: 3
  # Each pod gets stable DNS:
  # cassandra-0.cassandra.default.svc.cluster.local
  # cassandra-1.cassandra.default.svc.cluster.local
  # cassandra-2.cassandra.default.svc.cluster.local
Interview Question: How does Kubernetes service discovery avoid external dependencies?
Answer: Kubernetes uses its own API server as the source of truth. The Endpoints controller watches pod changes and updates Endpoint objects. CoreDNS queries the API server for Service ClusterIPs. kube-proxy watches Endpoints and programs iptables/IPVS rules on each node. This is all internal to the cluster, no external registry needed.

9. Configuration Management

Configuration management is closely related to service discovery. Modern systems need dynamic, versioned, environment-specific configuration.

Static vs Dynamic Configuration

Aspect Static Configuration Dynamic Configuration
Storage Files, environment variables Configuration service (Consul KV, etcd, AWS AppConfig)
Updates Requires restart or redeploy Hot reload without restart
Consistency Per-instance config files, can drift Centralized, consistent across instances
Use Case Connection strings, static thresholds Feature flags, rate limits, circuit breaker thresholds
Pros Simple, no dependencies, version controlled Fast iteration, A/B testing, instant rollback
Cons Slow to change, restart downtime Complexity, dependency on config service

Configuration Patterns

Pattern Implementation Example Tools
Environment Variables 12-factor app approach, injected at runtime Docker, Kubernetes ConfigMaps/Secrets
Config Files JSON/YAML/TOML files, mounted or bundled Volume mounts, config management (Ansible, Chef)
Config Service Centralized config store with API Consul KV, etcd, AWS AppConfig, Spring Cloud Config
Feature Flags Toggle features on/off without deploy LaunchDarkly, Split.io, Unleash, ConfigCat
Secrets Management Encrypted storage for sensitive config HashiCorp Vault, AWS Secrets Manager, Kubernetes Secrets

Dynamic Configuration Strategies

Strategy How It Works Trade-offs
Polling Periodically fetch config (e.g., every 30s) Simple, but introduces lag and load
Push (Webhooks) Config service notifies on change Instant, but requires HTTP endpoint and auth
Watch/Subscribe Long-lived connection, server pushes updates Efficient, low latency (etcd, Consul)
Sidecar Agent Local agent watches config, writes to file Abstracts config service, but extra process

Configuration Versioning

Approach Benefit Implementation
Git-based Full history, code review, rollback Store configs in Git, CI/CD deploys to config service
Timestamped Versions Audit trail, point-in-time recovery etcd MVCC, Consul KV with versions
Semantic Versioning Explicit version negotiation Clients specify required config version
Blue-Green Config Test new config before switching Maintain two config sets, swap via feature flag

Hot Reload Considerations

Interview Question: When should you use dynamic configuration vs static configuration?
Answer: Use static config (env vars, files) for infrastructure-level settings that rarely change (DB endpoints, API keys). Use dynamic config for operational tuning (rate limits, feature flags, circuit breaker thresholds). Static is simpler and safer (fewer dependencies), but dynamic enables rapid iteration and A/B testing without deploys.

10. Comparison

Comprehensive Tool Comparison

Feature Consul etcd ZooKeeper Eureka Kubernetes DNS
Consensus Algorithm Raft Raft ZAB (Zookeeper Atomic Broadcast) None (AP system) N/A
CAP Theorem CP (Consistency + Partition tolerance) CP CP AP (Availability + Partition tolerance) Depends on backend (etcd)
Service Registration Self or third-party Manual via API Manual via API Self-registration Automatic (Endpoints controller)
Health Checking HTTP, TCP, script, TTL, gRPC, Docker Lease-based (manual) Session timeout (manual) Heartbeat Readiness/liveness probes
Multi-Datacenter Native federation, WAN gossip Separate clusters, manual sync Separate ensembles, manual sync Manual (requires Netflix stack) Cluster federation (complex)
DNS Interface Yes (*.service.consul) No No No Yes (CoreDNS)
KV Store Yes Yes (primary purpose) Yes (znodes) No No (uses etcd backend)
Watch Mechanism Blocking queries, event stream Efficient watch API One-time watches Polling Watch API (from K8s API server)
Service Mesh Consul Connect (native) No (but used by K8s + service mesh) No No Requires add-on (Istio, Linkerd)
Maturity Mature (2014) Mature (2013) Very mature (2008) Maintenance mode Active (part of K8s)
Ease of Operations Moderate Simple Complex Simple Managed by K8s
Language Support Excellent (HTTP API) Excellent (gRPC API) Good (Java best) Java-centric N/A (uses DNS)
Best For Multi-DC, hybrid cloud, service mesh K8s control plane, simple KV store Legacy systems, coordination primitives Spring Boot microservices (legacy) Kubernetes-native workloads

Performance Characteristics

Tool Write Throughput Read Throughput Latency Scale
Consul ~1,000 writes/sec ~10,000 reads/sec Single-digit ms 10,000s services
etcd ~10,000 writes/sec ~100,000 reads/sec < 1 ms (local reads) 100,000s keys
ZooKeeper ~10,000 writes/sec ~100,000 reads/sec Single-digit ms Millions of znodes
Eureka High (eventually consistent) Very high (cached) Low (no consensus) 10,000s instances
K8s Endpoints Moderate (etcd-backed) High (cached by DNS/kube-proxy) Low EndpointSlices scale to 100,000s

Operational Complexity

Tool Setup Complexity Operational Overhead Failure Modes
Consul Moderate (servers + agents) Moderate (monitoring, upgrades) Network partition tolerance, quorum loss
etcd Low (just cluster) Low (stateless reads, compaction) Quorum loss, disk I/O bottleneck
ZooKeeper High (Java tuning, ensemble config) High (GC tuning, monitoring) Split-brain, session storms
Eureka Low (simple server) Low (stateless, can run single-node for dev) Stale data (AP system), no strong consistency
K8s DNS None (bundled with K8s) Low (managed by K8s control plane) etcd failure affects discovery

11. When to Use What

Decision Flowchart

graph TD Start[Need Service Discovery?] --> K8s{Running on
Kubernetes?} K8s -->|Yes| K8sNative[Use Kubernetes Services + CoreDNS] K8s -->|No| MultiDC{Multi-datacenter
or hybrid cloud?} MultiDC -->|Yes| Consul[Use Consul] MultiDC -->|No| Mesh{Need service mesh
features?} Mesh -->|Yes| ConsulConnect[Use Consul Connect
or Istio/Linkerd] Mesh -->|No| Existing{Existing
infrastructure?} Existing -->|ZooKeeper already deployed| StayZK[Continue with ZooKeeper] Existing -->|etcd already deployed| StayEtcd[Continue with etcd] Existing -->|Spring Boot only| Eureka[Use Eureka] Existing -->|Greenfield| Simple{Simple
requirements?} Simple -->|Yes| SimpleDNS[DNS + Load Balancer] Simple -->|No| ConsulOrEtcd[Consul for rich features
etcd for simplicity] K8sNative --> Advanced{Need advanced
features?} Advanced -->|Cross-cluster| ConsulK8s[Add Consul for multi-cluster] Advanced -->|Service mesh| IstioLinkerd[Add Istio or Linkerd] Advanced -->|No| Done[Done]

Use Case Recommendations

Scenario Recommended Solution Why
Kubernetes-native microservices Kubernetes Services + CoreDNS Built-in, zero config, well-integrated
Multi-datacenter application Consul Native multi-DC support, WAN gossip, prepared queries
Hybrid cloud (VMs + containers) Consul Works on VMs and containers, agent-based
Need service mesh + mTLS Consul Connect or Istio Built-in service mesh capabilities
Simple distributed config store etcd Lightweight, excellent watch API, easy to operate
Leader election, distributed locks etcd or ZooKeeper Strong consistency, transaction support
Kafka, Hadoop ecosystem ZooKeeper Already required by these systems
Spring Boot microservices (legacy) Eureka Spring Cloud integration, but consider migrating
AWS ECS/Fargate AWS Cloud Map (DNS-based) Managed service, integrates with ECS
Serverless (Lambda) API Gateway + DNS No long-lived connections, DNS sufficient
Low instance churn DNS + Load Balancer Simple, no extra infrastructure
Cross-K8s cluster federation Consul on K8s or Istio multi-cluster Kubernetes DNS is cluster-scoped

Anti-Patterns to Avoid

Anti-Pattern Problem Better Approach
Running service discovery as single instance Single point of failure Always run clustered (3+ nodes)
Storing large data in service registry Performance degradation, size limits Store references to external storage (S3, DB)
No health checks, only registration Dead instances remain in registry Always implement meaningful health checks
Relying solely on DNS with high instance churn TTL caching causes stale data Use proper service registry (Consul, etcd)
Client-side discovery without retry/circuit breaker Fails hard when instance is dead Always implement client-side resiliency
No monitoring of service registry health Silent failures, discovery outages Monitor registry metrics (latency, availability, quorum)
Using Eureka for new greenfield projects Aging tech, less active development Use Consul or Kubernetes DNS
Interview Question: Your service discovery system goes down. What happens and how do you mitigate?
Answer: Impact depends on client caching. Most clients cache discovered instances for some time (e.g., 30-60 seconds). During an outage, existing connections continue working, and cached instances are used for new requests. Mitigation: (1) Run registry in HA cluster, (2) Implement client-side caching, (3) Have static fallback endpoints, (4) Retry logic with exponential backoff, (5) Circuit breakers to prevent cascading failures.

Key Takeaways