Service Discovery

The problem

The Checkout service needs to call Orders. Yesterday Orders ran on two servers; today autoscaling started a third, and a deploy replaced all three with new ones at new IP addresses. A config file with addresses in it is out of date within minutes.

Service discovery answers one question reliably: "where are the healthy instances of service X right now?"

The registry

A service registry is a small, highly available database of instances. Each instance registers itself (or is registered by the platform) when it starts, then sends a heartbeat every few seconds. If heartbeats stop, the entry expires and callers stop seeing it.

A minimal registry with expiring entries looks like this:

import java.util.*;

class ServiceRegistry {
    private final double ttlSeconds;
    // service name -> (address -> time of last heartbeat)
    private final Map<String, Map<String, Double>> services = new HashMap<>();

    ServiceRegistry(double ttlSeconds) {
        this.ttlSeconds = ttlSeconds;
    }

    synchronized void heartbeat(String service, String address, double now) {
        services.computeIfAbsent(service, k -> new HashMap<>()).put(address, now);
    }

    synchronized void deregister(String service, String address) {
        Map<String, Double> live = services.get(service);
        if (live != null) {
            live.remove(address);
        }
    }

    synchronized List<String> instances(String service, double now) {
        Map<String, Double> live = services.getOrDefault(service, new HashMap<>());
        live.values().removeIf(last -> now - last > ttlSeconds);
        List<String> result = new ArrayList<>(live.keySet());
        Collections.sort(result);
        return result;
    }
}

import threading

class ServiceRegistry:
    def __init__(self, ttl_seconds):
        self.ttl = ttl_seconds
        self.services = {}  # service name -> {address: time of last heartbeat}
        self.lock = threading.Lock()

    def heartbeat(self, service, address, now):
        with self.lock:
            self.services.setdefault(service, {})[address] = now

    def deregister(self, service, address):
        with self.lock:
            self.services.get(service, {}).pop(address, None)

    def instances(self, service, now):
        with self.lock:
            live = self.services.get(service, {})
            for address, last in list(live.items()):
                if now - last > self.ttl:
                    del live[address]
            return sorted(live)

Client-side vs server-side discovery

In server-side discovery the caller does not talk to the registry at all. It sends the request to a load balancer or proxy at a stable address, and that component looks up the instances and forwards the request.

DNS is the simplest form of discovery: a name resolves to the current addresses. It works everywhere, but clients and resolvers cache answers, so keep TTLs short and remember that DNS alone has no health information unless the DNS server checks health itself.

Health, not just presence

A process that is running is not always able to serve. Use two kinds of checks:

• Liveness: is the process alive at all? If not, restart it.
• Readiness: can it serve traffic right now (warmed up, connected to its database)? If not, keep it out of the registry or load balancer, but do not restart it.

On shutdown, an instance should deregister first, finish in-flight requests, then exit. Otherwise callers keep sending requests to an address that is about to disappear.

Common mistakes

• Hard-coding IP addresses or host lists in config files.
• Running a single registry node, which becomes a single point of failure for every call.
• Treating "process is running" as "healthy"; use readiness checks.
• Killing instances without deregistering and draining them first.

Interview questions

Try answering out loud before you open each one.

Check yourself

Related topics

• Monolith vs MicroservicesApplication Layer
• Load BalancersEdge and Traffic
• DNS: The Internet’s Phone BookNetworking and Communication

Further reading

• Pattern: Client-side service discovery microservices.io
• Pattern: Server-side service discovery microservices.io
• Service Kubernetes documentation