> For the complete documentation index, see [llms.txt](https://docs.eventoframework.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.eventoframework.com/evento-server/evento-server-cluster.md).
# Evento Server Cluster
Several Evento Servers on one PostgreSQL event store, from one site to three continents
Evento Server can run as a **cluster**: several server instances share one PostgreSQL event store and one component catalog, and bundles move between them when a server dies or comes back. Bundles fail over *and* fail back on their own, commands can be made exactly-once across retries and servers, distributed locks are bounded, and requests are routed to the closest handler.
This section explains how the cluster works and how to deploy it:
| Page | What it covers |
| ----------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| [Bundle Failover and Fail-back](/evento-server/evento-server-cluster/bundle-failover-and-failback.md) | endpoint lists, cluster identity, what happens to requests in flight, the automatic return to the preferred server |
| [Exactly-once Commands](/evento-server/evento-server-cluster/exactly-once-commands.md) | idempotency keys, keys derived for sagas/projectors/observers, the service outcome store, caller deadlines |
| [Cells and Locality](/evento-server/evento-server-cluster/cells-and-locality.md) | one server per site, zones and `locality-preferred` routing, presence and the shared catalog |
| [Cluster Configuration](/evento-server/evento-server-cluster/cluster-configuration.md) | every server property and bundle setting that matters in a cluster, with defaults |
## How a cluster works
Several servers, one shared PostgreSQL
* **One shared state.** Every server reads and writes the same `es__events` table and the same catalog (`core__bundle`, `core__component`, `core__handler`, `core__consumer`, `core__bundle_instance`). A consumer can fetch from any server, and its checkpoint and lane lock live in the consumer state store, so a lane survives a change of server.
* **Each server routes to its own bundles.** Handler registry, forwarding table and correlations live in the server's memory, and servers do not forward requests to each other. A request reaches a handler only when caller and handler are connected to the **same** server; otherwise it fails at once with `no handler for `. This is a design choice: connectivity is regional, a client may reach only some servers, and the intended topology is a **cell** — a server with a complete set of handlers next to it. Requests stay in the cell, a failure stays in the cell, and consistency between cells comes from the shared event store.
* **Global ordering is preserved.** Appends run in one statement under a PostgreSQL advisory lock (`CPES` mode), so the sequence order equals the commit order on every server. Two commands on one aggregate can never both store an event decided from the same state: the database refuses the second one (the aggregate fence), and the server runs it again on the new state. Contended aggregates are serialised cluster-wide by distributed locks, which servers take in turn.
{% hint style="warning" %}
**Use `evento.es.mode=CPES` with more than one server** (the default). `APES` relies on a time window that does not hold across servers, and is deprecated.
{% endhint %}
## Topologies
The cluster grows one level at a time; each level keeps the rules of the previous one.
| Level | Shape | Use it when |
| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- |
| **Single server** | one server, one PostgreSQL, all bundles | development, small deployments |
| **Server HA** | 2–3 servers on one site and one database; every bundle lists the servers **in the same order**, so all of them prefer the same server and the others are hot standbys | a server crash or restart must not stop the application |
| **Cells** | one server per site (EU, AM, APAC…), each with a complete set of command and query handlers; bundles list their own cell's server first; `locality-preferred` routing | users on several sites should get answers from a nearby broker |
| **Regions as contexts** | cells plus regional application databases; events carry the region as their [context](/evento-framework/eventobundle/context.md) and each cell consumes its own regions | data must stay in its region, with a few global events |
PostgreSQL high availability (multi-host JDBC URLs with `targetServerType=primary`, a synchronous standby for the event store) applies to every level from Server HA up.
{% hint style="info" %}
Server HA is **active/passive for the bus**, not load balancing between servers: a request needs caller and handler on the same server, so all bundles of a site prefer the same one. Scale handlers out by running more instances of a bundle; the server picks among the live ones.
{% endhint %}
## Measured behaviour
The cluster mode is validated on a three-continent lab (7 regions, simulated WAN at 80–150 ms RTT to the event-store primary) with nineteen chaos scenarios: servers killed, frozen and restarted, a region cut off the network, database failovers, cold starts, clock skew and retry storms. In every one of them no acknowledged command is lost, nothing is executed twice, no aggregate forks, the read models converge and the catalog keeps every live bundle and consumer; during a region partition the other regions keep acknowledging all their commands. Details in [How Evento Is Tested](/testing-and-verification.md).
The latency of a command is dominated by round trips to the event-store primary: p50 ≈ 7 ms locally, ≈ 0.1–0.3 s in the primary's cell, ≈ 0.7 s at 80 ms RTT and ≈ 1.2 s at 150 ms RTT. Put the primary in the cell with the most writes.
## Checklist
* [ ] Every server: the same database, `evento.es.mode=CPES`, the same `evento.cluster.name`, a **unique** `evento.cluster.node.server.id` and `evento.server.instance.id`.
* [ ] Every bundle: the full list of server addresses, its own (or the site's active) server first, and `withExpectedClusterName(...)`.
* [ ] Every cell hosts every handler its callers use (commands, queries, and commands sent by sagas).
* [ ] Callers treat `BrokerChangedException` / `RequestTimeoutException` as *indeterminate* and retry commands **with an idempotency key**.
* [ ] PostgreSQL `max_connections` ≥ servers × 30 + the bundles' pools: each server uses an application pool of 20 and a lock pool of 10 (`spring.datasource.hikari.maximum-pool-size`, `evento.lock.pool.maximum-size`).
* [ ] PostgreSQL 12 or later, JDK 25.0.3 or later.
**Security:** all servers share one database, so protect it like the event store it is, and set `evento.server.bus.auth-token` so that only your bundles can join the bus.
