Manage Cluster Nodes

Last modified 14 Jul 2025 21:17 +02:00

Since 4.1, 4.0.2

This functionality is available since version 4.1, 4.0.2.

Table of Contents

How to Construct a Node Identifier
Node State Management
- Timeout and Check Interval Default Values

MidPoint can run in clustered mode with two or more nodes. This article describes the most important parameters influencing how nodes are named and managed.

How to Construct a Node Identifier

Each node in cluster must have a unique node identifier (name). The property to use is nodeId.

MidPoint configuration properties support expressions in the form of ${variable} or ${prefix:variable}. The first form evaluates using a configuration option specified by variable. The second one is more general and supports the following prefixes:

Table 1. Configuration properties for node identifier construction
Prefix	Meaning	Example
`sys`	References Java system properties.	`${sys:user.home}`
`env`	References operating system environment variables.	`${env:ENVIRONMENT}`
`hostname`	References local host name as determined by midPoint. The colon after hostname is obligatory.	`${hostname:}`
`random`	Generates random node ID. The full format is `${random:number:lower-limit:upper-limit}` but you can also use `${random:}`, `${random:number}`, and `${random:number:upper-limit}`. The default limits are between 0 and 999999999, inclusive.	`${random:}`
`sequence`	Uses first available node ID in a given sequence. The full format is `${sequence:start:end:format}` but you can also use `${sequence:}`, `${sequence:start}`, and `${sequence:start:end}`. Default values are: `start = 0`, `end = 100`, `format = %d`.	`${sequence:0:99:%02d}`

The sequence expression works like this:

The counter starts at the start value, incrementing by 1 up to the end value, inclusive.
At each step, the node name is determined using the formatting string and other parts of the expression, and is checked for availability.
If such a node does not exist in the repository, the name is used. Technically speaking, the node name is allocated by creating the node in the repository. If the operation succeeds, the node is acquired. This is to avoid race condition: only the first midPoint instance that successfully creates a node object can use this name.
If a node with the given name exists, but the node is permanently down (this is determined by the running property being set to false), the name is used. This is implemented by removing the node object and then retrying the allocation attempt.
Names of nodes that are not marked as down but are not alive either, are not used here. This is to avoid using names of nodes that are currently booting, or temporarily unavailable, for instance. See the Node state management section below.

The sequence expression can be combined with other name constituents. For example, you can specify the nodeId as ${env:ENVIRONMENT}-${sequence:0:99:%02d}, yielding names like Test-01, Test-02, …, QA-01, QA-02, …, Prod-01, Prod-02, …

Node State Management

A midPoint node can be in one of the following states:

Table 2. Possible midPoint node states
State	Characterization
up and alive	The node regularly checks into the repository. Its `operationalState` property is `UP` and its `lastCheckInTime` is regularly updated (less than `nodeTimeout` ago).
up, but not checking in	There’s an issue with the node. Its `operationalState` property is `UP` but its `lastCheckInTime` is older than `nodeTimeout` seconds. Nodes in this state are excluded from some operations, e.g., status querying or cache invalidation calls.
down	The `operationalState` property of the node is `DOWN`. This typically occurs when the node went down cleanly: the task manager marked it as being down. If a node goes down abruptly without having a chance to inform the task manager and get marked in the central repository DB accordingly, other nodes watch its `lastCheckInTime` in the DB. After it is older than `nodeAlivenessTimeout` seconds, the task manager marks the respective node as down by setting its `operationalState` property in the DB to `DOWN`. This check is occurring every `nodeAlivenessCheckInterval` seconds. Nodes in this state are excluded from almost all operations.
starting	The `operationalState` property of the node is `STARTING`. Nodes in this state are excluded from some operations, e.g., status querying or cache invalidation calls.
deleted	The node object no longer exists in the repository. The deletion can occur either manually or by a cleanup task. The task deletes nodes that have `lastCheckInTime` older than `deadNodes/maxAge`.

Timeout and Check Interval Default Values

Table 3. Timeout and check interval default values
Parameter	Location	Description	Default value
`nodeTimeout`	`taskManager` section of `config.xml`	When to start considering a node as not checking in.	30 seconds
`nodeAlivenessTimeout`	`taskManager` section of `config.xml`	When to start considering a node as being down.	900 seconds
`nodeAlivenessCheckInterval`	`taskManager` section of `config.xml`	How often is a node availability check carried out.	120 seconds
`nodeStartupTimeout`	`taskManager` section of `config.xml`	When to start reporting a node as starting for too long.	900 seconds
`deadNodes/maxAge`	cleanup policy, e.g., in the system configuration object	How long can a node go without check-in before it is deleted.	none^[1]

1. The nodes are not cleaned up by default. If you would like to enable this feature, you can set this parameter to 1 day, for example. Note that cleanup tasks run once per day by default. However, you can change this interval or you can schedule another cleanup task dedicated specifically to cleaning up dead nodes.

Was this page helpful?

YES NO

Thanks for your feedback