Improving bucket management

Bucket management is necessary to divide the work of a task among its workers. It comprises the following actions:

The first one has been traditionally the major source of contention. However, the second one produces contention as well, although (in the past) on a smaller scale.

The traditional approach (present since the beginning of bucketed tasks in 3.8) is based on the get + compute delta + modify-with-precondition + retry approach:

Actually, there are two independent retry mechanisms in play here:

The latter mechanism is related to how the repository ensures correct execution of individual deltas in the presence of concurrent modifications. For the majority of midPoint use cases, this implicit repository transaction serialization is sufficient. Unfortunately, for bucket allocation it is not, and therefore the additional collision detection and resolution method described above has been devised and implemented.

The new approach is based on so called dynamic updates, where deltas to be executed are computed within the database transaction, under the standard repository collision resolution mechanism. So the process looks like this:

All of this is executed in the repository service, using new experimental method called modifyObjectDynamically. The delta computation is done using a functional parameter provided by the caller, i.e. work state manager in this case.

There is no need for any extra collision detection and resolution. The serialization is fully ensured by the database (conflict detection) and the repository (retrying the operation).

Preliminary experiments show approximately 50% reduction of the bucketing overhead incurred, for both well-configured and slightly misconfigured scenarios. See below.

In order to compare old vs. new approach, the following scenario was set up:

Two scenarios were tested:

The machine itself had 12 physical cores, providing 24 logical CPUs in total. As the database, locally-run PostgreSQL 13.2 was used.

First we measured the results using default bucketing configuration. This implied a random delay of up to 5000 milliseconds when allocating the first bucket per worker task. However, after tests were run, the observation was that we can improve results for both old and new approach when this delay is turned off. Therefore, all the final test results are provided with this updated configuration.

We measured the time needed to allocate new buckets, marking buckets as complete, and checking that new buckets are not available. We explicitly excluded 20 second safety wait incurred by so called scavenger tasks that is implemented to allow for completion of any mis-allocated buckets. For real executions this should not play any significant role.

The bucket management time was then compared with the total wall clock time (again, excluding the final safety wait time), and the percentage - normalized to a single thread - was computed.

Each test was run five times, and the numbers were averaged.

The measured overhead is summarized in the following table.

Detailed information is in the attached spreadsheet file as well as the extracted relevant test results (old, new).^[1]

It looks like the new approach reduces the bucketing overhead for both standard and slightly misconfigured scenarios by approximately 50%. Even for the latter scenario the improved overhead is below 10%, which is most probably acceptable.

The experimental modifyObjectDynamically method might be useful to deal with such high-contention scenarios. Besides buckets, multi-node thresholds are the next candidate for its use.