On Multiple Node Reconciliation

Each task handler can provide its own data structure there.

On the state item
-----------------
Naive approach: what was the latest processed object?
Issues:
 1. multithreading: we fetch objects sequentially and offer them to worker
    threads; so it might be that e.g. 1-24: processed, 25: processing,
    26: processing, 27: processed, etc.
 2. multinode/bucketing: it might be much more complex ... see below



Cassandra case nr.1:
 - bucket is defined as [start, end) interval; e.g. if we have 2^64 IDs and
   24K buckets,
    - first bucket is [0, 384307168202282);
    - second bucket is [384307168202282, 768614336404564);
    - and so on

Cassandra case nr.2:
 - bucket is defined as [start, end) just like in the previous case; but
   it also contains a list of accounts to be processed - or, at least, their
   number.
So we can track progress more precisely.

LDAP case:
 - bucket is defined as

DB case:
 - bucket is

AIS case:
 - bucket is defined as [id-start, id-end) interval; e.g. [0-1000), [1000-1999), ...
 - progress is

CDO case:
 - buckets are: 000xxxxx, 001xxxxx, 002xxxxx, ... (100.000 users in 1000 buckets)
 - optionally with the number of users within it
 - progress: last processed UOC within bucket

Recompute case:
 - buckets are: [0-1000), [1000-2000), ... - this defines users by their indices
   when doing sorted list (by username)
 - progress is last user number (or last username) processed
 - disadvantage: when users are renamed, they can migrate between buckets; and
   boundaries of buckets can move

Recompute case 2:
 - buckets are defined by usernames (from-to)
 - when users are renamed, they can migrate between buckets; however, buckets
   boundaries will stay unchanged

Recompute case 3:
 - buckets are defined by OIDs (from-to)
 - renaming users has no effect on their placement within buckets nor buckets boundaries

So we need:
- AbstractTaskBucketType (and its concrete subtypes)
    - state (initial, processing, complete)
    - identification ?
    - boundary specification
    - progress specification (indicative, operative)



On storing bucket information
=============================
1. Initial and complete buckets should take minimal amount of space
2. Buckets in progress can be more verbose

E.g. if we would have 100.000 buckets each consuming 500 bytes of information, this is:
100.000 * 200 = 20.000.000 bytes.

<bucket>
  <id>23840</id>
  <state>initial</state>
  <from>384307168202282321</from>
  <to>38430716820228299999</to>
</bucket>

<bucket>
  <id>23840</id>
  <state>allocated</state>
  <taskRef oid="..."/>
  <from>384307168202282321</from>
  <to>38430716820228299999</to>
</bucket>

alt.

<bucket>23840:initial:384307168202282321:384307168202299999</bucket>

Worker task will have currentBucket with an information about what is in progress.

----------------------

Coordinator task:
 1. create bucket information
 2. create worker tasks
 3. wait for worker tasks completion

Worker task:
 1. obtain a bucket (something in initial state)
    - if none available, exit
 2. work on it, updating it in currentBucket item
 3. when completed, update data structure in coordinator task and go to point 1

What about collisions?
Two worker tasks try to take the same bucket.
We do it like this:
1. get task
2. select free bucket (id=X)
3. modify: update bucket[X].state = allocated
   with precondition: bucket[X].state = initial

First task will succeed, second one will get precondition violation exception.

Updates should not be frequent in order to avoid contention on the master task.

Alternatively, master task can pre-allocate buckets to worker tasks; so each one could work on its share without much overhead.

Framework / things to customize:

1. bucket structure
2. initial buckets creation (java / expressions / strategy ...)
3. current bucket processing
   e.g. conversion from bucket to query


AbstractTaskWorkBucketType
 - id (container id)
 - state
 - taskRef

CassandraTaskWorkBucketType
 - from, to (long)

RepositoryOidTaskWorkBucketType
 - from, to (string)
 - records
 - processedRecords
 - lastProcessedOid

RepositoryNameTaskWorkBucketType
 - from, to (string)
 - records
 - processedRecords
 - lastProcessedName

RepositoryIndexedWorkBucketType
 - from, to (int)
 - lastProcessedItem (int)