Migration of Tasks from 4.0/4.3 to 4.4

Last modified 26 Nov 2021 18:41 +01:00
Since 4.4
This functionality is available since version 4.4.
TL;DR There is a migration tool available. Besides, be sure to re-import all the tasks.

Legacy Task Definition Style

In midPoint 4.3 and before, the definition of the work done by a task was based on the following items:

  • handlerUri specifying what should be done (e.g. recomputation, import, reconciliation, and so on),

  • objectRef providing the related object - typically the resource that we should import from, or reconcile with,

  • various task extension properties defining further parameters of the work, like kind/intent/objectclass of resource objects to be dealt with, or a query specifying a set of objects to be recomputed, or execution options to be used, and so on.

Distribution of the work into threads was driven by workerThreads property. And the distribution of the work into worker tasks was specified by workManagement container that contained the specification of buckets, worker tasks, and partitions.

Error handling was driven by a task property of errorHandlingStrategy.

Thresholds were configured in quite a complex way.

New Task Definition Style

Starting from 4.4, the definition of the work and its various execution aspects is concentrated in a single place: the <activity> item.

Support of the Legacy Task Definition Style

In 4.4 the following parts of legacy task definition style are supported:

  • the work definition i.e. handlerUri, objectRef, and extension properties defining the parameters of the work,

  • workerThreads extension property providing the number of worker threads,

  • errorHandlingStrategy property.

This support is limited, though:

  • Tasks defined in this way are correctly executed, respecting the settings.

  • However, in GUI it is not possible to view or edit this kind of configuration. We are limited to the use of XML viewing and editing to do that.

As for the advanced distribution features, i.e. buckets, worker tasks, and task partitions, the legacy style of configuration is no longer supported. The users have to migrate these definitions to the new style.

Migration from the Legacy to the New Style

Migration of the Work Definition

  1. Create appropriate activity/work/X item, according to the specific type of the activity, given by handlerUri. For example, use reconciliation for reconciliation activity, import for import from resource, and so on. You can consult the list of all supported activity types.

  2. Fill-in the work/X item according to the configuration provided by objectRef and task extension properties.

An example:

Before
<task>
    <name>Import from services-inbound-pwd-copy (legacy style)</name>
    <extension>
        <ext:kind>account</ext:kind>
        <ext:intent>default</ext:intent>
        <ext:objectclass>ri:AccountObjectClass</ext:objectclass>
    </extension>
    <handlerUri>http://midpoint.evolveum.com/xml/ns/public/model/synchronization/task/import/handler-3</handlerUri>
    <objectRef oid="81c080f2-dce5-43b9-b748-a2a5fdb48c51" type="ResourceType"/>
</task>
After
<task>
    <name>Import from services-inbound-pwd-copy (new style)</name>
    <activity>
        <work>
            <import>
                <resourceObjects>
                    <resourceRef oid="81c080f2-dce5-43b9-b748-a2a5fdb48c51"/>
                    <kind>account</kind>
                    <intent>default</intent>
                    <objectclass>ri:AccountObjectClass</objectclass>
                </resourceObjects>
            </import>
        </work>
    </activity>
</task>

Migration of the Distribution Definition (Except for Partitioning)

  1. Create activity/distribution container.

  2. Migrate workerThreads property by simply copying it from the task extension to this container.

  3. Migrate workManagement/buckets item by copying it from the original position to this container.

  4. Migrate workManagement/workers item by copying it from the original position to this container.

An example:

Before
<task xmlns="http://midpoint.evolveum.com/xml/ns/public/common/common-3"
      xmlns:q="http://prism.evolveum.com/xml/ns/public/query-3"
      xmlns:s="http://midpoint.evolveum.com/xml/ns/public/model/scripting-3"
      oid="4ccd0cde-c506-49eb-9718-f85ba3438515">
    <name>Buckets, multiple threads</name>
    <extension xmlns:mext="http://midpoint.evolveum.com/xml/ns/public/model/extension-3"
               xmlns:se="http://midpoint.evolveum.com/xml/ns/public/model/scripting/extension-3">
        <mext:workerThreads>4</mext:workerThreads>
        <mext:objectType>UserType</mext:objectType>
        <mext:objectQuery>
            <q:filter>
                <q:equal>
                    <q:path>subtype</q:path>
                    <q:value>test</q:value>
                </q:equal>
            </q:filter>
        </mext:objectQuery>
        <se:executeScript>
            <s:execute>
                <s:script>
                    <code>
                        log.info('Found user {}', input)
                    </code>
                </s:script>
            </s:execute>
        </se:executeScript>
    </extension>
    <taskIdentifier>4ccd0cde-c506-49eb-9718-f85ba3438515</taskIdentifier>
    <ownerRef oid="00000000-0000-0000-0000-000000000002" type="UserType"/>
    <executionStatus>runnable</executionStatus>
    <category>BulkActions</category>
    <handlerUri>http://midpoint.evolveum.com/xml/ns/public/task/workers-creation/handler-3</handlerUri>
    <workManagement>
        <taskKind>coordinator</taskKind>
        <buckets>
            <stringSegmentation>
                <discriminator>name</discriminator>
                <boundary>
                    <position>1</position>
                    <characters>0</characters>
                </boundary>
                <boundary>
                    <position>2</position>
                    <characters>0-9</characters>
                </boundary>
            </stringSegmentation>
        </buckets>
        <workers>
            <handlerUri>http://midpoint.evolveum.com/xml/ns/public/model/iterative-scripting/handler-3</handlerUri>
            <workersPerNode>
                <count>2</count>
            </workersPerNode>
        </workers>
    </workManagement>
    <recurrence>single</recurrence>
    <binding>loose</binding>
</task>
After
<task xmlns="http://midpoint.evolveum.com/xml/ns/public/common/common-3"
      xmlns:q="http://prism.evolveum.com/xml/ns/public/query-3"
      xmlns:s="http://midpoint.evolveum.com/xml/ns/public/model/scripting-3"
      oid="4ccd0cde-c506-49eb-9718-f85ba3438515">
    <name>Buckets, multiple threads</name>
    <taskIdentifier>4ccd0cde-c506-49eb-9718-f85ba3438515</taskIdentifier>
    <ownerRef oid="00000000-0000-0000-0000-000000000002" type="UserType"/>
    <executionState>runnable</executionState>
    <activity>
        <work>
            <iterativeScripting>
                <objects>
                    <type>UserType</type>
                    <query>
                        <q:filter>
                            <q:equal>
                                <q:path>subtype</q:path>
                                <q:value>test</q:value>
                            </q:equal>
                        </q:filter>
                    </query>
                </objects>
                <scriptExecutionRequest>
                    <s:execute>
                        <s:script>
                            <code>
                                log.info('Found user {}', input)
                            </code>
                        </s:script>
                    </s:execute>
                </scriptExecutionRequest>
            </iterativeScripting>
        </work>
        <distribution>
            <buckets>
                <stringSegmentation>
                    <discriminator>name</discriminator>
                    <boundary>
                        <position>1</position>
                        <characters>0</characters>
                    </boundary>
                    <boundary>
                        <position>2</position>
                        <characters>0-9</characters>
                    </boundary>
                </stringSegmentation>
            </buckets>
            <workers>
                <workersPerNode>
                    <count>2</count>
                </workersPerNode>
            </workers>
            <workerThreads>4</workerThreads>
        </distribution>
    </activity>
</task>

Notes:

Migration of the Partitioning Definition

There are two kinds of legacy definitions using partitioning:

  • customization of predefined partitioned tasks like reconciliation or focus validity scanning,

  • creation of custom partitioned tasks.

These correspond to predefined and custom composite activities.

Migration of the Partitioning Definition (Predefined Tasks)

In this case, workManagement/partitions is translated into activity/tailoring/change.

An example:

Before
<task oid="10000000-0000-0000-565f-565600000204"
    xmlns="http://midpoint.evolveum.com/xml/ns/public/common/common-3"
    xmlns:syncext="http://midpoint.evolveum.com/xml/ns/public/model/extension-3"
    xmlns:ri="http://midpoint.evolveum.com/xml/ns/public/resource/instance-3"
    xmlns:icfs="http://midpoint.evolveum.com/xml/ns/public/connector/icf-1/resource-schema-3">

    <name>Reconciliation: Dummy Blue</name>

    <extension>
        <syncext:objectclass>ri:AccountObjectClass</syncext:objectclass>
    </extension>

    <ownerRef oid="00000000-0000-0000-0000-000000000002"/>
    <executionStatus>runnable</executionStatus>

    <handlerUri>http://midpoint.evolveum.com/xml/ns/public/model/synchronization/task/partitioned-reconciliation/handler-3</handlerUri>
    <objectRef oid="10000000-0000-0000-0000-000000000204" type="ResourceType"/>
    <workManagement>
        <partitions>
            <partition>
                <index>2</index>
                <workManagement>
                    <taskKind>coordinator</taskKind>
                    <buckets>
                        <stringSegmentation>
                            <discriminator>attributes/icfs:name</discriminator>
                            <boundary>
                                <characters>abcdefghijklmnopqrstuvwxyz</characters>
                            </boundary>
                        </stringSegmentation>
                    </buckets>
                    <workers>
                        <workersPerNode>
                            <count>4</count>
                        </workersPerNode>
                    </workers>
                </workManagement>
            </partition>
        </partitions>
    </workManagement>
</task>
After
<task oid="10000000-0000-0000-565f-565600000204"
        xmlns="http://midpoint.evolveum.com/xml/ns/public/common/common-3"
        xmlns:ri="http://midpoint.evolveum.com/xml/ns/public/resource/instance-3"
        xmlns:icfs="http://midpoint.evolveum.com/xml/ns/public/connector/icf-1/resource-schema-3">

    <name>Reconciliation: Dummy Blue</name>

    <ownerRef oid="00000000-0000-0000-0000-000000000002"/>
    <executionState>runnable</executionState>

    <activity>
        <work>
            <reconciliation>
                <resourceObjects>
                    <resourceRef oid="10000000-0000-0000-0000-000000000204"/>
                    <objectclass>ri:AccountObjectClass</objectclass>
                </resourceObjects>
            </reconciliation>
        </work>
        <distribution>
            <subtasks/>
        </distribution>
        <tailoring>
            <change>
                <reference>resourceObjects</reference>
                <distribution>
                    <buckets>
                        <stringSegmentation>
                            <discriminator>attributes/icfs:name</discriminator>
                            <boundary>
                                <characters>abcdefghijklmnopqrstuvwxyz</characters>
                            </boundary>
                        </stringSegmentation>
                    </buckets>
                    <workers>
                        <workersPerNode>
                            <count>4</count>
                        </workersPerNode>
                    </workers>
                </distribution>
            </change>
        </tailoring>
    </activity>
</task>

The subtasks item in distribution means that we want to create separate tasks for individual sub-activities of the main reconciliation activity. This is not strictly needed. We use it here just to reproduce the original pre-4.4 behavior of task partitioning.

We see that instead of using index=2 we now have reference=resourceObjects. Instead of referencing to existing partitions by number (1, 2, 3), we reference sub-activities by their identifiers (operationCompletion, resourceObjects, remainingShadows).

Migration of the Partitioning Definition (Custom Tasks)

This functionality is experimental.

In this case, workManagement/partitions is translated into activity/composition.

As an example, let us consider a task that imports first from resource A, then from resource B:

Before
<task xmlns="http://midpoint.evolveum.com/xml/ns/public/common/common-3"
      xmlns:ri="http://midpoint.evolveum.com/xml/ns/public/resource/instance-3"
      xmlns:t="http://prism.evolveum.com/xml/ns/public/types-3"
      xmlns:mext="http://midpoint.evolveum.com/xml/ns/public/model/extension-3"
      oid="3894b082-f708-46e4-b6dd-26a54a459409">
    <name>Custom partitions</name>
    <ownerRef oid="00000000-0000-0000-0000-000000000002" type="UserType"/>
    <executionStatus>runnable</executionStatus>
    <handlerUri>http://midpoint.evolveum.com/xml/ns/public/task/generic-partitioning/handler-3</handlerUri>
    <workManagement>
        <taskKind>partitionedMaster</taskKind>
        <partitions>
            <sequentialExecution>true</sequentialExecution>
            <durablePartitions>false</durablePartitions>
            <partition>
                <index>1</index>
                <taskName>Import form A</taskName>
                <handlerUri>http://midpoint.evolveum.com/xml/ns/public/model/synchronization/task/import/handler-3</handlerUri>
                <copyMasterExtension>false</copyMasterExtension>
                <extension>
                    <mext:objectclass>ri:AccountObjectClass</mext:objectclass>
                    <mext:kind>account</mext:kind>
                    <mext:intent>default</mext:intent>
                    <mext:workerThreads>5</mext:workerThreads>
                </extension>
                <otherDeltas>
                    <t:modificationType>replace</t:modificationType>
                    <t:path>objectRef</t:path>
                    <t:value oid="0e5b7304-ea5c-438e-84d1-2b0ce40517ce" type="ResourceType" />
                </otherDeltas>
            </partition>
            <partition>
                <index>2</index>
                <taskName>Import from B</taskName>
                <handlerUri>http://midpoint.evolveum.com/xml/ns/public/model/synchronization/task/import/handler-3</handlerUri>
                <copyMasterExtension>false</copyMasterExtension>
                <extension>
                    <mext:objectclass>ri:AccountObjectClass</mext:objectclass>
                    <mext:kind>account</mext:kind>
                    <mext:intent>default</mext:intent>
                    <mext:workerThreads>5</mext:workerThreads>
                </extension>
                <otherDeltas>
                    <t:modificationType>replace</t:modificationType>
                    <t:path>objectRef</t:path>
                    <t:value oid="2db718b6-243a-11e7-a9e5-bbb2545f80ed" type="ResourceType" />
                </otherDeltas>
            </partition>
        </partitions>
    </workManagement>
</task>
After
<task xmlns="http://midpoint.evolveum.com/xml/ns/public/common/common-3"
      xmlns:ri="http://midpoint.evolveum.com/xml/ns/public/resource/instance-3"
      oid="3894b082-f708-46e4-b6dd-26a54a459409">
    <name>Custom partitions</name>
    <ownerRef oid="00000000-0000-0000-0000-000000000002" type="UserType"/>
    <executionState>runnable</executionState>
    <activity>
        <composition>
            <activity>
                <order>1</order>
                <identifier>import-from-A</identifier>
                <work>
                    <import>
                        <resourceObjects>
                            <resourceRef oid="0e5b7304-ea5c-438e-84d1-2b0ce40517ce"/>
                            <kind>account</kind>
                            <intent>default</intent>
                            <objectclass>ri:AccountObjectClass</objectclass>
                        </resourceObjects>
                    </import>
                </work>
                <distribution>
                    <workerThreads>5</workerThreads>
                </distribution>
            </activity>
            <activity>
                <order>2</order>
                <identifier>import-from-B</identifier>
                <work>
                    <import>
                        <resourceObjects>
                            <resourceRef oid="2db718b6-243a-11e7-a9e5-bbb2545f80ed"/>
                            <kind>account</kind>
                            <intent>default</intent>
                            <objectclass>ri:AccountObjectClass</objectclass>
                        </resourceObjects>
                    </import>
                </work>
                <distribution>
                    <workerThreads>5</workerThreads>
                </distribution>
            </activity>
        </composition>
        <distribution>
            <subtasks/> <!-- optional -->
        </distribution>
    </activity>
</task>

Migration Using MidPoint Studio

MidPoint Studio offers an assistance to the task configuration migration: a tool that does the migration for you. (Be sure to check the task after migration, though.)

Migration of Run-Time Data

There were significant changes in run-time data related to the task/activity execution, like progress, statistics (items processed, synchronization statistics, actions executed, …​), bucket state, and so on.

Because of this, it is strongly advised that any tasks are re-imported, without any run-time information.

The only exception to this rule is substantial state information for recurring tasks, like token for live synchronization, or last scan timestamp for scanner tasks (validity, trigger, shadow refresh). Although this information can be migrated manually, it is also possible to simply keep it "as is" in its original place - i.e. in the task extension. When the task will run under midPoint 4.4, it will fetch the values from the extension, and store updated ones to the correct place.