Tasks Design Meetings

Task state and progress should be reported in a clear and consistent way. It should be shown how much work has been done, and how much is yet to be done. Time per item, throughput, and estimated completion time should be visible.

For tasks that run for hours or days it is crucial to see where we are on the road, and how long that road is.

There are several progress and performance indicators, e.g.

These are sometimes imprecise, or even outright conflicting. Estimated completion information is missing. It is often not clear which counters are reset on task re-execution (e.g. for live sync or async update tasks) and which are not.

Task state is misleading for multi-node tasks ("Waiting").

Errors should be reported appropriately.

Failures inevitably occur. They are not only caused by configuration issues, but often by external factors: wrong data, infrastructure outages, and so on. Failures and their effects have to be diagnosed and fixed quickly, and without inappropriate effort.

There are some mechanisms for error reporting: audit records, operation execution records, iterative task information, operation result. They sometimes complement each other, sometimes they overlap, often leading to inconsistent overall view.

Effects - i.e. actions done by the task - should be reported in a clear and consistent way.

When a task is started, administrator usually has some assumptions on the tasks effects. For example if it’s a first execution of an import task, it is expected that new users will be created in midPoint, and accounts on target resources will be created as well. By comparing expected and actual task effects the administrator can estimate if his/her configuration is good, or there is something wrong with it.

There are "actions executed" and "states of processed objects". Although they basically work, there are many corner situations in which they provide data that are either plain wrong, or - although technically correct - they are misleading.

Relevant task changes should be audited.

This is a basic prerequisite for security and troubleshooting.

Auditing is not consistent now.

Creation and configuration of the tasks should be easy.

Lower the entry barrier.

Creation of a multi-node task is currently hard and error-prone process. The XML notation is powerful but too complex for a "standard" user. No templates/wizards are available. The only way is to study the documentation and play with some samples.

Operations like task suspension, resuming, "start now", and so on, must be simple and reliable, even for multithreaded/multinode tasks.

Better user experience.

Needs improvement, especially for multi-node tasks.

Automated monitoring and management:

Situations that require human operation attention (excessive errors, slow or even stalled tasks) should be detected automatically, and appropriate notification should be carried out.

Thresholds for operations should be observed and maintained.

Administrative effort should be kept at reasonable level. For example, requiring an operator to inspect a list of tasks and to try to detect anomalies is counter-productive.

We have dashboards and notifications, but they are of limited use now. We have some diagnostics tools (like collecting thread dumps for stalled tasks automatically) but they are also limited in scope. Not all crucial information (e.g. staleness flag) is available to external clients by REST. Do we have thresholds for success/failure operations?

MidPoint should utilize multiple nodes effectively. The load should be evenly distributed, taking into account user requirements (where appropriate). Node addition and removal should be dynamically taken into account.

This is a basic requirement for scalability.

There is some missing functionality and issues with existing features. Dynamic clusters are not supported at all. As the issues are concerned, for example, when a cluster is restarted, all tasks try to execute on the first node that goes live.

The recovery from errors should be efficient, both from the point of user’s time (partially covered by Visibility:Errors) but also from the point of processing time.

The system should be able to get into consistent state (cf. eventual consistency) even when having millions or tens of millions objects in total.

We do have consistency mechanism for handling failures during outbound provisioning. It allows us to efficiently resolve (typically by retrying) failures that have occurred there. But we need a similar mechanism for inbound processing, i.e. synchronization: either live sync, import, or reconciliation. So a few failed records should be able to be fixed without the need of manual intervention or massive reconciliation effort. (An experimental work has been done in this area.)

Bucket management should be efficient and reliable. Overhead incurred should be low.

Performance increase.

But an indirect effect is that bucket configuration will be less fragile and more forgiving, improving ease of use and lowering entry barrier.

The bucket management works but requires elaborate tuning to be efficient enough. It does not always work reliably.