Practical Identity Management With MidPoint

The central concept of identity management is a data record that contains information about a person. This concept has many names: user profile, persona, user record, digital identity and many more. The most common name in the context of identity management is user account. Accounts usually hold information that describes the real-world person using a set of attributes such as given name and family name. But probably the most important part is the technical information that relates to the operation of an information system for which the account is created. This includes operational parameters such as location of users home directory, wide variety of permission information such as group and role membership, system resource limits and so on. User accounts are represented in a wide variety of forms ranging from relational database records through structured data files to semi-structured text files. But regardless of the specific method used to store and process the records the account is undoubtedly one of the most important concepts of IAM field. And so are the databases where the accounts are stored as accounts, being data records, they have to be stored somewhere.

The account databases are as varied as the account types. Most account databases in the past were implemented as an integral part of the monolithic information system using the same database technology as the system itself used. This is an obvious choice and it remains very popular even today. Therefore many accounts are stored in relational database tables and similar application data stores.

Application data stores are usually tightly bound to the application. Therefore accounts stored in such databases are difficult to share with other applications. However, sharing account data across the organization is more than desirable. It makes very little sense to maintain account data in each database separately – especially if most the accounts are the same in each application. Therefore there is a strong motivation to deploy account databases that can be shared by many applications.

Directory servers are built with the primary purpose to provide shared data storage to applications. While application databases usually use their own proprietary protocol, directory servers implement standardized protocols. While databases are built for application-specific data model, directory servers usually extend standardized data model which improves interoperability. While databases may be heavyweight and expensive to scale, directory servers are designed to be lightweight and provide massive scalability. That makes directory servers almost ideal candidates for a shared account database.

Shared identity store is making user management easier. An account needs to be created and managed in one place only. Authentication still happens in each application separately. Yet, as the applications use the same credentials from the shared store, the user may use the same password for all the connected applications. This is an improvement over setting the password for each application separately.

Identity management solutions based on shared directory servers are simple and quite cost-efficient. Therefore we have been giving the same advice for many years: if you can connect all your applications to an LDAP server, do not think too much about it and just do it. The problem is that this usually works only for very simple systems.

Directory servers are just databases that store information. Nothing more. The protocols and APIs used to access directory servers are designed as database interfaces. It means that they are good for storing, searching and retrieving data. While the user account data often contain entitlement information (permissions, groups, roles, etc.), identity stores are not well suited to evaluate them. I.e. directory server can provide information what permissions an account has, but it is not designed to make a decision whether to allow or deny a specific operation. And that is not all. Directory servers do not contain data about user sessions. It means that directory servers do not know whether the user is currently logged in or not. Many directory servers are used for basic authentication and even authorization. Yet, the directory servers were not designed to do this. Directory servers provide only the very basic capabilities. There are plug-ins and extensions that provide partial capabilities to support authentication and authorization. But that does not change the fundamental design principles. Directory servers are databases, not authentication or authorization servers. They should be used as such.

However, many applications use directory servers to centralize password authentication. In fact, this is a good and cost-efficient way to centralize password-based authentication, especially if you are just starting with identity and access management. Nevertheless, you should be aware this a temporary solution. It has many limitations. The right way to do it is to use an authentication server instead of directory server. Access Management (AM) technologies can provide that.

While directory systems are not designed to handle complex authentication, access management (AM) systems are built to handle just that. Access management systems handle all the flavors of authentication, and even some authorization aspects. The principle of all access management systems is basically the same:

After that procedure, the user works with the application normally. Only the first access goes through the AM server. This is important for AM system performance and sizing, and it impacts session management functionality.

The applications only need to provide the code that integrates with the AM system. Except for that small integration code, applications do not need to provide any authentication code at all. It is the AM system that prompts for the password, not the application. This is a fundamental difference when compared to LDAP-based authentication mechanisms. In the LDAP case, it is the application that prompts for the password. In the AM case, the Access Management server does everything. Many applications do not even care how the user was authenticated. All they need to know is that he was authenticated and that the authentication was strong enough. This feature brings a very desirable flexibility to the entire application infrastructure. The authentication mechanism can be changed at any time without disrupting the applications. We live in an era when passwords are migrated to a stronger authentication mechanisms. The flexibility that the AM-based approach brings may play a key part in that migration.

Identity management (IDM) is maybe the most overlooked and underestimated technology in the whole identity and access management (IAM) field. Yet IDM is a crucial part of almost every IAM solution. It is IDM that can bring substantial benefits to almost any organization. So, what that mysterious IDM thing really is?

Identity management is exactly what the name says: it is all about managing identities. It is about the processes to create Active Directory accounts and mailboxes for a new employee. IDM sets up accounts for students at the beginning of each school year. IDM makes it possible to immediately disable all access to a suspicious user during a security incident. IDM takes care of adding new privileges and removing old privileges of users during reorganization. IDM makes sure all the accounts are properly disabled when the employee leaves the company. IDM automatically sets up privileges for students and staff appropriate for their classes. IDM records access privileges of temporary workers, partners, support engineers and all the third-party identities that are not maintained in your human resources (HR) system. IDM automates the processes of role request and approval. IDM records every change in user privileges in the audit trail. IDM governs the annual reviews of roles and access privileges. IDM makes sure the copies of user data that are kept in the applications are synchronized and properly managed. IDM makes sure data are managed according to data protection rules. And IDM does many other things that are absolutely essential for every organization to operate in an efficient and secure manner.

It looks like IDM is the best thing since the sliced bread. So where’s the catch? Oh yes, there is a catch. Or it is perhaps better to say that there was a catch. The IDM systems used to be expensive. Very expensive. The IDM systems used to be so expensive, it was very difficult to justify the cost even with such substantial and clear benefits. But that time is over now.

Identity governance is basically an identity management taken to a higher business level. The identity management proper is focused mainly on technical aspects of identity life-cycle such as automatic provisioning, synchronization, evaluation of the roles and computing attributes. On the other hand, identity governance abstracts from the technical details, focusing on policies, roles, business rules, processes and data analysis. E.g. a governance system may deal with segregation of duties policy. It may drive the process of access re-certification. It may focus on automatic analysis and reporting of the identity, auditing and policy data. It will drive remediation processes to address policy violations. It will manage application of new and changed policies, evaluate how is your system compliant with policies and regulations and so on. This field is sometimes referred to as governance, risk management and compliance (GRC).

Almost all IDM systems will need at least some governance features to be of any use in practical deployments. Moreover, many governance features are just refinement of concepts that originated in the IDM field many years ago. Therefore, the boundary between identity management and identity governance is quite fuzzy. The boundary is so fuzzy that new terms were invented for the unified field that includes the identity management proper together with identity governance. Identity governance and administration (IGA) is one of these terms. For us the governance is just a natural continuation of identity management evolution.

Back in 2010s, it was a common practice for identity governance features to be implemented by specialized products that are separated from their underlying IDM platforms. Many IDM and governance solutions are still divided into (at least) two products. This strategy obviously brings new revenue streams for the vendors. Yet, it makes almost no sense at all from customer point of view. It perhaps comes without saying that reasonable IDM solutions should offer both the IDM and governance features in one unified and well aligned product.

Identity professionals, often motivated by marketing needs, like to invent new names and use them to describe the same thing. Therefore there are many overlapping, overloaded and similar terms in use.

Identity management (IDM) is usually used to describe the low-level parts (technology), while identity governance is used to describe the high-level parts (business). Yet the boundary is very fuzzy and many IDM systems provide governance capabilities, and many governance systems provide low-level functions. Identity governance and administration (IGA) is a term supposed to describe both parts together. Governance, risk management and compliance (GRC) is a terms that was mostly used in the past to represent the high-level identity governance functionality, later known simply as identity governance. Identity security is a marketing term that roughly covers IGA functionality.

Overall, the terminology is very fluid. Vendors use their own terms, often choosing overloaded or confusing terminology. Industry analysts and consultants also adding their own terms and meanings to existing terms. Marketing terms are invented faster than the documentation can adapt, making the situation quite confusing. We have tried to compile the terminology as precisely as we could, while still making the terms understandable. We have chosen to follow established industry terminology when possible, even though many terms are overloaded and ambiguous. However, we did not want to increase the confusion by re-inventing the terminology. We are pointing out the ambiguities in the text as needed. At the very least, we are trying to use a consistent terminology in this book. When in doubt, please refer to the glossary.

A comprehensive Identity and Access Management solution cannot be built by using just a single component. There is no single product or solution that will provide all the necessary features. And as the requirements are so complex and often even contradictory it is very unlikely that there ever will be any single product that can do it all.

A clever combination of several components is needed to build complete solution. The right mix of ingredients for this IAM soup will always be slightly different as no two IAM solutions are the same. But there are three basic components that are required for any practical IAM deployment:

The following diagram shows how all these components fit together.

This is truly a composite solution. There are several components that have vastly different features and characteristics. But when bound together into one solution, the result is something that is much more than just a sum of its part. The components support each other. The solution cannot be complete unless all three components are in place.

However, building a complete solution may be quite expensive and it may take a long time. You have to start somewhere. But if you have resources for just one product then choose identity management. IDM is a good start. It is not that expensive as access management. And IDM brings good value even quite early in the IAM program. Especially the second generation IDM systems are very good at repaying the investment. Going for open source product will also keep the initial investment down. Staring with IDM is usually the best choice to start the IAM program.

Strictly speaking, Identity and Access Management (IAM) does not entirely fit into the information security field. The IAM goes far beyond information security. IAM can bring user comfort, reduce operation costs, speed up processes and generally improve the efficiency of the organization. This is not what information security is concerned with. But even though IAM is not strictly part of information security there is still a huge overlap. IAM deals with authentication, authorization, auditing, role management and governance of objects that are directly related to the information security. Therefore IAM and information security have an intimate and very complicated relationship.

It is perhaps not too bold to say that the IAM is a pre-requisite to good information security. Especially the identity management (IDM) part is absolutely critical - even though this may not be that obvious at the first sight. But the evidence speaks clearly. Security studies quite consistency rate the insider threat as one of the most severe threats for an organization. However, there is not much that the technical countermeasures can do about the insider threat. The employee, contractor, partner, serviceman - they all are getting the access rights to your systems easily and legally. They will legally pass through even the strongest encryption and authentication because they have got the keys. Firewalls and VPNs will not stop them because those people are meant to pass through them to do their jobs.

Vulnerabilities are there, obviously. And with the population of thousands of users there is a good change that there is also an attacker. Maybe one particular engineer was fired yesterday. But he still has VPN access and administration rights to the servers. And as he might not be entirely happy about the way how he has been treated the chances are he might be quite inclined to make your life a bit harder. Maybe leaking some company records would do the trick. Now we have a motivated attacker who will not be stopped by any countermeasures and who can easily access the assets. Any security officer can predict the result without a need for a comprehensive risk analysis.

Information security has no clear answers to the insider threat. And this is no easy issue to solve as there is obviously a major security trade-off. The business wants users to access the assets easily to do their jobs. To keep the wheels of an organization turning. But security needs to protect the assets from the very same users. And there is no silver bullet to solve this issue. However there is a couple of things that can be done to improve the situation:

Theoretically all of these things can be done manually. But it is not feasible in practice. The reality is that information security seriously suffers - unless there is and IDM system that brings automation and visibility. Good information security without an IDM system is hardly possible.

There is no single identity and access management solution that would suit everybody. Every deployment has specific needs and characteristics. Deployment in a big bank will probably focus on governance, role management and security. Deployment in small enterprise will focus on cost efficiency. Cloud provider will focus on scalability, user experience and simplicity. Simply speaking, one size does not fit all. Almost all IAM solutions use the same principal components. However, product choice, solution topology and configuration will significantly vary. Do not expect that you download a product, install it and that it will solve all your problems. It won’t. Customization is the key.

We consider identity management to be heart and brain of any IAM solution. This is one of the reasons why we have started midPoint project. The rest of this book will focus almost exclusively on identity management and the use of midPoint as the IDM component. This is the place where theory ends and practice begins.

MidPoint does what any identity management system is supposed to do: it manages identities. The very basic functionality of midPoint is the synchronization of identity data that are stored in various applications, databases, directory servers, text files and so on. We call all these systems identity resources or resources for short. MidPoint is using connectors to reach the resources. MidPoint can propagate a change that happened in one resource to other resources. E.g. an employee record appears in the HR system, it is picked up by midPoint, processed, transformed and then new Active Directory and CRM accounts are created. This is the process that we call synchronization.

MidPoint has a rich graphical user interface (GUI) that can be used to manage the identities. Changes made by system administrators are automatically propagated to all affected resources. E.g. security officer disables a user by clicking on disable button in midPoint user interface. Then MidPoint makes sure that all accounts that belong to the user are immediately disabled.

This is the essence of midPoint operation. This may sound simple, but this description is extremely simplified. The reality is much more complicated. Most of the important things happen inside midPoint before the changes are applied to target resources. For each change that midPoint detects it needs to evaluate:

This is a lot of things to process, evaluate and execute. Some of these steps are quite complex. And indeed there are many complex algorithms implemented in midPoint. There are algorithms that evaluate complex role structures, organizational structures, temporal constraints, password policies and so on. The only thing that is needed is to configure them properly.

However, midPoint does even more than that. MidPoint does not only manage identities, it can also manage any object that is anyhow related to identity management. MidPoint can manage roles, role catalogs, organizational structures, groups, projects, teams, services, devices and almost any other object.

MidPoint is also an identity governance system. The job of identity management features is to make sure that the policies are consistently applied through the organization. The governance features assist with the maintenance and evolution of those policies. MidPoint implements access certification process. This is a recurring process that asks managers to confirm that the users still need the privileges that they have previously received. MidPoint contains mechanism to sort roles into hierarchies and categories. That is necessary to maintain order during role engineering and maintenance of role definitions. MidPoint has mechanisms for selective enforcement of role which comes very useful during migrations and when new system is connected to midPoint. MidPoint has support for policy lifecycle, general policy rules and so on. And more work in that direction is planned in future midPoint versions. We fully understand that it is not enough to simply apply the policies. Policies are living things and they need to evolve.

This book is about practical identity management. Therefore, we will get very close to a practice by demonstrating midPoint features using a case study. This is a case study of a fictional company ExAmPLE, Inc. The name stands for "Exemplary Amplified Placeholder Enterprise". ExAmPLE is a mid-sized financial company. Its operation heavily relies on information technologies, therefore there is a diverse set of applications and information systems ranging from legacy applications to cloud services. As ExAmPLE has few thousand employees and there is a good potential for growth the management has decided to start an IAM program. The first step of the program is deployment of midPoint as the identity management system.

Eric is an IT engineer at ExAmPLE, Inc. He has taken the responsibility to install and configure midPoint. Eric spins up a new Linux virtual machine for midPoint. He downloads midPoint distribution package and follows the installation instructions. Couple of minutes later midPoint instance starts up. Eric logs in to the midPoint user interface.

MidPoint instance is almost empty after fresh installation. It contains only a couple of essential objects. But Eric is a smart engineer. He has already read through this book, and he knows exactly what he needs to do.

First thing to do is to populate midPoint with employee data. The primary source of ExAmPLE employee data is an HR system. The HR system is quite big piece of software, it is not easy to connect to that system directly. Fortunately, it is quite easy to get a text export of the employee data in comma-separated (CSV) format. Eric plans to use this file to get employee data to midPoint.

MidPoint communicates with all the source and target systems by the means of connectors. Connectors are relatively small Java components that are plugged into midPoint. There is usually one connector for each type of the connected system. There are connectors for LDAP servers, Active Directory, databases, UNIX operating systems and so on. The responsibility of a connector is to translate protocols. E.g. LDAP connector will translate midPoint search commands to LDAP search requests. The UNIX connector will create an SSH session and translate midPoint create command to the invocation of Linux useradd binary. Database connector will translate the requests to SQL language. And so on. Each connector talks using its own communication protocol on one side. But on the other side the connectors translate the information to a common format that is understood by midPoint.

There is no distinction between source and target system when it comes to the connector. The same connectors are used for source and target systems. The difference is only in midPoint configuration.

The connectors are distributed as Java binaries (JAR files). To deploy them to midPoint you just need to place them in the correct directory and restart midPoint. MidPoint will automatically discover and examine the connectors during start-up. A handful of frequently used connectors is bundled into midPoint distribution. These connectors do not need to be deployed. They are automatically available.

Connector of a specific type works for all the systems that communicate by the protocol supported by connector. E.g. LDAP connector works for all the LDAP-compliant servers. Connector is a very generic piece of code. It does not know the hostname, port or passwords that are needed to establish a connection to a particular server. The configuration that specify connection parameters for individual server is stored in special configuration object called resource. The term resource in midPoint terminology generally means any system which is connected to a midPoint instance.

Therefore, Eric the Engineer needs to define a new resource in midPoint in order to get ExAmPLE employees into midPoint. This resource represents the CSV file exported from the HR system. MidPoint distribution contains CSV file connector already, therefore there is no need to deploy the connector binary package. All that Eric has to do is to create a new resource definition. There are (at least) two ways to do it. Firstly, there is a configuration wizard in midPoint user interface. Eric can use the wizard to configure a new resource from scratch. As you will see later in this book, the resource definition is quite complex, it has many configuration options. The abundance of options and choices makes the configuration wizard very rich, and the wizard may be quite confusing for new users. Therefore it is better for Eric to use the other approach: start from an example. There are examples of various resource definitions in the midPoint distribution package, and even more examples are available on-line. Eric quickly locates an XML file that contains a complete example of a CSV resource. He edits the file to change the filesystem path to his CSV file and adjusts the names of the columns to match the format of his file. The very minimal resource configuration specifies just the resource name, connector and connector configuration. The XML file that Eric creates looks approximately like this (simplified for clarity):

Then Eric navigates to Configuration section of midPoint user interface and imports the XML file into midPoint. Import operation creates new resource definition in midPoint. Eric now navigates to Resources section of the midPoint user interface. The new CSV resource is there. When Eric clicks on the resource name a resource details screen appears.

Eric can click on the button at the bottom of the screen to test connection to the resource. As this is a local CSV file, there is no real connection. The test checks that the filesystem path is correct, that the file exists and that it can be opened. The connector reads the CSV file header to discover the structure of the data in the CSV file, and presents the information to midPoint. This is stored in midPoint as resource schema, which describes structure of accounts in this resource. The resource is now prepared for use.

There is not much that Eric can do with the resource yet. We need to explain a couple of essential midPoint concepts before moving forward with our case study.

The concept of user is perhaps the most important concept in the entire IDM field. The term user represents physical person: an employee, support engineer, temporary worker, student, teacher, customer, etc. On the other hand the term account refers to the data structure that allows the user to access applications. This may be an account in the operating system, LDAP entry, row in the database table and so on. Typically, one user has many accounts – usually one account for each resource.

The data that represent users are stored directly in midPoint. While the data that represent accounts are stored "on the resource side". Which means accounts are stored in the connected applications, databases, directories and operating systems. Accounts are not stored in midPoint. Under normal circumstances MidPoint keeps just account identifiers and some meta-data about the accounts. All other attributes are retrieved when needed. MidPoint is using the connectors to fetch account data.

We will strictly distinguish the terms user and account in this book. Such a strong distinction is also made in the midPoint user interface and documentation. It is very useful to get used to this terminology.

Accounts are linked to users that own the accounts. MidPoint knows which account belongs to which user. MidPoint can list all the accounts for any user, it can synchronize the data, it can disable all the accounts of a particular user and so on. This user-account link is almost always automatically established and maintained by midPoint.

MidPoint comes with a built-in data model (schema) for users. It contains properties that are very often used to describe users such as full name, e-mail address and telephone number. There is a reasonable set of properties that should be a good starting point for most deployments. Of course, as most midPoint objects, the user schema can be extended with custom properties if needed.

However, there is no built-in data model for accounts. Such data model would not be possible. Every resource may have different account attributes. There may be different names, different types and the values may have different meaning. MidPoint is designed to handle those differences. Schema for resource accounts is dynamically discovered when midPoint connects to the resource for the first time. MidPoint interprets the schema and automatically adapts to it. E.g. when midPoint displays information about an account, the user interface fields are dynamically generated from the discovered schema. MidPoint does that all by itself. No extra configuration and no coding is necessary.

Account schema may significantly differ from resource to resource. Yet midPoint must be able to synchronize all the accounts from any kind of resource imaginable. In this case the user schema works as a unified data model. The schema of each account is mapped to the user schema.

Getting back to our ExAmPLE story, Eric has an HR resource configured. Now he can see the "accounts" that the users have in the HR system. Eric opens the resource detail page in the midPoint GUI, clicks on Accounts tab and then on the Resource button (we’ll explain that later). The list of accounts appears:

All that can be seen in this list are just employee numbers, because employee number is set as the primary identifier for the HR system. Clicking on the link will display more details. In fact these are not real accounts. These are lines in the CSV file exported from the HR database. But they describe some aspects of identity and therefore midPoint interprets them as accounts. For midPoint, "account" is a generic term used to describe any resource-side data structure that represents the user.

The user is a central concept for any IDM system and midPoint is no exception. MidPoint needs reliable information about users to work correctly. The HR system is usually a relatively good source of user information. Eric needs to get that information from the HR system into midPoint. He has already set up a resource that connects to the CSV file exported from the HR system. But the resource does not do anything by default. It has to be configured to pull the information from the file into midPoint. What Eric needs is a set of mappings. Mapping is a mechanism for synchronization of attribute values between user and linked accounts. In this case Eric needs inbound mappings to import the data. Inbound mappings synchronize the value in the direction from the resource into midPoint. Eric can open the resource definition in the configuration wizard in GUI and he can add the mappings there. Or he can simply look at the configuration samples again and add the mappings in the XML form. Inbound mapping looks like this:

This is a mapping that maps the account (HR) attribute firstname to user (midPoint) property givenName. This tells midPoint to always update a value of user’s given name when the mapped HR attribute changes. Eric adds similar mappings for all the attributes in the HR export file. Eric also needs to add synchronization section to the resource definition. The synchronization section instructs midPoint to create a new user for each new account. This is exactly what Eric wants: create a user for each HR account. Eric then re-imports the modified XML file into midPoint.

MidPoint is now ready to synchronize the attributes. But we still need a task to pull all the data from the HR system. Eric navigates to the page that shows the list of HR accounts. At the bottom of that page there is a big Import button that can be used to manage the import tasks. Eric clicks on that button and creates a new import task. The task is started, and it runs for a couple of seconds. After the task is done Eric can look at users in midPoint:

Eric can see details about the user by clicking on the username:

This page shows all the details about the user that midPoint knows about.

The details are sorted into several sections. We are going to explain all of that later in this book. For now, we only care about first two sections. The Basic section shows user properties as midPoint knows them. These properties are stored in midPoint repository. MidPoint has quite a rich data model that can be used out-of-the-box, but the GUI only shows those properties that are actually used. The "name", given name and family name were imported from the HR resource and that’s what the page shows.

Let’s have a look at the second tab now:

The Projections tab shows user’s accounts. Currently, there is only one account. It is the HR account that was used to import the data. Account details are displayed by clicking on account identifier:

The data that are displayed here are really fresh. Account details were retrieved from the resource at the very moment that the account was displayed. This is the difference between user data and account data: user data are kept in midPoint repository, while account data are retrieved from the resource as needed.

The user and the account are linked. MidPoint remembers that this user originated from this specific HR account. If the HR account is modified then the change is synchronized and applied to the user data. The mappings are not just for the import. They can work continually and keep the account and user data synchronized all the time.

The concepts of an account is all about the reality: it shows the data that are there at this very moment. It shows what is there. But identity management is all about policies. Policies, by definition, specify what should be there. Policies specify what is right. But as every citizen knows all too well, the things that are and the things that should be do not always match perfectly. We are no idealists. Therefore we have designed midPoint from the day one to acknowledge that there may be a difference between reality and policy. Primary role of midPoint is to manage that difference. And completely align policy and reality in the long run.

This kind of thinking is easy to see in midPoint user interface. There is Projections tab in the user details page. It shows the accounts that the user has right now. It shows the real state in which the accounts are. It shows the reality. And then there is Assignments tab. This tab shows the policy. This tab shows what accounts, roles, organizations, or services are assigned to the user. This tab shows what user should have.

To demonstrate how the assignments work we need a new resource. Therefore, let Eric connect a new resource to midPoint. This time it will be new, clean and empty LDAP server. So Eric once again locates the proper example, modifies the configuration and imports it to midPoint. In a while there is a new LDAP resource. Eric wants to synchronize all the users to the LDAP server. To do that, Eric has to define mappings once again. But this time these will be outbound mappings as Eric wants to propagate data out of midPoint into the (LDAP) resource. We will cover the details of mapping configuration later, so now let’s just see the results. We have two resources now:

But how do we create an account on that LDAP resource? The right way to do this is to let midPoint know that a user should have an account on that resource. In midPoint terminology we say, that we are assigning the resource to the user. All that Eric needs to do is to navigate to user details page, click on the Assignments  Resource tab, use New button to add an assignment for the LDAP resource and click Save:

After the click on Save button a lot of complex things happen. But simply speaking midPoint recomputes what the user should have and what the user has. MidPoint detects that the user should have an LDAP account now (because there is a new assignment for it). But no such account exists. Therefore, midPoint creates the account.

When Eric opens the user details again and navigates to the Projections tab he can see that there are two accounts now:

There is an HR account that was used to create the user in the first place. And there is also LDAP account that was created as a reaction to a new assignment.

There is reality and there is policy. There are accounts and there are assignments. Ideally these two things should match perfectly. And midPoint will try really hard to make them match. But there may be exceptions. Careful reader surely noticed that there is HR account but there is no assignment for that account. And yet midPoint has not deleted the HR account. That is because the HR system is what we call a "pure source" system. MidPoint does not write to the HR, it only reads from it. Writes to the CSV export file would be overwritten by the next export anyway, so there is no point in writing there. Therefore the HR resource has an exception specified in its configuration: it allows the HR account to exist even if there is no assignment for it. We can keep the HR account linked to the user by using this method. We can see the data that were used to create the user. This improves overall visibility and it greatly helps with diagnostics of configuration issues.

It would be a daunting task if Eric had to assign every individual account for every individual resource to every user. Typical IDM deployment has thousands of users and dozens of resources. Such deployment would be very difficult to manage using only direct resource assignments.

But there is a better way, of course. We can define some roles. The concept of role-based access control (RBAC) is a well-established practice and the roles are really the bread-and-butter of identity management. The basic idea of RBAC is to group privileges into roles. Then the roles are assigned to the users instead of privileges. E.g. let’s create a Webmaster role. Then put all the privileges that webmaster should have into that role. And let’s assign the role to every user that works as a webmaster. This simplifies the privilege management. If there are two webmasters there is no need to think about the individual privileges that a webmaster should have. Just assign the role and the role has everything that is needed. It is also easy to change webmasters: unassign role from one user, assign to another user. It is also easy if you add a new web server. Just add the privilege for accessing new server into the Webmaster role. And all webmasters will have it.

That’s the theory. But how does it work for Eric? First of all, let’s add a handful of new resources – to get some material for the roles. So now we have four resources: HR, LDAP, CRM and Portal. That’s a good start. Let’s do some role engineering now.

Many organizations have one role that almost every user has. It is often Employee or Staff role. This role gives access to all the systems that an employee should have access to: Windows domain login, e-mail, employee portal – things like that. The ExAmPLE company is no exception. In this case the basic role should create accounts in two systems:

It is simple to create such role in midPoint user interface. Eric navigates to Roles  New role. Fills in the name of the new role (Employee). Then he goes to the Inducements  All tab. This is where the role definition takes place. Inducements are almost the same as assignments. However, inducements do not give access to the role itself. Inducements give access to the users that have this role. So they are kind of indirect assignments. Eric clicks on New inducement button and adds inducements for the two resources into the role:

Eric clicks on Save button and the new role is created. Now it is ready to be assigned to the users. Eric goes on and assigns Employee role to user Bob:

MidPoint automatically creates all the accounts given by the role:

There is the HR account that was used to create the Bob user record in the first place. And then there are the two accounts that were created because Bob has the Employee role.

This operation works in both directions: if Eric unassigns the Employee role, the accounts given by the role will be deleted. Eric can create any number of roles like this: roles for Sales agents with CRM access, roles for Sales managers with higher CRM privileges and so on. MidPoint is designed to handle large number of roles. Each role can have its own combination of resources. MidPoint seamlessly merges the privileges given by all the roles a user has. E.g. if two roles give CRM access to the user, only one CRM account will be created. If one of these roles is unassigned then CRM account remains there. The account is not deleted yet because it is given by the other role. Only when the last CRM role is removed that’s the point where the account gets deleted. MidPoint takes care of all that logic.

Of course, there is much more that the roles can do:

Roles are really the essence of identity management. We will be dealing with roles in almost all the parts of this book.

Eric the Engineer has done a few basic steps to configure midPoint as an identity management system for his company. But this is still a very basic configuration. Careful readers have already noticed a lot of things that need to be done. E.g. employee full name is not automatically generated. Employee numbers are used as identifiers and we would like to have something that is more user-friendly. We would like to automatically assign the Employee role instead of doing that manually. And so on. There are still a lot of things to improve. Fortunately, all of that is very easy to do with midPoint once you know where to look. And we will be dealing with all these things in the rest of this book. New functionality will be administered to the ExAmPLE solution in small doses in each chapter – together with a proper explanation of midPoint principles. MidPoint is a very flexible and comprehensive system and there are still a lot of things to learn. This chapter covered only a minuscule part of midPoint functionality.

Now you probably have some idea what midPoint is. However, it is also very important to understand what midPoint is not. Identity and Access Management (IAM) field is a combination of many technologies and it may sometimes be quite confusing. That is perhaps the reason why the midPoint team occasionally gets questions about midPoint functionality that simply do not make much sense.

First of all, midPoint is not an authentication server. MidPoint is not designed to validate your username and password. Yes, midPoint maintains data about users (including passwords). But the data model that midPoint maintains is quite complex. It is not meant to be exposed to applications directly. That would not be efficient.

If you want midPoint to manage users but you also want your applications to have a centralized authentication services there is a solution: publish the data to the LDAP server. Connect LDAP server to midPoint as a resource and let midPoint populate and maintain the LDAP sever data. The application will not talk to midPoint directly. They will talk to the LDAP server. This is better for everybody: LDAP is a standard protocol well supported in many applications. LDAP servers are also extremely fast and scalable ad nauseam. Therefore use the combination of midPoint and an LDAP server of your choice. That’s what people usually do and it works perfectly.

As midPoint is not an authentication server it obviously is not a Single Sign-On (SSO) server either. If you want SSO you will need a dedicated SSO server. There are plenty of SSO servers to choose from in both the closed-source and open-source worlds. You will also need a scalable directory system (LDAP) to store the data for the SSO server. And you will probably still need midPoint to manage the data.

One of the things that seems to be shrouded in a lot of confusion is authorization. To get the record straight from the beginning: midPoint is not an authorization server. It is not a policy decision point (PDP) and it definitely is not a policy enforcement point (PEP). You cannot rip authorization out of your application and just “use midPoint for that”. That does not work.

You can think of midPoint as a policy management point (PMP). MidPoint has a lot of sophisticated authorization-related logic inside its core. But that logic is not designed to answer questions such as "Is subject S authorized to execute operation O on object X?". MidPoint logic is different. MidPoint is not concerned with making authorization decisions. It is concerned about managing the authorization policies. MidPoint sets up the authorization policies in target applications. And the applications evaluate these policies themselves. This is a much more efficient and more reliable method. Unlike authentication, the authorizations decisions are done all the time. Authorization is evaluated at least once per every request. If the application makes these decision internally then there is no need to a round-trip to the authorization server. Performance is significantly increased. And there is no single point of failure. MidPoint failure will not interrupt authorization flow because the application has all the data inside. One less component to cause a failure. And still, the policies are centrally managed by midPoint. When a policy changes midPoint updates all the affected applications. You get all the benefits without the usual drawbacks.

MidPoint does what it is supposed to do: it manages identities, entitlements, organizational structures and policies. But midPoint does not do things that are not necessary. It does not do the things that other technologies already do well. MidPoint does not reinvent the wheel. There is no need for this. MidPoint is not the wheel. MidPoint sits above all the wheels. MidPoint is the chauffeur.

MidPoint will run on almost any machine. All you need is approximately 4GB RAM. That’s perhaps the only real limiting factor. If you look for more formal system requirements definition, then you will find that in midPoint docs (see Additional Information chapter at the end of the book).

From the software side you will need:

We recommend using the latest available versions of all software packages when dealing with midPoint. We are trying really hard to always keep midPoint up-to-date with the rest of the technologies.

MidPoint is Java web application distributed in a stand-alone package. The distribution package contains everything that midPoint needs to run – except for Java platform itself. Therefore, as long as the Java platform is installed all that is needed to run midPoint is to start it:

And that is pretty much it. MidPoint will start. It will initialize the embedded web container, database and all the other midPoint components. That can take a minute or two. After the application is initialized you can access it by connecting to midPoint HTTP port, which defaults to 8080. You can start working with midPoint now.

Use the following URL to access midPoint user interface:

http://hostname:8080/midpoint

Log in with the following credentials:

Username: administrator
Password: 5ecr3t

Now you are logged-in as the administrator. This user has superuser privileges therefore you can see everything, and you can do anything in the midPoint user interface.

MidPoint user interface is structured. It has the same layout and controls for all the user interface areas:

Primary tool for user interface interaction is the menu. MidPoint user interface is functionally divided into three parts, therefore there are also three parts of the menu:

The menu can be hidden by clicking on the button at the top of the screen. The top bar also contains the title of the current user interface page and breadcrumbs. Breadcrumbs show where you currently are in the user interface and how you got there. The breadcrumbs can be used to "find your way home" and back to the previous page. The use of browser Back button is not recommended. Please use the Back buttons that are present in midPoint user interface or use breadcrumbs.

MidPoint user interface is quite rich. Following list provides short description of the most important parts of the user interface.

MidPoint user interface is using the same concepts and controls in all its parts. For example all the lists of all the objects (users, roles, …​) look like this:

Each row represents one object: user, roles, service, task, etc. There is also a color-coded object icon. The search bar at the top can be used to look for a specific object or to filter the object view. Right side is reserved for action buttons. Buttons in the table header trigger actions that apply to all selected objects. Buttons in each table row trigger actions that apply only to that individual object. The buttons in the bottom-left corner execute global actions, such as creating or importing new object, exporting objects and refreshing the view. Paging controls are in the bottom-right corner.

MidPoint has a unified color-code that makes the navigation easier. Users, roles and other object types have their specific color and icon. This indicates the object type and it is used whenever possible: menu, information boxes, lists, box title accents and so on. The primary colors and icons are shown in the dashboard:

All user-related controls are red, all controls that deal with organizational structure are yellow. Roles are green. And so on. This color code is applied consistently through the midPoint user interface.

Similar color code applies to object icons when displayed in user lists. However, the color that is used there does not indicate object type but rather an archetype. Archetypes are sub-types that are often used to distinguish similar objects. For example archetypes can be used to sort users to employees, contractors, customers and so on. Look and behavior of "archetyped" objects is configurable. Default midPoint configuration contains just a couple of archetypes. Those archetypes apply red color to system users and roles. Objects that do have any archetype behave in a different way. Their color indicate status of the object:

When a user clicks on a name of any object in the object list then object details page appears. The detail pages for common midPoint objects such as user or role are very similar to each other. They have the same layout and controls. E.g. user details page looks like this:

The screen below the information area is divided into several panels. There is a summary panel at the top of the page. This is an information area which shows user photo (or icon) and provides some basic information such as user identifier and title. It also shows where the object belongs in the organizational structure. There is also a couple of "tags" that show interesting details about the object: whether the object is enabled, whether it has special privileges and so on.

Below the summary panel, there is a panel with operation controls. There are buttons to save the changes, go back to the previous page, controls to set operation options and so on.

The data are displayed in details panel. However, midPoint data structures are often quite complex. Displaying the data in their full complexity would make it very difficult to understand. Therefore, the data are divided into several tabs, each tab containing a portion of the data. Navigation tabs are displayed left to the main details panel. The tabs allow easy navigation through data structures.

The first tab is perhaps the most interesting one. It contains fields that show basic object properties: the attributes of the object. Properties are displayed, and they can be edited – depending on the authorizations of currently logged-in user. The other tabs shows more complex information about the object. E.g. activation tab shows whether the object is enabled or disabled, it shows the activation dates and other activation details. The password tab provide password management functionality. And so on.

The view control buttons present in the details panel can be used to adjust the way the information is presented. These buttons control sorting of the fields, can toggle metadata display and there may be additional control buttons for more complex fields. MidPoint shows only some fields by default, to make the presentation easier to read. For example, most empty fields are not shown by default. There is a small Show empty fields button under the fields than can be used to display empty fields.

The content of the details panel changes its form, adapting to the type of displayed information. Basic properties are shown as a series of editable fields. Content of some tabs is similar to the Basic tab, displaying a set of fields. Other tabs provide lists of more complex data structures such as projections, assignments and personas.

MidPoint user interface often needs to present objects that are internally quite complex. It does not make sense to present all the details at once. These objects need to be presented in quite a compact form that can be expanded to show the details. This applies to list of user’s accounts, assignments, role inducements, etc. The objects are initially displayed as in a form of a simple list, displaying only the basic data:

The list above shows user’s projections. Those are usually accounts that are linked to user object. Click on account name shows account details:

Account details display is shown in place of user details. This may be slightly confusing. But account details can be often complex, therefore all the available screen space is needed to display them. The Cancel and Done buttons at the bottom can be used to return back to list of projections. Click on Done button will not start the operation yet, it only changes the view. Therefore, do not confuse those buttons with Back and Save buttons located in the control panel.

Each tab provides the information in an editable way, provided that the user has adequate privileges to edit the information. When the data are edited, the changes can be confirmed by using the Save button in the operations panel. Saving the changes is a universal way how to start almost any operation: change of user properties, assignment of roles, change of password, user disable, etc. When you make edits in any of the tabs on the details page then nothing really happens yet. MidPoint just remembers what you are editing. The operation is executed only when you click the Save button. This is our method how to execute several changes in one operation. It may require some time to get used to it. Just do not forget to click the Save button.

Operation options are used to modify the behavior of the operation. These options may force execution of operations that fail to pass midPoint internal checks. There is an option to reconcile the data even if midPoint thinks that reconciliation is not needed. And so on. Checking or unchecking these options influences the way how midPoint executes the operation.

The principle of midPoint configuration is quite different from what would a typical system administrator expect. There are almost no configuration files in midPoint. MidPoint is storing vast majority of its configuration in its configuration database. There are several reasons for this:

Therefore midPoint has a completely different approach to configuration. The configuration is stored in a form of well-defined structured objects in the midPoint database. We call that database midPoint repository.

Everything is an object in midPoint. Every piece of configuration is represented as a structured object and stored in midPoint repository. Object may look like this:

Every object has its identifier. We call that identifier OID which stands for "object identifier" (it has nothing to do with LDAP or ASN.1 OIDs). OID is usually randomly-generated universally unique identifier (UUID). OID value has to be unique in a whole system. This identifier is persistent – it is assigned to the object and it should never change. OID is used for internal purposes and it is almost never displayed to midPoint user.

Every object has a name. Name is human-readable and it can change any time. The value of name is usually displayed to users. This is the values that ordinary users understand as an identifier.

And then there are other object properties. Or rather items. Each type of midPoint object has a slightly different set of these items. That’s what we call schema. The items may be simple properties such as string, integer or boolean values. But there also may be complex structures and references between objects. MidPoint data model is quite rich. It is in fact so rich that its description will take better part of this book, because description of the data model is also description of midPoint features.

You can see midPoint configuration objects in midPoint user interface by navigating to Configuration > Repository Objects and selecting object type. The following picture shows objects of type "Role":

The objects are stored in the midPoint repository in a native form which is hidden from midPoint users. However, the objects also have a human-readable representation. They can be represented in XML, JSON and YAML forms. All the objects can be imported into midPoint in any of those forms. They can be exported from midPoint in any of those forms. They can be even edited directly in midPoint using embedded editor. Just click on any object in the Repository objects page:

The ability to export, import and edit objects in XML/JSON/YAML form is absolutely essential, because:

Therefore the midPoint configuration has the best of both worlds. It is stored in database, so it can be processed efficiently, it can be made available and so on. But it also has a text form, so it can be easily managed.

The XML, JSON and YAML forms are considered to be equivalent. Objects can be written in any of these forms.

Most of the examples in this book are in XML notation. The XML form is almost always simplified for clarity: there are no namespace definitions, no namespace prefixes and so on. The complete files with all the details can be found in midPoint distribution package, midPoint source code or in other places. See Additional Information chapter for more details.

When it comes to maintenance of midPoint configuration there are two practical methods how to do that.

First method is to maintain the configuration in midPoint: use midPoint wizards and user interface tools to create new objects and modify them. Export the objects in regular intervals, so they are backed up, placed under version control and so on. This is an easy method to start with. But sooner or later you will probably figure out that you need the ability to copy and paste parts of the configuration. That you need to share the configuration with other team members. And that no user interface is ever as efficient as an experienced engineer with a good text editor.

Then there is a second method: maintain the configuration files in text form outside midPoint. Import them to midPoint as needed. The objects can be imported in midPoint user interface by going to Configuration  Import object page. There are also import buttons in almost all the object list tables that also lead to that page.

It is much easier to maintain a proper version control and a good teamwork using the import method. It also seems to be more efficient once you get used to midPoint: pieces of configuration can be copied from samples, documentation or from other projects. This makes the work efficient. Although work with midPoint is "just" configuration and there is usually almost no programming, this method of work is quite close to the methods that software developers use. And we know that these methods work quite efficiently.

If you maintain the configuration files out of midPoint, you can import them individually using midPoint user interface. This may seem like quite an uncomfortable way. But it works surprisingly well even for a mid-size projects. However, there is also a much better way. There is MidPoint Studio, an integrated development environment (IDE) based on IntelliJ IDEA. MidPoint Studio allows you to maintain the configuration files in form of development project. You can edit the files with all the usual luxury of IDE such as syntax highlight and autocompletion. Studio allows easy download and upload of changed configuration files to midPoint instance. As the IntelliJ platform has good integration for version control systems and other development tools this seems like an ideal approach for large and complex projects.

Now we have a running midPoint installation, and you should have some understanding of how to configure it. But before we plunge into the details about configuration let’s have a look at midPoint installation. There are few things that need to be understood before going on. It will save a lot of time later on.

MidPoint needs its own database to work. We call that database midPoint repository. The database is used to store the configuration, users, resource definitions, account links, audit trails and a lot of other things. Proper relational database (PostgreSQL) is strongly recommended for production deployment. But for development and demonstration purposes midPoint contains an embedded database engine (H2 database). This embedded database is initialized by default when midPoint is installed. And it is that embedded database that is used to store the configuration objects right now in your fresh midPoint deployment. This database does not need any special configuration, the database schema is applied automatically, and it is started and stopped together with midPoint. Therefore, it is ideal for demonstration and development purposes.

Vast majority of midPoint configuration is stored in the database. But there are few things that cannot be stored there. Such as connection parameters to the database itself. For that purpose midPoint has a small configuration file called config.xml. MidPoint also needs a place where to store other data that cannot be in the database, such as cryptographic keys, connector binaries and so on.

MidPoint needs a special directory on a filesystem for that purpose. We call it midPoint home directory. New directory with the name var is created in directory where midPoint distribution package was installed, i.e. at the same level as the bin directory where midPoint start start scripts are located. Assuming that midPoint was installed in /opt/midpoint directory then midPoint home will be located in /opt/midpoint/var directory.

The location of midPoint home directory can be changed by using the midpoint.home Java system property. This is done by specifying -Dmidpoint.home in the JVM command-line. Or in case that the default midPoint start scripts are used MIDPOINT_HOME environment variable can be used to set the location of midPoint home directory.

When midPoint starts for the first time it starts with an empty database. MidPoint populates the database with a minimal set of configuration objects. This set contains objects such as the Superuser role and user administrator. These objects get imported automatically because if they are not there you will not be able to log into the new midPoint instance. These objects are imported only if they are not already present in the database. If you modify them later then midPoint will not overwrite them.

Logging is perhaps the most important mechanism to diagnose any issues with midPoint. Logging should be the thing that comes to your mind anytime you cast a puzzled look at midPoint user interface. We try to make midPoint user interface convenient to use and we pay a lot of attention to good error reporting. But there are always some limits. The error that the user interface displays may be just a result of a long chain of causes and effects. Error messages in the user interface may not directly point to the primary cause. Or maybe there is no error at all, just midPoint does not do what it is supposed to do. That is the point where logging comes to the rescue.

MidPoint is using Java logging facilities to log its messages. MidPoint log file name is midpoint.log and it is stored in the log subdirectory in midPoint home directory (/opt/midpoint/var/log/midpoint.log). The default logging level is set up more-or-less to suit normal midPoint operation. This means that the messages on level INFO and above are logged while the finer levels are not logged. If you want to diagnose midPoint issues you will need to switch the logging levels to DEBUG, or in extreme cases even to TRACE. The logging levels can be adjusted in midPoint user interface. Navigate to Configuration  System  Logging.

MidPoint is not a simple system. There are complex interactions, there is usually a lot of custom configuration, customizations, expressions and so on. Diagnostics of midPoint issues is in itself no easy task. Therefore, there is a dedicated Troubleshooting chapter in this book.

Identity resource is one of the most important concepts in midPoint. Any system connected to midPoint is an identity resource. Identity resources (or just resources for short) are typically target systems where midPoint manages accounts. Moreover, source systems such as HR databases are also considered to be resources. There is no strict distinction between the source and target resource in midPoint. Both source and target resources are defined in exactly the same way. Resource can even act as both source and target at the same time.

MidPoint needs a way how to communicate with the resource. MidPoint has to know communication protocol, hostname, passwords, etc. For that purpose midPoint has resource definition objects. These are ordinary midPoint configuration objects stored in midPoint repository. Resource definition usually contains:

Resource definition looks like this in its XML form (simplified):

Resource definition is a very rich (and powerful) configuration object. It is maybe the richest configuration object in the entire midPoint system. Creating resource definition from scratch is usually no easy task. There is a lot of things to consider: connector configuration, identifier conventions, mandatory attributes, attribute value formats and so on. What we usually do is to locate a resource definition sample for a similar resource. Then we modify the sample to suit our needs. However, you need to understand how the resource definitions work to do this efficiently. Next few sections will explain the structure and function of resource definitions.

Believe it or not, there are people that do not like XML/JSON/YAML. There are also people that really want to start creating the resource from scratch. For all those people there is a resource wizard in the midPoint user interface. The wizard can be used to create and edit resource using a graphical user interface.

However, even if the resource wizard is your preferred way, it may still be easier to start with an existing sample. Find the sample that is the best match for your situation, import it in midPoint and then use the wizard to modify it.

There are many resource samples to start from. Most of them are located in midPoint distribution package. But there are other places to look for samples. Please see Additional Information chapter for suggestions.

Every resource needs a connector to work. Connectors are small pieces of Java code that are used to communicate with the resource. MidPoint looks for available connectors when it starts up. MidPoint will automatically create new configuration object for each connector that it discovers during the startup. The list of discovered connectors can be seen in midPoint user interface in Configuration  Repository objects  All objects and select Connector in the type field located at the top of the screen. The connector objects look like this:

The resource definition needs to point to the appropriate connector object. Therefore select the right connector from the connector list and remember its OID. Then use the connector OID in the resource configuration like this:

This is a straightforward way how to link connector and resource. However, it is not the most convenient one. MidPoint creates connector objects automatically. Therefore the OIDs of the connector objects are not fixed. Every midPoint instance will have different OID for the discovered connectors. Therefore, if we want a resource that is always using the LDAP connector in all the midPoint instances we cannot do that by just using OIDs. But there is another way. You can use search filter instead of fixed OID:

The detailed explanation of the search filters will come later. For now it is important to know just few basic principles. When this resource definition is imported, midPoint notices that there is no OID in the connectorRef reference. It also notices that there is a search filter. Therefore midPoint executes that search filter. In this case it looks for an object of ConnectorType type that has property connectorType with value com.evolveum.polygon.connector.ldap.LdapConnector. Therefore midPoint finds LDAP connector regardless of the OID that was generated when midPoint discovered that connector. Then midPoint takes the OID of the object that it has found. The OID is placed to the connectorRef reference, so midPoint can find the connector directly and it does not need to execute the search every time the resource is used.

This is the method that is frequently used to bind resource definition to a specific connector type. It has the advantage that it works in all midPoint deployments. Therefore it is also used in the configuration samples.

Each class of resources needs its own connector. There is an LDAP connector that supports all the common LDAP servers. There are connectors that work with generic database tables. These connectors are quite generic. But most connectors are built for a specific application or software system: Linux servers, SAP R/3, Siebel, etc.

There is a handful of connectors that are so generic that they are used in almost all midPoint deployments. These connectors are bundled with midPoint. That means that they are part of the midPoint application package and they are always available. These three connector bundles are part of midPoint:

These connectors are always available in midPoint. Other connectors must be deployed into midPoint. Connector deployment is a very straightforward process:

MidPoint will scan the icf-connectors directory when it starts up. It will discover any new connectors and create a connector configuration objects for them.

Connector needs a configuration to be able to work with the resource. This configuration usually consists of connection parameters such as hostname, port, administrative username, password, connection security settings and so on. The connector configuration properties are specified in the resource definition object. In a simplified from it looks like this:

There may be a very broad range of configuration properties - and every connector has its own set. While working just with the text representation of the resource definition you will need to find out the names of the configuration properties by looking at the samples, connector documentation or maybe even connector source code. It may look difficult but this is a perfectly viable approach. However, there are other ways. Firstly, there is the resource wizard. The wizard knows all the connector configuration properties and it will present the properties in a configuration form. The wizard takes the definition of the configuration properties from the connector schema. The connector schema is a definition of the properties that the connector supports: their names, types, multiplicity and so on. The connector schema is stored in the connector configuration object under the schema tag. Therefore even if you are working only with the XML/JSON/YAML files you can have a look at that schema to figure out what connector configuration properties are supported.

The connector schema also defines the connector namespace. Generally speaking namespaces in midPoint are used to isolate schema extensions that might conflict, and they are also used for data model versioning. The use of namespaces is optional in almost all parts of midPoint - but not yet in all the parts. Connector configuration is one of the few parts where namespaces should still be used. And it also makes some sense, as namespaces are used here as an additional safety mechanism. To keep a long story short, the configuration properties should be properly namespace-qualified:

The use of namespaces will be completely optional in later midPoint versions. For now just copy the namespace URIs from the samples. You do not have to completely understand what is going on. Just one thing: the namespace of the configuration properties should be the same as the namespace defined in the connector object. This is a long URI that is composed of connector bundle name and connector name.

For example: http://midpoint.evolveum.com/xml/ns/public/connector/icf-1/bundle/com.evolveum.polygon.connector-ldap/com.evolveum.polygon.connector.ldap.LdapConnector

If the namespace does not match then the connector will refuse to work. This is a safety mechanism that prohibits accidental use of configuration from one connector in another connector where the configuration properties may have the same name but a completely different meaning.

Minimal resource definition has just the name, connector reference and connector configuration properties. After that the resource should show the first signs of life. Therefore select a suitable sample file now. Strip it down to the minimum, modify connector configuration properties and import the resource into midPoint. You should be able to see your resource in the list in Resources > List resources. The icon next to your resource is most likely black - not green and not red. Green icon means that the resource is working, red icon means that there is an error, black means "I do not know yet". Click on the resource label. The resource details page should appear. There is a Test Connection button at the bottom of the page. Click on that button. It may take a while now. MidPoint is initializing the connector with the configuration properties that you have specified. Then the connector will be used to check connection to the resource. If the parameters were correct and midPoint can reach the resource you will see the green lights:

If there are any errors during connector initialization, configuration or network connection you will see the errors here. In that case correct the configuration properties and try again. If everything works well then the resource icon turns green. Now we have a minimal working resource.

There are a few more things that you can do with such a minimal resource. For example, you can look at the resource content. Navigate to the resource details page and switch to Uncategorized tab. Select one of the object classes that the resource supports. Just click inside the Object class input box and the suggestions will appear. Now click on the Resource button on the right side. MidPoint connects to the resource, lists all the objects of the given object class and displays the list. Now you can click on any object to see the details.

That is a very useful feature for several reasons. Firstly, you can check that not just the resource connection works, but that the connector can actually retrieve the objects. Secondly, you will get some idea about the object classes that the resource supports. And thirdly, by looking at several objects you can get a basic overview of how the data are structured: what attributes are used and what are the typical values. You will appreciate that information later on when we will be setting up mappings.

The only resource object that early identity management systems dealt with was an account. That is not sufficient anymore. Good identity management system needs to manage may different types of resource objects: accounts, groups, organizational units, privileges, roles, access control lists and so on. In midPoint these are the object classes: types of resource objects that are made accessible to midPoint by the connector. A minimal resource supports at least the account object class, but a typical resource supports more object classes. Each object class may have a completely different set of attributes: different names, different types, some may be mandatory, some optional.

The collective definition of the object classes and their attributes is what we call resource schema. Obviously, resource schema is different for every resource. Even resources that are using the same connector may have different resource schema (e.g. two LDAP servers with different custom schema extensions). MidPoint is a smart system and it is capable of automatic resource schema discovery. MidPoint will reach out to the resource and retrieve the schema when the resource is used for the first time. Retrieved resource schema is stored under the schema tag in resource definition object. You can have a look and examine the schema there. But beware, the schema may be quite rich and big.

Resource schema is an absolutely crucial concept. MidPoint takes advantage of resource schema whenever it needs to work with resource objects such as accounts or groups. MidPoint uses resource schema to validate mappings. The schema is used for automatic type conversions. And most importantly of all: resource schema is used to display resource objects in user interface. MidPoint adapts to resource schema automatically. Not a single line of custom code is needed to do that.

MidPoint topology is a star (a.k.a. "hub and spoke") with midPoint at the center. This is both physical and logical topology of midPoint deployments.

This means that the account A can be synchronized with midPoint user and then midPoint user can be synchronized with account B. But account A cannot be synchronized directly to account B. This is a deliberate decision that was made very early in midPoint design, and we have very good reasons for it.

Accounts and user that represent the same person are linked together. This link is a relation that midPoint creates and maintains. Therefore, midPoint knows who is the owner of a particular account. MidPoint also knows which accounts the user has. That is how midPoint knows which account needs to be synchronized with which user. It is critical for the links to be correct otherwise midPoint cannot reliably synchronize the data. For that reason, midPoint takes great care to maintain the links. And that is not always an easy task. There are strange corner cases such as renamed accounts or accounts that were deleted by mistake and re-created. But midPoint is built to handle such cases. The links are always maintained. And it is the link that allows midPoint to list all user’s accounts in the user interface.

The user in midPoint is known as focus in midPoint terminology. The accounts are known as projections. You can imagine a light projector that sends many light beams from its focal point to create a projection on the screen. This is the metaphor that we have chosen when building midPoint. And for the lack of better words this terminology remains in use even now. We will get back to the concept of focus and projections many times in this book. For now you just need to remember that projection means an account.

MidPoint knows which account belongs to which user by following links that it maintains. But how does midPoint know which attributes to synchronize? How to transform the values? And which side is the authoritative one? Mappings take care of that. Mapping is like a flexible data replication recipe. MidPoint allows to define mappings for each attribute in any direction. The mappings are used to control the synchronization on a very fine granularity.

Perhaps the best way to summarize synchronization principles is to illustrate them using a couple of examples. The first example is a modification of user properties in midPoint user interface. When the Save button is pressed then midPoint user interface sends the modification to midPoint core engine. The synchronization code in midPoint core follows the links to find all the accounts that belong to this specific user. Then the mappings are applied to synchronize the changed user properties to the accounts. Account changes are propagated to the resources and user changes are stored in midPoint repository.

The second example is slightly different. This case starts with a change of account data. This may be a change of an employee record in HR system. MidPoint detects that change and reads the changed account. MidPoint follows the link to find the user to which the account belongs. Then it follows other links from the user to find all the other accounts that may be affected. Similarly to the previous case the mappings are applied. The mappings from the HR account to the user are applied first. The result is a modification of user properties. Then a process identical to the previous case takes place. User modifications are automatically applied to all affected accounts.

Those two cases look to be quite different. First case is a manual change of data by system administrator. Second case is an automatic data feed from the HR system. But as you can see the principles that are used to implement those two cases are almost exactly the same. This is the consequence of midPoint philosophy: radical reuse of functionality and generic application of principles. You just need to define what you want to do (the policy). MidPoint takes care that it is done when it needs to be done.

Resource schema is a very important concept. It defines what object classes are supported by the resource and how they look like. But it is important to know not only how the objects look. It is also important to know what to do with them. And that is what the schema handling is all about.

Schema handling is a part of the resource definition object. It specifies which object classes from the resource schema are actually used by midPoint. And most importantly of all it specifies how they are used. This is the place where mappings are stored. This is the place where account-group associations are defined. This is the place where schema can be augmented and tweaked. Simply speaking, this is the place where most of the resource-related configuration takes place.

Schema handling section contains definition of several object types. Each object type refers to one "thing" that midPoint works with: default account, testing account, group, organizational unit and so on. Let’s start with something simple and let’s define just one object type now: default account. It looks like this:

This may seem trivial, but even such a minimal definition is important for midPoint. This definition tells midPoint that default account on this resource has inetOrgPerson object class. Resources such as LDAP servers may have dozens of object classes. Most of them are not used at all. There are often several alternative object classes that can be used to create accounts. It is important to tell midPoint which object class is the right one. And that’s what this definition does. Once this definition is in place, the accounts appear on the Accounts tab of the resource details page (they were visible only on the Generics tab before). This is a sign that the definition works correctly.

A clever reader surely noticed definition of kind in the above example. Setting kind to account indicates that this object type definition represents (surprisingly) an account. MidPoint supports many types of objects. But two types have a special place: accounts that represents the users and entitlements that give privileges to the accounts. MidPoint can handle the objects in a smart way if it knows that it is either account or entitlement. And the kind definition tells just that. There is also optional intent setting that can be used to define subtypes. But more on that later.

The schema handling section can also be used to augment (or even override) some parts of the resource schema. E.g. following example sets a display name for this object type. The display name will be used by the user interface when it displays the account.

However, the most powerful feature that is used in the schema handling is the ability to deal with attributes. Following sections are all about that.

Resource objects such as accounts or groups are mostly just a bunch of attributes. Almost all of the IDM magic is about setting the correct attribute to the correct value. The schema handling section of the resource definition is the place where the basic attribute behavior is defined.

The object type definition contains sections that define behavior of each attribute that we care about:

There is an attribute element for every attribute that we need. Lot of details can be defined here: display name of the attribute that will be used by the user interface, limitations and schema augmentation, override settings and so on. But the most important things that go there are the mappings. In the simplest form a mapping looks like this:

This means that the value of the cn attribute will be taken from the fullName property of the focal object (which is typically a user). This a very simple mapping, there is no value transformation, no condition – nothing complicated at all. This is how a lot of mappings look like. But mappings can also be very powerful and complex. That will be described in next section.

The attribute sections are used to set up the attributes that a typical user account on the resource has. Those will assign identifiers, set up full name, set description and telephone number attributes and things like that. It is a very convenient approach to have this directly in the resource definition. We can simply assign the accounts to the user without specifying any details. MidPoint evaluates the mappings in attribute sections to populate account attributes with the correct values. Now it is perhaps a good time to have a look at some sample resource definitions to get a feel how a real-world resource definition looks like. The samples are located in the midPoint distribution package or you can find them on-line. See Additional Information chapter for more details.

Mapping is a very flexible mechanism that takes one or more input properties, transforms them and puts the result in another property. Mappings are used all over midPoint. But perhaps the most important use of mappings is in the schema handling part of the resource definition where they set up account attribute values. We have already seen a very simple mapping that simply copies the values from one place to another. Now it is the time to look at mapping in its entirety.

Mapping consists of the three basic parts:

The three parts of the mapping as well as the basic principle is illustrated in the following diagram:

The diagram shows a mapping that takes employeeNumber user property and transforms it to description account attribute by using a simple Groovy script expression.

The source part of the mapping defines that there is a single source which is based on employeeNumber user property. Source definitions are important for the mapping to correctly process relative changes (deltas), mapping dependencies, etc. The source definition tells mapping that the value of employeeNumber user property should be passed to an expression.

The expression part contains a simple Groovy script that prepends the prefix emp# to the employee number value specified by the source definition. The expression part of the mapping is very flexible and there is a lot of ways that can be used to transform a value, generate new value, use a fixed value, pass a value without any change and so on.

The target part defines how the result of the expression should be used. In this case the result is to be used as a description account attribute. The target definition is necessary so the mapping can locate appropriate definition of the target property and therefore make sure that the expression produces a correct data type and that other schema constraints are maintained (e.g. single vs multiple values).

This mapping can be expressed in XML:

Not all parts of the mapping are mandatory. If the expression is not present then "as is" expression is assumed. Such expression simply copies the source to target without any transformation. Some parts of the mapping may be implicitly defined by the surrounding context. E.g. target or source is implicit if the mapping is used to define attribute behavior in the schema handling section. Therefore it is usually sufficient to define either source or target for mappings in schemaHandling:

This is the notation that you have seen in the previous section. Mapping source is explicitly specified as the familyName property of the user. Mapping target is implicitly set to be the attribute for which the mapping is defined. As no expression is explicitly defined it defaults to a simple copy of the value without any transformation (asIs).

Mapping notation can even be shortened a bit more in this case. It is quite clear that the mapping source will be one of the properties of the focal object (user). Therefore the $focus prefix can be omitted:

Those examples are still very simple. Mappings can do much more – as you will learn later on. But there is one more thing that we need to explain here. Mappings are designed to work with more than just a single source. Following diagram illustrates a mapping that takes two arguments: given name and family name. The mapping produces full name by concatenating these value with a space in between. This is the kind of mapping that is frequently used to construct user’s full name from its components. While the mapping may seem simple there are some sophisticated mechanisms hidden inside.

The mapping is represented in the XML form as follows:

There are two sources specified by the source definitions: user property givenName and another user property familyName. The mapping is using script expression to combine the values into a single value which is used to populate user’s fullName property.

This example also illustrates that the mappings are smart. The mapping may be evaluated only if one of the sources changes or if a full recompute is requested. In case that neither givenName not familyName changes there is no need to re-evaluate that expression. This is one of the reasons for requiring explicit source definition in the mappings. Without such definitions it is not (realistically) possible to reliably determine when and how the expression should be re-evaluated.

Mappings are used all over midPoint, in many places and situations. Sometimes a mapping needs to be really authoritative. It has to enforce the value to the target. But sometimes we want to provide a default value and the mapping should never change the target value once it is set. Therefore mapping can be set to various levels of strength: from weak to strong. Following table describes how that works:

The strength can be specified in any mapping by using the strength tag:

When it comes to mapping strength then the following rule of the thumb may be useful: If you want to enforce policy use strong mappings. If you just want to set a default value use weak mapping. If you are not sure what you are doing then normal mappings will probably work just fine.

Expression is the most flexible part of the mapping. There are approximately dozen different type of expressions ranging from the simplest as is expression through the scripting expressions all the way to a special purpose expressions that search through midPoint repository. Expression type is determined by the element that is used inside the expression section of the mapping. We refer to those elements as expression evaluators. You can find detailed description of expression evaluators in midPoint docs. We are going to deal only with few types that are most frequently used now:

The simplest expression evaluator is asIs. It simply takes the source and copies that to the target. It obviously works only if there is just one source. It is also the default expression evaluator. If no expression is specified in the mapping then asIs is assumed. It is used like this:

Literal expression evaluator is used to place a constant value in the target. This expression does not need any source at all. It always produces the same value. It looks like this:

The generate expression evaluator is used to generate a random value. As such it is used almost exclusively to generate passwords. We will deal with that expression later when we will be dealing with credentials.

The most interesting expression evaluator is undoubtedly the script expression evaluator. It allows to execute arbitrary scripting code to transform the value. Basic principle is simple: values from source properties are stored in the script variables. Script is executed and it produces an output. The output is stored in the target.

We have already seen a mapping that has a scripting expression:

There are two sources: givenName and familyName. The values of these user properties are placed in variables used by the script. The variables that have the same names: givenName and familyName. Then the script may do whatever it does with the variables. At the end the script has to return a value. The script above is written in Groovy, therefore the return value is the value of the last evaluated expression. In this case it is the only expression in the script which concatenates the two variables with a space in between. Script return value is placed in the output, which in this case is fullName user property.

Scripts are often used to transform the values before they are stored in account attributes. One very common case is construction of LDAP distinguished name (DN). The DN is a complex value in the form of uid=foobar,ou=people,dc=example,dc=com. However it is easy to construct such value using a simple script:

(A clever reader surely has a suspicious look on his face now. Of course, this is not entirely correct way how to compose LDAP DN. But please bear with us. We will correct that later.)

Midpoint supports three scripting languages:

All three languages can be arbitrarily mixed even in a single midPoint deployment. Although quite understandably such a practice is not recommended. The language can be selected for each individual expression by using language URI:

When writing scripting expression, please keep in mind that some characters must be properly escaped in the text format that you are using (XML, JSON or YAML). E.g. the ampersand character (&) so frequently used for logical operations needs to be escaped as & in XML.

Scripting expressions can do almost anything. And there is still more to them that meets the eye. This section provides only the very basic description to get you started. Will get back to the scripting expressions many times in this book.

The term activation is used in midPoint to denote a set of properties that describe whether an object is active. This includes properties that describe whether the user is enabled, disabled, archived, since when he should be enabled, to what date he should be active and so on. The simple enabled/disabled flag might have been sufficient in the 1990s. But that was a long time ago. We need much more than that. Therefore the activation is quite a rich data structure in midPoint. We are going to describe just the basic idea now, the details will follow later.

The most important activation concept is administrative status. Administrative status defines "administrative state" of the object (user). I.e. the explicit decision of an administrator whether the user is enabled or disabled. Except for administrative status there are also validity times, lockout status, various timestamps and metadata. But we will get to that later.

The important thing to realize is that both user and the accounts have activation properties - and they are almost the same. The user and account activation are using the same property names, meaning and data formats. This is important, because you would probably want account activation to follow user activation. E.g. if user is disabled then also all his accounts should be disabled. This is very easy to do in midPoint because the user and account activation are compatible. Therefore all it takes is a very simple mapping. There is a special place in the resource schema handling section for that:

It is as simple as that. Just an empty mapping represented by empty outbount element. User has administrativeStatus property, account has administrativeStatus property, therefore midPoint knows what is the source and target of the mapping. The values of the administrativeStatus property has the same type and meaning on both sides. Therefore the default asIs mapping is just fine. All that midPoint needs to know is that the mapping exists at all. That we want to map the value. That is a reason for having empty outbound element there. MidPoint will fill in all the details.

When this mapping is in place and the user gets disabled, the account will be disabled as well. When the user gets enabled, the account will follow suit.

Credential management is important part of identity management. MidPoint is built to easily synchronize credentials to many accounts. Similarly to activation, credential data structures of user and accounts are aligned. Therefore all that is needed to synchronize password to an account is a simple empty mapping:

When the user password in midPoint is changed the changed password will be propagated to all the resources that have a mapping like this.

This section describes a complete working example of connection to the LDAP directory. The configuration below is used to automatically create accounts in OpenLDAP server. Entire configuration is contained in a single resource definition file. Following paragraphs explain individual parts of the file. Simplified XML notation is used for clarity. Complete file in a form directly usable in midPoint can be found at the same place as all the other samples in this book (see Additional Information chapter for details).

Resource definition begins with object type, OID, name and description. These are self-explanatory:

Connector reference comes next. We want to point to the LDAP connector. Here we use dynamic reference that is using search filter to locate the connector:

The reference is resolved when this object is imported to midPoint. The resolution process takes the search filter and it looks for connector object with the connectorType specified in the filter.

Connector configuration goes next. This block specifies connector configuration properties such as hostname, port, passwords and so on.

The last part of this block defines that the ConnId framework result handlers should be disabled. ConnId result filtering is a legacy mechanism and most connectors do not need that any more. It may even be harmful in many cases. Unfortunately this mechanism is turned on by default. Therefore most resource configurations contain this part to explicitly turn all the handlers off.

These parts alone should already define a minimal resource. If you define just the name, connector reference and connector configuration you should be able to import the resource to midPoint. The connection test should pass and you should be able to browse resource content. However there is absolutely no IDM logic or automation yet. That is what we are going to add next.

Element that usually follows connector configuration is schema. However if you look at almost any file that contains resource definition you will find no such element. The schema element is automatically generated by midPoint when midPoint connects to the resource for the first time. Therefore there is no need to include it in the definition.

What we have to include in the definition is the way how midPoint handles the schema. This is defined in schemaHandling section. Our schemaHandling section contains just one objectType definition. We are going to define how to handle ordinary user accounts on our OpenLDAP server.

This is the place where we define the kind of objects that we are going to handle. In this case it is account. This object is default account. Which means that it will be used in case that the account type is not explicitly specified. There is also specification of a display name. Display name is not used in automation logic. It is used by the user interface when referring to this definition. And finally, there is specification of the object class. The inetOrgPerson object class will be used to create new accounts. The object class specification determines what attributes the account can have.

The objectType definition also includes a specification of attribute handling. There is one section for each attribute that we want to handle in automated or special way. It starts with the most important attribute: LDAP distinguished name (DN):

The ref element specifies name of the attribute that we are going to work with. In fact this is a reference to automatically-generated schema part of resource definition. Definition of display name follows. Display name is used by the user interface as a label for the user interface elements (fields) that work with this attribute. This definition sets a nice "Distinguished Name" label instead of cryptic "dn" which would be used by default.

Let’s skip the limitation definition now. We will come back to that later.

The outbound mapping definition follows. This is where the automation logic is specified. This is the place where the DN value is computed. The name property of the user object is the source for this mapping. The name property usually contains username (login name). This value is used by scripting expression in this mapping. The expression is supposed to create a DN in the form:

uid=username,ou=people,dc=example,dc=com

The expression here is a clever one. It does not do the work all by itself. It invokes a library function to compose the DN. It may look like a good idea to use simple string concatenation to construct a DN. However, that will fail in case that the DN components contain certain characters that need to be escaped in the final DN. The composeDnWithSuffix library function takes care of that and creates a proper DN.

The outbound mapping will be evaluated whenever we need to construct a DN. This obviously happens when a new object is created. But the same mapping is used when a user is renamed (i.e. his username changes). This is the reason that the mapping needs specification of source. Rename is often quite tricky and complicated operation. It may not be cheap and in some applications it may not be entirely reversible. We definitely do not want to trigger DN changes unless they are really needed. The specification of the mapping source tells us when the DN change is needed. In this case it tells us to change the DN in the name property of the user object changes.

Now it is the right time to go back to the limitations section. The dn attribute is defines as mandatory attribute by the schema. And that is perfectly correct: LDAP object cannot be created without a DN. MidPoint is using schema for everything. Therefore when midPoint displays a form to edit this LDAP account it will check that DN has a value because it is a mandatory attribute. However, normally we do not want users to enter the DN in the user interface forms. We want to compute DN automatically. And that is exactly the point of the outbound mapping. Yet midPoint does not know when the expression computes a value and when it does not. The expression is a generic piece of Groovy code. As far as midPoint can see the expression can produce any value, including empty one. Therefore even if there is an expression, midPoint sticks to the schema and it still requires that DN value is entered by the user. However we have written the expression and we know that it will produce a value for any (reasonable) input. Therefore we want to tell midPoint that the DN is no longer mandatory – that it is OK for user to enter no DN value in user interface forms. And that is exactly what the limitations section does. This section overrides the automatically generated schema and it turns the dn attribute from mandatory to optional.

And that is all. Now we have defined the behavior of the dn attribute. We can use similar approach to define the behavior of other attributes as well. E.g. the handling of the cn attribute has similar definition:

In this case there is outbound mapping, but it has no explicit expression. Which means that the value is taken from the source without any change ("as is"). Therefore the attribute cn will have the same value as user property fullName.

It is also possible to define an attribute without any mapping:

This means that midPoint will not provide any automatic handling for the entryUUID attribute. This definition is used just to set a user-friendly display name for the attribute.

Mappings and expressions have almost unlimited flexibility. E.g. the following definition sets a static value for the description attribute:

This mapping has no source because the source does not make any sense for static value. Static values are always the same. You can also notice that this mapping is weak. It will be used to set the description attribute only if that attribute does not have any value already. It will not overwrite existing values.

The inetOrgPerson object class has much more attributes than those defined in the schemaHandling section. Those attributes will be automatically displayed in the user interface. MidPoint will use the generated resource schema to determine their names and types. MidPoint will display these attributes, the user can change them and midPoint will execute those changes. But apart from that midPoint will not do any special handling on those attributes. It is all right not to enumerate all the attributes in schemaHandling section. You only need to define those attributes which you want to handle in a special way.

There are two more definitions to describe before our example is complete. First definition is the activation definition. It is very simple:

This is a definition that specifies handling of the activation administrative status. This status property specifies whether account is enabled or disabled. Activation properties are somehow special in midPoint. MidPoint understands the meaning and the values of activation properties. MidPoint also expects that user activation and account activation will usually be mapped together. Therefore it is enough to tell midPoint that you want such mapping. MidPoint already knows the source (user activation) and the target (account activation). If the user is disabled then the account will get disabled. If the user is enabled than the account will get enabled.

The last thing that we need for the resource to work well is to define a mapping for credentials. In this case it is a password mapping:

Similarly to activation, the credentials are handled in a special way in midPoint. MidPoint understands how credentials work, what their values are and how they are used. MidPoint also expects that user credentials such as passwords will usually be mapped to the account credentials. Therefore all that midPoint needs to know is that you want to do that mapping. It can automatically determine the source and target. The account will have the same password as the user. User’s password is used when a new account is created. And when user changes his password the change is also propagated to the account.

That is it. Now you have your first (almost) working resource. You can import the definition to midPoint and test it. Simply assign the resource to a user. The OpenLDAP account will be created - the DN and all the essential attributes will be automatically computed. When midPoint creates an account for a user it remembers who is the owner of that account. Therefore it can easily delete the account when needed. Unassign the resource and the account will get deleted. This is how automated provisioning and deprovisioning works. No real programming is needed. Just a declarative specification and a line or two of very simple scripting. Such configuration can be done in a couple of minutes. And it is essentially the same process for all the applications. Just the connector is different. The connector has different configuration properties, the attribute names are different - but the principles and the tools are the same. It is easy to connect many heterogeneous applications in this way. The connectors and the mappings are hiding the differences. In the end all the "resources" look the same to midPoint. The same principles are used to manage them. Therefore the management can be done efficiently even at a large scale.

Yet this configuration is still extremely simple. We are just scratching the surface of what midPoint can do. There is much more to see in the next chapters. But we need to explain more of the fundamental midPoint concepts before getting there.

Linking users and accounts is one of the basic principle of any decent identity management system. However, it is surprisingly difficult to implement such link. There are numerous methods how to reliably identify accounts and they vary from system to system. Some systems identify accounts only by username - which makes reliable detection of renames quite difficult. Other systems improve on that by introducing another identifier, an identifier that is persistent. Identifier value is assigned by the resource and it never changes. However, username is still used as secondary identifier and it has to be unique. Yet another system has compound identifiers that consist of two or more values. Some system have globally-unique identifiers while other systems have identifiers that are only unique in their own object class. Some systems have hierarchical structured identifiers, others have flat unstructured identifiers. Some identifiers are case-sensitive strings, others are case-insensitive, some identifiers follow complex normalization rules and yet another identifiers are binary and completely opaque. To make long story short: reliable identification is really complicated.

We do not want to pollute user object with all the delicate details of account identification. Therefore we have created a separate midPoint object that hides all the resource-related details and identification complexities. We call it simply shadow because it behaves as a shadow of the real account. When midPoint detects new account a shadow is automatically created to represent the account in midPoint repository. When midPoint detects that account has changed (e.g. it was renamed) then midPoint automatically updates the shadow. When the account is deleted midPoint also deletes the shadow. Technically, shadow is still an ordinary midPoint object. Therefore shadow has object identifier (OID). Other objects can simply point to the shadow using ordinary object reference. And that is exactly how user-account links are implemented:

The shadow objects contain all the data that are needed to reliably identify an account - or any other resource object such as group or organizational unit. Shadow also points to the corresponding resource definition to make the identification complete. Shadows are multi-purpose objects and they have many uses in midPoint. Shadows record meta-data about resource objects. They are used to hold cached values of the attributes (this functionality is still experimental in midPoint 4.0). Shadows can be used to hold the state of the resource objects for which midPoint does not have on-line communication channel and the operations are executed manually (a.k.a. "manual resources"). Therefore shadows are quite complex objects. Following picture provides more substantial example of a shadow.

Do not worry if that picture looks a bit scary. Shadows may be complex, but they are almost always invisible to midPoint users. Shadows are automatically and transparently maintained by midPoint core. Under normal circumstances, MidPoint takes all that is needed to maintain the shadow and no special configuration is needed for that. We describe the mechanics of the shadow objects here mostly for the sake of completeness. But there may be situations when this knowledge may be useful. These are usually situations when midPoint was mis-configured and the shadows were created incorrectly. In that case you may need to purge all shadows and start over. But beware: shadows are used to link users and accounts therefore if you purge the shadows you will lose the links. Yet, even that is usually not that painful. The synchronization methods described in the next chapter may be used to easily re-create the links.

Synchronization is one of the basic mechanisms of midPoint. Synchronization mechanisms are integral part of midPoint design from its very beginning. Many of the things that midPoint normally does are in fact just different flavors of synchronization. There are obvious cases such as reconciliation process, synchronizing account attributes with data in midPoint repository. However, there are also less obvious cases, such as ordinary provisioning operation when midPoint needs to create a new account for a user. Even that case is in fact a synchronization: midPoint user properties are synchronized with a new empty account on the resource. Majority of midPoint operations are directly or indirectly using the synchronization principles.

MidPoint synchronization is almost a continuous functionality spectrum that can be tweaked and tuned to match specific needs. Yet, the synchronization mechanisms can be divided to several broad and slightly overlapping categories:

Individual mechanisms differ in a way data inconsistency is discovered: livesync will actively look for new changes, reconciliation will compare the data one-by-one and opportunistic synchronization will discover inconsistency by pure chance. Yet, all the mechanism react to inconsistency in the same way. There is only one policy that specifies how to fix the system. Of course, there may be slight deviations in the behavior. For example, we usually want import to behave in slightly different way than reconciliation. MidPoint allows that. Yet, there is just one big synchronization mechanism overall. This has a very good reason. It does not really matter how the problem was discovered. What really matters is that the problem gets fixed. We do not want to maintain four separate configurations for that. Having one policy is enough. MidPoint knows which part of the configuration need to be applied in each specific situation, and it does it automatically. This unifying approach significantly simplifies the configuration of midPoint synchronization mechanisms. That is also a reason why the boundaries of individual synchronization mechanisms are quite fuzzy. In fact this is just a single big mechanism with several facets.

The tale of idealistic identity management deployment starts with a human resources (HR) system. The HR system is supposed to have records for all the identities, therefore it is authoritative source system. Identity management system pulls in all the data from HR database, recomputes them and creates accounts on target systems. And they lived happily ever after.

Now, let’s get back to reality. The HR database is indeed an authoritative source of data in many real-world cases. However, it is a limited source. It contains only data about employees, and it has only partial information about them. For example there is no username. Username has to be generated by IDM logic. There is no initial password. Organizational structure assignment is often incomplete, missing or unreliable. Therefore, HR database is only a partially-authoritative source. There may be additional authoritative sources for contractors, partners, suppliers, support engineers and other identities that need to access our systems. These are additional source systems. Then there is a directory system, which is often Microsoft Active Directory. This should be a target resource in theory. Yet, there often are pieces of authoritative information in here. For example an algorithm to generate a username may be based on the usernames that are already used in the Active Directory. The data in Active Directory may also be needed to create a unique e-mail address. Directory systems are also used as a semi-authoritative sources for telephone numbers, office numbers and so on. Therefore, such systems are both target and source systems. Then there are "pure" target systems. These are not supposed to be authoritative in any way. Identity management system will only write to these. Or …​ will it? What happens when a conflicting account already exists on such system, and therefore we cannot create a new account for a new employee? And how do we check if there are no accounts that are not supposed to be there? It turns out that even the "pure" target systems contain valuable sources of information after all.

The reality brings a wild mix of source, target, semi-source, target/source and quasi-target systems that are almost impossible to put into a pre-defined boxes. Therefore, midPoint does not bother to define a concept of "source" or "target" resource. All resources can be both sources and targets, and the authoritativeness of each attribute can be controlled on a very fine level. Almost every real-world situation can easily fit into this model.

MidPoint is firmly based on the principle of reuse. Previous chapter explained that behavior of attributes during provisioning is controlled by mappings. Therefore, it is perhaps no big surprise that the behavior of attributes during synchronization is also controlled by mappings. In fact, provisioning is just a special case of synchronization. Following picture explains the combined mechanism.

There are two types of mappings:

The mappings themselves are almost the same regardless whether they are inbound or outbound. They have sources, targets, expressions, conditions, etc. Just the sources and targets are reversed:

That is it. Think about the same mappings that were used in previous chapter, just flip the direction. Now the mapping will take data from the account and the results will be applied to user object. Like this:

This mapping will take the value of lastname attribute from the resource and store the value in familyName property of midPoint user.

The rest is the same as outbound mappings. All the expressions and evaluators can be used for inbound mappings in the same way as for outbound mappings. For example a Groovy expression can be used to sanitize the value before it is stored in midPoint:

The same approach can also be taken for activation, and even for password mappings. However, there is one difference for password mappings. Password is usually write-only value. When the password is written, it is usually hashed, and the original value cannot be retrieved any longer. Then there are resources such as HR systems that do not store employee passwords at all, because those are not really accounts that we are reading. Those are just regular database entries that the connector presents as accounts. Inbound password synchronization is almost never easy, and it often requires a lot of planning and ingenuity. However, there is one method that is used quite often. The initial user passwords are usually randomly generated. As this is a very common case midPoint can do this easily:

This mapping generates random password for a user. Both the mapping and generate expression evaluators are quite smart. The mapping knows that the target is user password, without any need to explicitly specify that. In addition to that, generate expression evaluator will take user password policy into consideration. It does not make sense to generate any random password. If we do not consider password policy, then we can generate password that is too short, too long, too weak to pass the policy or too strong to be useful in any way. Therefore, generate expression will look for password policy, and generate a random password that just matches requirements for password length and complexity.

There are more important details to see here. The inbound password mapping is weak. There is good reason for this. We do not want existing midPoint password to be replaced by randomly generated password. We only want to set a random password in case that it is an initial password, the first password ever. That is exactly what a weak mapping does: it sets new value only if the target does not have any existing value. Therefore, this mapping will not overwrite passwords that are already set.

It is all quite easy to import all HR records into an empty midPoint. Set up inbound mappings, start import task, wait a bit and all is done. But practical situations are much more complex. Synchronization algorithm usually do not run on a green field. Live synchronization and reconciliation are supposed to work with pre-existing midPoint users. Import is usually not trivial either, for example in cases when we try to import data from an additional data source into a running midPoint deployment. Some users in the import set are new, but there may be accounts for existing users. We need to tell the difference between brand-new account and an account that belongs to an existing user. We need to handle those situations in a different way. Of course, midPoint has an easy solution for this: correlation mechanism.

Correlation expression is a method to connect newly-discovered accounts and existing users. It works like this: whenever midPoint discovers new account it will try to link that account to an existing user. Correlation expression is used to do this. Correlation expression is in fact a parametric search query. Such search query is constructed for every new account, and it is used to look for users that the account belongs to. The easiest form of the correlation expression is to search users using an identifier:

This correlation query takes the value of empno attribute of the account. This value is placed into the search query that midPoint computes in memory. Given an account with empno attribute set to 007, the resulting search filter looks like this:

MidPoint looks for users that match this search filter. If there is a user with employeeNumber property set to 007 then such user is considered to be an owner of the account.

MidPoint has its own data representation mechanism and object structure. Therefore, midPoint also has its own query language that is designed to work with the object structure. The query language is not difficult to learn as it follows the structure of other common query languages. The language itself is described later in the book and in midPoint documentation. But do not worry about this too much now. Vast majority of correlation expressions is very simple. In fact, it is usually just a single equal clause just like that one used in the example above.

Using search queries for correlation may seem a little bit too complex. Yet, it is necessary. The correlation expression must be a search filter because that is the only efficient way how to find single user in a large set of other users. We cannot scan the users one-by-one. We need to utilize the search power of the database for this.

Correlation expression can be used to find an owner for a new account. That is a part of the solution, but not entire solution. If the owner is found, then the action is quite obvious: link the account to the user and proceed as usual. But what to do if the owner is not found? This resource may be an authoritative resource, and therefore we want to create a new user based on the account. On the other hand, this may be a reconciliation with a target resource, and we have just found an illegal account. We probably want to disable such account in this case. Moreover, what to do if more than one owner is found? This can all become quite complicated. Therefore, midPoint has a concept of synchronization situations to make all the possible cases understandable and manageable.

Whenever midPoint deals with a change on an account, the situation of that account is determined. The situation reflects whether this account is already linked to the user, whether we know the candidate owner, whether we cannot determine the owner and so on. Individual situations are explained in the following table.

After synchronization situation is determined, midPoint continues by figuring out what a proper reaction is. The reaction may be quite clear for some situations (e.g. unlinked), but there is a lot of variability for other situations (e.g. unmatched). This variability is a reason that midPoint allows to set a reaction for each situation individually. There are several pre-defined reactions:

If no reaction is explicitly configured for a situation, then midPoint does nothing. The situation is recorded in midPoint repository, but no other action is taken. This is part of midPoint philosophy: not to change the data unless an action was explicitly configured.

The reactions can be defined in the synchronization section of resource configuration:

Most of the configuration is perhaps self-explanatory. This is a typical authoritative resource. If there is a new account on the resource, and we do not have an owner (situation: unmatched), then create a new user (action: addFocus). If the account is deleted from the resource (situation: deleted), then delete the user as well (action: deleteFocus). If we happen to find an account which should be linked but it is not (situation: unlinked), then link it (action: link). The only think that deserves special explanation is the reaction to linked situation. In this case there is not much to do. Everything seems to be in order. However, there may still be attributes that are not set correctly. Remember the inbound mappings? The inbound mappings were not even mentioned in this section yet - for a good reason. The inbound mappings are not evaluated at correlation stage of synchronization process. Evaluation of inbound mappings happen only after the situations and reactions are evaluated. We need to properly correlate accounts first, so all the accounts are properly linked (or unlinked), and the inbound mappings have valid sources and targets. Even when the accounts are correlated, the evaluation of inbound and outbound mappings do not happen automatically. MidPoint does not change the data unless it is explicitly configured to do so. There are reactions for unmatched, deleted and unlinked situations. Therefore, in those cases midPoint assumes that it is expected to fully synchronize everything, and therefore all the mappings and policies are evaluated automatically. However, there is no reaction for linked situation. In that case midPoint assumes that it should do nothing, as nothing is explicitly configured. Hence the synchronize property. This property can be used to force midPoint to do full synchronization, even if there is no explicit action configured. It can also be used to avoid full synchronization, even if explicit action is configured.

The figure above illustrates the usual sequence of events during inbound synchronization:

Please note that the description of this process is slightly simplified for clarity. There are also obvious deviations from this process. E.g. inbound mappings are skipped in case that the user is about to be deleted, the mappings are also skipped if the reaction does not include "synchronization" and so on. However, generally speaking, the sequence above is what usually happens during inbound synchronization.

Now we know how the inbound synchronization works: midPoint reads the account, then correlation is applied, situation determined and reaction executed. However, we have not yet discussed the details of the very first step: how does midPoint actually read the account? Nothing happens without a reason, therefore there must be some active component in midPoint that actually looks for the new, changed and deleted accounts. That component is a synchronization task.

MidPoint task is an active process that runs inside midPoint server. This is the first time that we encountered the concept of a task, but it is definitely not the last one. Tasks are used for numerous purposes in midPoint. They are used to track long-running operations, approvals and actions that work on large sets of objects (bulk actions). There are tasks that execute cleanup jobs, compile reports and provide variety of other functions. The concept of a task is a very powerful and flexible one. Tasks can be used to track execution of a short one-off operations. Tasks can be used to execute scheduled actions in regular intervals. Tasks can be used to track long-running processes. We will be using tasks in almost every chapter of this book.

Tasks are used as an active component to "run" almost all synchronization mechanisms:

Each type of synchronization task is detecting changes using a different mechanism. However, once the task detects the change or reads the account, then the processing is the same for all tasks. All the tasks lead to the same algorithms based on the same configuration and policies. Therefore, it does not matter whether it has all started in reconciliation or live synchronization task. It will all end up in the same correlation-situation-reaction-mapping process.

The tasks are necessary to initiate the synchronization. They are the active part, the spark that starts the synchronization process. Without the tasks the synchronization does not really work. There are ways the synchronization can "happen" even without a task, e.g. as a reaction to user interface operation or if a new account is discovered during an unrelated operation. Desprite that, practical deployments need at least one synchronization task to work properly. This task takes care of the vast majority of synchronization cases.

Strictly speaking a task is quite a strange kind of animal. Tasks have their data and configuration as most other midPoint objects. Unlike other objects, tasks are active, they run. Therefore, there are CPU threads associated with the tasks when the tasks are running. There are mechanisms to monitor task progress. The tasks need to be cluster-aware, they have to fail over to a different midPoint node if one node fails. The tasks are quite rich, and they may be a bit tricky to handle. Fortunately, midPoint is making task handling reasonably simple. Tasks are represented as ordinary midPoint objects. Therefore, tasks can be imported to midPoint in XML/JSON/YAML form as any other object. Tasks can be easily edited in their XML/JSON/YAML form to change the scheduling, modify the parameters and so on. Of course, there are some special functions that only the tasks have (such as suspend and resume). Such functions cannot be directly controlled using the XML/JSON/YAML format. However, the vast majority of task management can be done using the very same methods that are used to control other midPoint objects.

Tasks can be created by taking the XML/JSON/YAML file and importing that to midPoint. That is the way synchronization tasks are often managed. When an XML-formatted resource definition is created, then there is often an associated synchronization task. Which means that both resource and all the necessary synchronization tasks can be imported together. Synchronization tasks can also be created from midPoint user interface. They are usually created by using special-purpose Defined tasks tab in resource detail pages.

Once the synchronization tasks are created, they can be managed in the same way as other tasks are managed: in the Server tasks part of the midPoint user interface.

This section describes complete working example that feeds HR data into midPoint. The ExAmPLE company HR system is an old and complex system. Therefore, the easiest integration method is to use structured text exports. The system can export employee data in a form of a comma-separated text file (CSV). The HR system is set up to export fresh data every night. MidPoint takes this export file, crunches the content, and updates the data about users.

This configuration is done in three steps. First, we will use a simple setup to import the data into midPoint. This is an operation that is executed only once. Then the configuration will be updated to run scheduled reconciliation task. Reconciliation compares all the data records every time, and it makes any necessary updates. Even though this method would be perfectly acceptable for the company of this size, we are still goging set up a live synchronization task.

The core of the configuration is contained in a single resource definition file. Following paragraphs explain individual parts of the file. There are few additional configuration files for reconciliation and live synchronization tasks. Simplified XML notation is used for clarity. The complete file in a form that is directly usable in midPoint can be found at the same place as all the other samples in this book (see Additional Information chapter for details).

The resource representing HR system is configured as data source. It will be used to "pull" the data inside midPoint. However, as we have described previously, there is no fundamental difference between source and target resources in midPoint. Therefore, this HR resource starts in entirely ordinary way. There is a reference to the CSV connector and the connector configuration:

The next section is schema handling configuration. That is where it becomes slightly more interesting. The schema handling section contains inbound mappings for HR account attributes:

The account attribute empno is mapped to midPoint user properties name and employeeNumber. Account attributes firstname and lastname are mapped to givenName and familyName properties respectively. This is perhaps self-explanatory.

The next part of the configuration specifies mappings for activation and credentials:

Activation is a very specific concept in midPoint. MidPoint knows activation attributes and their meaning. Therefore, there is no need to specify a lot of details. That makes activation mapping a very simple thing. It simply specifies that the administrative status should be mapped in the inbound direction. And that is it.

However, the mapping for credentials needs a bit of explanation. What midPoint sees as HR accounts are not exactly accounts. They are usually just records in the HR database. Nobody is using these HR records to log into the HR systems. Therefore, there is no password associated with them. Yet, we need a password for the users in midPoint. Therefore we are going to generate the passwords. For that we are going to use the weak mapping with generate expression that was explained above.

The mappings are undoubtedly important. The mappings specify how are the account data reflected to midPoint user. HOwever, the mappings do not specify whether the accounts should be created or deleted. Mappings control the data, but they do not control the lifecycle. It is the next configuration section that makes this resource really authoritative:

Given the information in this chapter, this configuration should be quite easy to read. This is how a typical authoritative resource works. If there is a new account on the resource, and we do not have an owner (situation: unmatched) then we create a new user (action: addFocus). If there is a new account for which we can find existing owner (situation: unlinked), then simply link it (reaction: link). If the account is linked already (situation: linked), then we just synchronize the data. In fact, we will synchronize data for all the other situations as well. Except the last one. If the account is deleted in the HR system (situation: deleted), then we want to delete midPoint user as well (reaction: deleteFocus). As the user gets deleted there is no point in synchronizing the data. MidPoint knows that and skips application of mappings.

The ownership of the accounts that are not already linked is determined by the correlation expression. In this case, the expression will be comparing account attribute empno with user property employeeNumber. If the values match then the user is considered to be an owner of the account.

There is one more detail in this resource that we have skipped:

This is a setting that adjusts the behavior of midPoint assignments. As was already mentioned, all resources in midPoint are created equal. The source resources must follow the same rules as target resources. One of the fundamental rules of midPoint is that there should not be any account without a specific reason to exist. In midPoint terminology, every account exists because there is an assignment that justifies its existence. While this approach is exactly what we want for the vast majority of (well behaving) resources, it is not exactly ideal for source resources. Those resources work the other way around. The HR account is in fact a cause for midPoint user existence, not its effect. Therefore, there is really useful assignmentPolicyEnforcement setting that controls the behavior of assignments. This setting is used in a variety of scenarios, mostly for data migration, and to tame resources that just won’t behave in a civilized manner. However, in this case the setting is used to turn off the assignment enforcement for this resource entirely. As this resource is an authoritative source, the assignment enforcement does not make much sense. Behavior of this resource is defined by the synchronization section of resource configuration.

Resource configuration is complete now. This configuration sets up the connector, mappings and synchronization policies. The configuration is the same for all the synchronization flavors: import, reconciliation and live sync - they will all use the same settings. When it comes to configuration, the only difference between those synchronization flavors is the way how the synchronization tasks are set up. If an import task is set up, then import of resource accounts will be executed. If reconciliation task is set up, the reconciliation will be executed. It is all in the tasks. Synchronization tasks can be easily set up using those convenient buttons in the user interface. However, we like to make our lives a bit painful in our part of the world. Therefore, we are going to go hardcore, and we import the tasks in the XML form.

First task is an import task. This task lists all the accounts in the HR CSV file. The task pretends that each of the accounts was just created. If the task is executed for the first time, then resulting situation of the accounts is going to be either unmatched or unlinked. Therefore, the task creates new midPoint users or links the accounts to existing users.

This is a very basic structure of a task. Similarly to all midPoint objects, a task has a name. Then there is an assignment of Import task archetype. We will describe archetypes later. For now, it is only important to know that this classifies the task as import task, so the user interface can place the task in proper categories. Task needs definition of an owner. The owner is a user that is executing the task. This is important, because authorizations of task owner determine what the task is allowed to do. This is also the identity that will be recorded in the audit log. In this case administrator is owner of this task. Task execution status tells whether the task is running, it is suspended or finished. Tasks are often executed periodically, therefore they need a schedule. In this case, we will start with task that is executed just once, to test the configuration. Hence the single recurrence, which specifies that the task runs only once. Then there is definition of activity. Activity specifies what the task really does. In this case the activity specifies that this is a synchronization task which imports accounts from the resource. The resource is specified by the resourceRef reference in the activity specification. This points to our HR resource.

When this XML definition is imported to midPoint, the server tries to execute the task. That means that import of accounts from the HR resource starts immediately. Progress of the task can be monitored in the Server tasks section of midPoint user interface. The import task is not a recurring task, it will run only once. If you need to re-run the task, you can do that from midPoint user interface. However, the task will not get executed again, unless you explicitly tell midPoint to do so. This is how typical import tasks work. They are usually executed when a new resource is connected to the system. Once everything is set up, correlated and linked, then the import task is not needed any more.

Import task will get the data from the resource into midPoint. As import is not a recurring task, it will not keep the data synchronized. Import tasks are not designed to do so. Fortunatelly, there are other tasks that are designed for continuous synchronization. Reconciliation task is one of these. Reconciliation task lists all the accounts on a resource and compares that with data in midPoint.

Definition of a reconciliation task is almost the same as the definition of import task. However, there are crucial differences. First of all, there is different activity. This is what makes this task a reconciliation task. Then, the task is recurring. This means that midPoint will repeat execution of the task. Therefore, there is also execution schedule, so the server knows when to execute the task. Reconciliation tasks are usually resource-intensive, therefore we usually want to execute them at a very specific off-peak times. For that reason the execution schedule is defined using a cron-like pattern. UNIX-friendly readers will be surely familiar with this. The format is:

seconds minutes hours day-of-month month day-of-week year

The string 0 0 1 ? * SAT means that this task will be executed every Saturday at 01:00:00am. There is also definition of misfire action. Misfire is a situation when the server is down at the time when the task is supposed to run. In this case, if the server is down in the early hours of Saturday, this task will be executed as soon as the server starts up.

Reconciliation task is a real workhorse of identity management. It can be used for almost any resource. It is very reliable. It is often used to fix data problems, apply new policies, look for missing accounts, illegal accounts and so on. It is indeed a really useful tool. Yet, it has its downside. Reconciliation iterates through all the accounts, it recomputes all the applicable policies for every account, one-by-one. Therefore, it may be quite resource-intensive. It may be even quite brutal if the policies are complex, user population is high and the resources are slow. This can take hours or even days in extreme cases. Even for smaller deployments, reconciliation is not entirely easy. The problem is not in midPoint. MidPoint can be usually scaled up to handle the load. However, listing all the accounts often may put unacceptable load on the resources. Therefore reconciliation is not executed often. Daily, weekly or even monthly reconciliation seems to be a common approach. Reconciliation is reliable, but it is not entirely what we would call "real-time". But of course, midPoint has a faster alternative.

Live synchronization is the way to go for real-time synchronization. Or rather almost real-time synchronization. Practical latencies for live synchronization are in the range of seconds or minutes, which is fast enough for most practical cases. Live synchronization is also quite resource-efficient. Overall, it is much faster and much lighter than reconciliation. Unfortunately, live synchronization is not available for all resources. Live synchronization depends on the ability to get recent changes from the resource in a very efficient way. Therefore, it is only available for resources that record the changes. The specific mechanism to record the changes may vary from resource to resource. It may be as basic as a simple modification timestamp, or it may be a complex real-time change log. The mechanism has to be good enough for the connector to discover recent changes, and it must be efficient enough for the connector to do that every couple of seconds. If such mechanism is available, and the connector knows how to use it, then setting up live synchronization is easy. All that is needed is synchronization task.

This task definition should be easy to understand by now. There is a different activity that makes this a live synchronization task. There is also a different type of scheduling. We do not want to execute this task at a specific time. We rather want to execute it all the time, at regular intervals. In this case the interval is set to 10 seconds. That is all. We have live synchronization running. If the HR CSV file is changed now, the changes will get automatically processed by midPoint.

Setting up synchronization flavors is just a matter of setting up the tasks. The rest of the configuration is the same for all flavors. Therefore, it is very easy to run both live synchronization and reconciliation for the same resource. Just create two tasks. In fact, this is quite a common setup. Live synchronization is used to get the changes quickly and efficiently. Reconciliation is used to make sure all the changes were processed and that the policies are applied consistently.

Now we have the HR feed up and running. However, there are still few issues. A clever reader would surely notice that this is not a very good HR resource. MidPoint users created from this HR feed have given name and family name, but the full name field is empty. But do not worry. We will sort that out in later chapters, with the help of object template. Also, the users have employee number as their username. This may be in fact a very good approach for some deployments as it avoids the need to rename accounts. However, it is not a very user-friendly approach. Therefore, most deployments need to generate more convenient usernames. This is easy to do with midPoint, and we will also address that later. There is still a lot of things to learn before we get to a complete synchronization set up.

All resources are created equal in midPoint. However, source resources almost always have a slightly special standing. Even though midPoint mechanisms are the same for all resources, the data coming from the sources often have significant impact on the entire solution. There is this traditional computer engineering wisdom: garbage in, garbage out. An error in data feed may cause a lot of problems everywhere. Therefore, it is important to get the data sources right. This is usually one of the first steps in an IDM project.

Unfortunately, source data feed is usually quite difficult to set up correctly - and it is almost impossible to get it right at the first try. There may be good old configuration problems, which are usually easy to fix. There may be data compatibility problems, such as presence of non-ASCII national characters where they are not expected. Worst of all, source data may be of poor quality, there may be inconsistencies, typos, the data may be out of date, not reflecting the reality very well. These problems are the most difficult to correct, as the right way to correct them is to modify the data at its source. That takes time, meetings, mail messages, management decisions, processes, excuses, delays and a lot of patience. Therefore, setup of a data source is usually an iterative process. The process usually goes like this:

If the initial IDM deployment step includes an HR feed, we strongly recommend to start with that HR feed. It is significant benefit to have authoritative HR data in the midPoint to start with. It is usually easier to correlate other resources to midPoint users later on, if the users were created from a reasonably reliable HR data. Also, it will usually take some tweaking to get the HR import right. The possibility to easily clean up midPoint and get to a clean slate is extremely useful. HOwever, this "wipeout" approach is possible only if the HR feed is the first resource that is connected to midPoint.

A clever reader would notice, that we assumed that the source feed will be taken from a CSV file. This is indeed the case in many deployments. The CSV file is usually produced as an automatic scheduled export of HR system data, running every night. If a new employee or contractor is about to join the company, there is usually no hurry. This information is entered into the HR system at least a few days in advance, therefore daily CSV export is perfectly acceptable. However, there may be cases when we want a faster response. Maybe we do not even want additional burden of dealing with CSV exports. Of course, there is a solution. In theory, any connector can be used for source resource. There are specialized connectors that are taking data directly from the HR system. For example, there is a connector for Oracle HCM system. Unfortunately, there is no connector that can take data from SAP HR system yet.

Synchronization and provisioning are intimately related. Everything that we have explained about provisioning in the previous chapter also applies to synchronization. In fact, provisioning and synchronization are just applications of the same basic mechanisms. Provisioning starts with modification of a user. Synchronization starts a bit earlier: inbound mappings are used to map values from source system to the user. The result of inbound mapping evaluation is modification of the user object. According to midPoint principles, it does not matter how the user was modified. The reaction is the same: accounts are provisioned, modified or deleted.

The synchronization (inbound processing) and provisioning (outbound processing) usually happen in one seamless operation. For example, the HR connector detects update in the last name of an employee. That modification is applied to midPoint user, therefore the family name of midPoint user is updated. The operation continues by evaluating all templates, roles and outbound mappings. The outbound mappings usually map the family name change to the resource attributes. Therefore, the resource accounts linked to the user are immediately updated. All of that happens in a single operation. That is how midPoint works. MidPoint is not a human. It will never procrastinate (unless explicitly instructed do to so). MidPoint will not postpone the operation for later if the operation can be executed immediately. MidPoint tries to get the data right on the first try. Therefore, there are no specialized propagation or provisioning tasks that you might know from older IDM systems. MidPoint does not need them.

There are other advantages in doing everything in one operation. It is all one operation, therefore midPoint knows all the details: what was the cause, what is the effect, what exactly has been changed. Such context is extremely important for troubleshooting. Some IDM systems decouple the cause from the effect. Such a divided approach may have its advantages, but it is an absolute nightmare when an engineer needs to figure out why a certain effect happened. For that reason, midPoint has both the cause and the effects bundled together in a single operation. Therefore, it is much easier to figure out what is going on. Having the cause and effect connected in one operation makes it possible to neatly record entire operation in the audit trail. Then there is another huge advantage: midPoint knows exactly what has been changed. This means that midPoint does not only know the new value of a property. MidPoint also knows the old value and values that were added or removed. This is a complete description of the change that we call a delta. This is recorded at the beginning of the operation, and propagated all the way until the operation is done. Therefore the mappings can be smart. This approach enables a lot of interesting behavioral patterns. For example, it is quite easy for midPoint to implement the "last change wins" policy. In this case midPoint will simply overwrite only those attributes that are really changed in operation. MidPoint can leave other values untouched. In fact, this is the default behavior of midPoint. It is a very useful behavior during deployment of a new IDM system.

Careful processing of the operations allows configurations that are not feasible with older IDM systems, e.g. a resource that is both a source and a target. In fact a lot of IDM systems can have resource that is both a source and a target - as long as it is a source for one attribute and a target for another attribute. However, midPoint can live with a resource where the same attribute is both a source and a target. In fact there may be many sources and many targets for the same property at the same time. This is a very useful configuration, indeed. Just think about telephone number property. It is usually something that the user sets up himself. This may be set up by some kind of specialized self-service, it may be updated by a call center call, the user may update that in his Active Directory profile …​ there are many ways how this information is changed. Yet, we want this property to be consistent. We want telephone number to be the same everywhere. We do not care where it was changed. We just want to propagate the last change from anywhere to all the other systems. MidPoint can easily do this. Just specify both inbound and outbound mappings for the same attribute:

In this case the change in user property telephoneNumber will be propagated to the account attribute mobile (outbound change). But also a change in the account attribute mobile will be propagated back to user property telephoneNumber (inbound change). Last change wins.

A clever reader certainly grumbles something about infinite loops now. No need to not worry here. MidPoint can see complete operation context, both inbound and outbound sides. Therefore, midPoint knows when to stop processing the operation. There are even mechanism how to avoid loops caused by connectors detecting changes caused by the connector itself. MidPoint will break those loops automatically.

Synchronization is simple, in theory. However, as usual, the devil is in the details. Similarly to provisioning, there is no one single "synchronization protocol" that would work for all the source systems. Every system type has its own way to synchronize data. Some systems (such as LDAP servers) even have several mechanisms to choose from. Then there are source systems that have no practical way to implement efficient synchronization. We would refer to such methods as synchronization strategies.

Specific details of each synchronization strategy should be an internal matter of connector implementation. The strategy is configured on connector level, and the details should be, theoretically, hidden inside the connector. MidPoint would not know and would not need to know what synchronization strategy is used. That might work in an ideal world. Yet, we live in a practical world, and there are many details that leak through the synchronization abstraction.

Let us use LDAP as an example. LDAP is, theoretically, a standard. However, the standard does not specify any synchronization mechanism. There is experimental RFC 4533, however it is not widely adopted. Yet, synchronization capabilities are necessary, and every major LDAP server provides some synchronization mechanism. Some mechanisms are quite good, some are not. There is an ancient "Retro change log", going back to Netscape/iPlanet LDAP servers, that is still used today. Active Directory, in a very typical way, has its own "DirSync" synchronization mechanism. OpenLDAP has yet another mechanism based on the access log. There is RFC 4533, which is used so rarely, that there was no request to implement it in midPoint LDAP connector. And then there is a catch-all synchronization mechanism that looks for recent changes based on modifyTimestamp attribute.

In theory, all the synchronization strategies above should be equivalent. But they are not. For example, some variants of "Retro change log" synchronization cannot reliably detect rename operations. There may be problems with delete operations as well, especially if coupled with rename operations. Almost every mechanism has its quirks. And then there is the modifyTimestamp, which is the most problematic of all.

Unfortunately, it is quite common to use a synchronization strategy based on last modification timestamp. Not just for LDAP, but also for database tables and other types of source systems. This is perhaps understandable, as this is a very simple mechanism. However, it has a lot of problems. The obvious problems can be caused by de-synchronized time on network, although in the age of Network Time Protocol (NTP) this should not be a problem at all. The other problem is a timestamp granularity. If the timestamp is granular to one second, that can be a big problem. One second is a very long time for a computer. A lot can happen in one second. Therefore, the connector has to include the "boundary" second to both consecutive synchronization runs, which means that the records may be processed twice. Going for millisecond granularity makes the problem less severe, but the problem is still there.

However, the worst problem is that this strategy cannot detect deleted objects. Deleted objects are not there anymore, they do not have last modification timestamp, therefore they will not be included the search. This means that there must be a reconciliation process running together with live synchronization. But wait a minute, it is usually recommended running reconciliation anyway, as a form of "safety net", isn’t it? It is, but the difference is in the timing. It is one thing to run reconciliation once a week to make sure that no records were missed. Yet, it is a completely different thing to run reconciliation every hour to make sure deleted objects are properly handled. This makes a huge difference, especially for deployments with millions of entries. Strategies based on last modification timestamp may look like a good idea at the beginning. However, they usually turn into a major liability in the long run. Avoid them if you can.

The bottom line is, that synchronization strategies are not created equal. In fact, the individual strategies tend to have vastly different characteristics. Our advice is to learn how each synchronization strategy works, what are the limitations and when it fails. Also, avoid the use of strategies based on last modification timestamp if there is any other viable alternative.

Mappings and expressions form a very powerful mechanism. In fact, most of midPoint configuration is about setting up correct mappings. However, with great power comes great responsibility, and mappings may look a bit intimidating at a first sight. Fortunately, there are some tips and tricks that make the life with mappings and expressions a bit easier.

Most mappings are aware of the context in which they are used. Therefore, paths of mapping sources and targets can be shortened - or even left out entirely. Activation and credential mappings used in the HR feed example are the obvious cases. Yet, even paths in ordinary mappings may be shortened. For example take the outbound mapping source:

As the mapping knows that its source is a focus (user) the definition may be shortened:

Typical midPoint deployment has tens or hundreds of mappings. Deployments with thousands of mappings are definitely feasible. There are two things that can make maintaining the mappings easier. Optionally, you can specify the mapping name. Mapping name will appear in the log files and some error messages. It may be easier to identify which mapping is causing problems, or it may help locate the trace of mapping execution in the log file. Mapping can also have a description. The description can be used as a general-purpose comment or a documentation explaining what the mapping does.

Mappings can become quite complex. There may be multi-line scripting expression in the mapping, and it may not entirely obvious what is the input and output. Therefore, each mapping and each expression have an ability to enable tracing:

If tracing is enabled, then the mapping or expression execution will be recorded in the log files. Tracing can be enabled at both mapping level and expression level. Mapping tracing is shorter. It provides overview of the mapping inputs and outputs. Expression-level tracing is much more detailed.

However, even this level of tracing may not be enough to debug expression code. Therefore, there is a special expression function for logging. Arbitrary messages may be logged by script expression code:

Generally speaking, troubleshooting of mappings may be quite difficult as it is often intertwined with midPoint internal algorithms. Still, there are ways how to do it. The Troubleshooting chapter provides much more details on this.

The systems that midPoint connects to are not created equal. In fact, those systems significantly differ in their capabilities. Most systems can create accounts. However, not all of them can delete accounts. There are systems that keep the accounts forever, the accounts can be only permanently disabled. Yet another systems cannot enable or disable accounts. While most systems support password authentication, other system do not. There is a lot of natural diversity in the provisioning wilderness. The connector may introduce additional limitations as well. Even if target system supports a particular feature, connector may not have appropriate code to use it. MidPoint needs to take all these differences into consideration when executing synchronization and provisioning operations.

MidPoint refers to these features of the systems and connectors as resource capabilities. Although capabilities may look quite complex, they are essentially just a list of things that a connector and resource can do. MidPoint is aware of the resource capabilities, and their limitations. Therefore, midPoint can work with resource data correctly. E.g. midPoint will not try to modify account on a read-only resource.

Capabilities are usually automatically discovered by midPoint, and everything just works out of the box. There is usually no extra work to maintain the capabilities. Yet, sometimes there is a need to tweak the capabilities a bit. Maybe the connector cannot detect resource capabilities well enough. Maybe there is a read-only resource, but the connector has no way of knowing this. In that case, the write capabilities have to be manually disabled in midPoint. For that reason there are two sets of capabilities:

There are many ways how the capabilities can be tweaked by the administrator. Yet, there is one case that is particularly interesting for synchronization and provisioning: simulated activation capability.

MidPoint connectors can be tailored specifically for a particular system. E.g. there are often connectors that are developed specifically for one custom enterprise application. At the other side of the spectrum are generic connectors, that can fit a wide variety of systems and applications. LDAP, CSV and database table connectors are examples of such generic connectors. Such connectors are very useful, they are used in almost every midPoint deployment. However, there is no standardized way to disable an account in database table or a CSV file. Various columns and various values are used to represent account activation status. Quite surprisingly, there is no standardized way to disable an account in LDAP directory either. That is bad news for midPoint. MidPoint takes a significant advantage from knowing whether account is disabled or enabled. We had to do something about this "disable ambiguity". And we did.

There is way to tell midPoint which attribute and what values are used to represent account activation status. Configured activation capability is used for that purpose:

Configured capability above specifies resource attribute active as the attribute that controls account activation status. If this attribute is set to value true then the account is enabled. If the attribute is set to value false then the account is disabled. That is it. Once this configured capability is part of resource definition, then midPoint will pretend that the resource can enable and disable accounts. Attempt to disable account will be transparently translated to modification of active attribute. Moreover, it also works the other way around. If an account has attribute active set to false value, midPoint will display that account as disabled. No extra logic or mapping is needed to achieve that. The capability does it all.

Previous example demonstrated the use of synchronization for HR data feed. That is the most obvious use of synchronization mechanisms. However, midPoint synchronization can do more tricks than just feeding data to midPoint. Synchronization can be used even for target resources. In that case the synchronization is usually used for several purposes:

Older IDM systems used synchronization mostly to get data from the source resources to IDM system. Synchronization in midPoint is much more powerful than that. It can be applied both to source systems and target systems, it can pull data, push data, detect inconsistencies and fix them. Synchronization is a general purpose mechanism, it is a real work-horse of identity management with midPoint. This is the principle of reuse again. Synchronization mechanism can be reused for variety of purposes.

In this example we will be using synchronization to connect existing LDAP server to midPoint. We assume that our midPoint is already connected to the HR system. We have imported the HR data. Now we have midPoint users created for all our employees. Then there is this LDAP server. It is really important LDAP server. This server is used by company intranet portal and also by a variety of smaller web applications. Those applications are using the LDAP server for user authentication and access authorization. The LDAP server was deployed many years ago. Initially it was populated by the HR data. However, the LDAP server was managed manually by a system administrator during all these years. Therefore, it is expected that there will be some accounts that belong to former employees. Also, it might have happened that some accounts are missing. It is quite likely that a lot of the accounts have wrong data.

First task is to set up the connector for this resource. As LDAP servers are used for identity management purpose all the time, MidPoint comes with a really good LDAP connector. All we need is to set up the resource to use that connector:

What we can see here is a slightly more sophisticated method to reference the connector. So far we have seen only a direct connector reference by OID. This works well for almost all the references in midPoint, because OID never changes. However, connectors are somehow elusive objects. Objects that represent connectors are dynamically created by midPoint when a connector is discovered. Therefore, the OID is generated at random when midPoint discovers new connector. There is no practical way how a system administrator can predict that OID. Yet, we still want our resource definitions to refer to a particular connector when we import the definition. Therefore, there is an alternative way to specify object references. This method is using a search filter instead of direct OID reference. When this resource definition is imported to midPoint, then midPoint will use that filter and look for LDAP connector. If that connector is found, then the OID of that connector is placed in the reference (connectorRef). Therefore, the next time midPoint will be using this resource, it can follow the OID directly. This is a very convenient method. However, there are few limitations. Firstly, the filter is resolved only during import. Which means that it is resolved only once. If the connector is not present at import time, then the reference needs to be corrected manually. Secondly, this approach works if there is only one LDAP connector deployed to midPoint. This is usually the case. However, the connector framework can contain several connectors of the same type in different versions. This is a very useful feature for gradual connector upgrades, testing of new connector versions and so on. Yet, in case that the filter matches more than one object, the import will fail. In that case the connector reference has to be set up manually.

Once we have proper reference to LDAP connector we need to configure the connection:

This is all very similar to the configuration of the other resource that were already presented in this book. It should be quite self-explanatory – except perhaps for the configuration of result handlers. Result handlers are little helpers that come with the ConnId connector framework. The purpose of the result handlers is to assist simpler connectors in filtering and post-processing search results. However, LDAP connector is no ordinary simple connector. LDAP connector is mature and full-featured connector that can do everything without any help from such annoying little creatures as those result handlers. ConnId result handlers do not add any value here. In fact, they may even be harmful. LDAP protocol has a lot of peculiarities, such as case-insensitivity that applies to almost all the aspects of LDAP data – except for some notable exceptions. The connector is aware of those peculiarities, but the handlers are not. If the handlers are turned on (which is the default) they may get in the way and ruin the data. Therefore, it is always strongly recommended to explicitly turn off the handler when a full-featured connector is used.

The basic resource configuration above is sufficient to connect to the resource. Therefore, the test connection operation on resource details page should be successful. This configuration may also be used to list the accounts. However, LDAP servers support many object classes and midPoint does not yet know which object class represents account. Therefore, we need to add schema handling section to our resource:

There should be outbound mapping for each mandatory LDAP attribute for the inetOrgPerson object class. Such mappings are very typical for a target resource definition.

Once we set up the schema handling, we should be able to conveniently list LDAP accounts in midPoint. However, we need to switch to the Resource view instead of Repository view. The accounts are stored in the LDAP server and midPoint can access them. Therefore, the accounts are listed in the Resource view. HOwever, midPoint have not processed the accounts yet. Therefore, there are no account shadows in midPoint repository yet. That is the reason that the Repository view is empty. Now we are going to do something about it.

We are going to import (or reconcile) the resource accounts. However, if we try to do this now, nothing would really happen. The accounts are not linked to users, therefore midPoint will not synchronize the attributes. MidPoint was not told to do anything with the accounts. Therefore, midPoint will do nothing. That is one of midPoint principles: midPoint will not change the accounts in any way unless it is explicitly told to do so. We would rather do nothing than to destroy the data.

Before we can import the accounts, we need to set up the synchronization configuration for this resource. There are accounts in the LDAP server that should belong to users that already exist in midPoint. We want to link them. However, we do not want to do the linking manually. We would rather set up a correlation expression that does this automatically:

This correlation expression is going to match account attribute employeeNumber and user property that is also named employeeNumber. Simply speaking: if account and user employee numbers match then we assume that they should be linked. In that case midPoint decides that synchronization situation is unlinked (they should be linked, but they are not yet linked). We want midPoint to link the account in this case, therefore we define appropriate reaction:

This will take care of accounts for whose we can find an owner. But what to do with other accounts? We will do nothing about them just yet. Therefore, we do not need to define any other reactions. This may be somehow surprising. We do not want illegal accounts in our LDAP server, do we? Then perhaps we would like to see a reaction to delete unmatched accounts, right? That would be a good approach, but it is just too early for this. We do not want to delete unmatched account just now. There may be accounts that are perfectly legal, just the employeeNumber attribute is missing or mistyped. Data errors like those happen all the time, especially when the data were managed manually. We do not want to over-react and start deleting accounts too early. Therefore, we will go just with this one synchronization reaction for now.

Now it is the right time to start import or reconciliation task. After the task is finished the situation may look like this:

It looks like we had quite a good data in the LDAP server. Most of the accounts were successfully correlated and linked to their owners. Yet, there are few accounts that were not correlated. Those accounts ended up in unmatched situation. You can resolve this situation by manually linking the unmatched accounts to their users. Simply click on the small triangle button next to the unmatched entry and select Change owner from the context menu. Then select the right user (Isabella Irvine) in the dialog that appears. After that the account is linked to the user. Repeat this process to link all unmatched accounts.

There is one interesting thing in the screenshot above. Have a look at the LDAP account identified by uid=carol. While most other accounts have their uid value taken from the surname of the user, this account is an exception. Even though the uid value is obviously wrong, midPoint have linked the account correctly to the user (Carol Cooper). The reason is that we have set up midPoint to use employeeNumber for correlation. Even accounts whose usernames violate the convention can be automatically linked to their owners - as long as there is any reliable piece of information that can be used for correlation.

When all the accounts are linked to their owners, it is the right time to complete the synchronization policy. Now we can tell midPoint to delete any unmatched account. That is the case when an illegal account is created in LDAP server. We can also tell midPoint to unlink any account that was deleted in LDAP server:

There may be some accounts in the LDAP server that have wrong attribute values. By "wrong" we mean that the attributes have different values than the values that are computed by the outbound mappings. However, midPoint will not correct those values just yet. Remember the midPoint principle that it will not change the account unless we have explicitly told to do so? Those accounts are in the linked situation, and we have not configured any reaction for this situation. Therefore midPoint did nothing. Now we need to tell midPoint to synchronize the values:

A clever readers is now surely wondering whether we have forgotten something. We have, indeed. Attribute values are synchronized by running reconciliation process. However, our outbound mappings will not work in reconciliation. They do not have any explicit definition of strength, therefore midPoint assumes normal strength. Those mappings are supposed to implement the last change wins strategy. Therefore, reconciliation cannot overwrite the account data, as midPoint does not know whether it was account attribute or user property that was the last to change. If midPoint is not sure about something, then it will assume a conservative position and do nothing. We do not want to destroy the data. Therefore, what we need to do now is to let midPoint know that we really mean it, that the mappings are really strong:

Clever reader is uneasy once again. What is this limitations thing here? Simply speaking, the limitations specify that the attribute is optional (minOccurs=0) and that it is single-valued (maxOccurs=1). But, isn’t midPoint supposed to be completely schema-aware and figure that all by itself? Yes, it is. In fact, that is the reason why we need to override the information from the schema using this limitations element here. The cn attribute is specified in LDAP schema as a mandatory attribute. However, we have just specified outbound mapping for that attribute. Even if midPoint user does not provide any value for attribute cn, we can still determine that value by using the expression. Therefore, even though LDAP schema specifies attribute cn as mandatory, we want to present that attribute as optional in midPoint user interface. Hence the minOccurs limitation. The maxOccurs limitation is immediately obvious to anyone who is intimately familiar with LDAP peculiarities. In the LDAP world, almost everything is multi-valued by default. Even commonly used attributes for account identifiers and names are multi-valued. Nobody is really using them as multi-valued attributes, because vast majority of applications will probably explode if they ever encounter two values in the cn attribute. Yet, those attributes are formally defined as multi-valued in LDAP schema, and that is what midPoint gets from LDAP connector. The maxOccurs limitation is overriding the schema, and forcing midPoint to handle this attribute as if it was single-value attribute.

That is all. Now you can schedule reconciliation tasks to keep an eye on the LDAP server. The task will correct any attribute values that step out of line and delete any illegal accounts. This is how synchronization tasks can be useful, even in case of pure target resources.

However, there is one last word of warning. Those accounts were synchronized and linked to existing midPoint users. The accounts were not created by midPoint. Therefore, there is nothing in midPoint that would say that those accounts should exist. In midPoint parlance, there are no assignments for those accounts. MidPoint makes clear distinction between policy and reality. Therefore, midPoint is aware that those accounts exist, but there is no policy statement that would justify their existence. By default, midPoint does nothing, and it will let the accounts live. The accounts will be created or deleted only if there is an explicit change in the assignments. There is no such change now, therefore the accounts are not deleted. However, this is quite a fragile situation. Accounts that are linked but not assigned can easily get deleted if midPoint administrator is not careful. Of course, there are methods to handle such situations. One way would be to create the assignments together with the links. Those that are interested in this method should look up keyword "legalize" in midPoint docs. However, there are much better methods how to handle this. Perhaps the best approach would be to utilize the roles (RBAC). This is the topic of the Role-Based Access Control chapter later. Yet, there are still more things to learn about synchronization until we get there.

Reconciliation is a process of comparing current state of an account (reality) to a desired state of the account (policy). Reconciliation does not only compare the accounts, it is actively fixing the inconsistencies. Reconciliation can correct wrong data on resources. Yet, it also works the other way. It can correct the data in midPoint. Therefore, reconciliation is one of the most useful tools in the identity management toolbox.

Reconciliation can be used in a variety of ways. Reconciliation can be initiated for one specific user by using midPoint user interface. In that case, midPoint compares the values of all user’s accounts to the values that were computed using the mappings. If there is difference, midPoint corrects account values. This approach is perfect for testing reconciliation setting using just a single user. This feature is also useful for fixing values of one specific user.

Reconciliation of a specific user may be useful, but it is an ad-hoc approach. We usually favor systemic approaches in identity management. Therefore, reconciliation is usually used in a form of a reconciliation task. Reconciliation task lists all the accounts on the resource, and then it reconciles each account, one by one. This is a way how to keep all resource accounts continuously synchronized.

There is a couple of things about reconciliation that can be somehow surprising. Firstly, reconciliation of an account may cause modification of a user. This happens if there are inbound mappings for that account. This is perhaps quite expected. However, the operation does not stop there. If a user is changed, then such change may propagate to other accounts on other resources, usually by the means of outbound mappings. MidPoint does not like procrastination, and therefore it will try to apply all the changes immediately. It means that reconciliation of one account may cause changes to other accounts. Which makes a lot of sense, yet it may be quite surprising. Secondly, reconciliation will skip any normal-strength mappings. We have already explained the reasons for that, but this is something that can surprise even an experienced midPoint engineer from time to time. If we are sure that we want the mapped value to be present in the account all the time then strong mappings are the way to go.

A curious reader that has already explored midPoint user interface has surely noticed recompute function. To the untrained eye, recompute does looks almost exactly the same as reconciliation. However, there are subtle differences. Firstly, recompute will not force the fetch of account data. When recomputing, the account attributes will be fetched from the resource, unless midPoint inevitably needs them for the computation. This usually happens if weak or strong mappings are used. However, if there are normal mappings only, then recompute may skip reading account data. MidPoint will compare and correct account attribute values only for those accounts that are fetched from resource during this process. That is how recompute works. The purpose of recompute is to correct data of midPoint users, which means evaluation of object templates and other internal policies. Correcting account data is more or less just a side effect of recompute.

On the other hand, reconciliation always tries to read all the accounts regardless whether they are needed for computation or not. Therefore, all the attributes on all the accounts are fixed. That is the purpose of reconciliation: correct the account data.

There is yet another difference between recompute and reconcile tasks. The purpose of recompute task is to correct user data. Therefore, recompute task will iterate over midPoint users. Recompute task will not detect new accounts on the resource, and it may even overlook if an account is deleted. However, reconciliation task is different. In fact reconciliation task has several stages. Main reconciliation stage lists all resource accounts. It determines owner of each account, compare the attributes and corrects them. As this process iterates over real accounts on a resource, it can easily detect new accounts. When the main stage is completed, then the next phase looks at account shadows stored in midPoint. The task looks for shadows that have not been processed in the main phase. Those are accounts that used to be on the resource some time ago, but have disappeared since. That is how reconciliation detects deleted accounts.

Rule of the thumb for the reconciliation versus recompute dilemma is this:

Reconciliation is really useful mechanism. It is reliable and thorough, but it is also quite slow, and it consumes o lot of computational and network resources. There are good reasons why reconciliation is such a heavyweight beast. Reconciliation works with absolute state of accounts. It means that reconciliation is reading all the accounts with all the values of all the attributes. Then it recomputes everything. Even the attributes and values that were not changed are recomputed. This is a very traditional and reliable way of computation, and that is also the way how the most of older identity management systems work.

Yet, there is also a better way. If we know that just one attribute was changed, we can recompute that single attribute only. We do not need to care about other attributes. Moreover, if we know that attribute foo has changed in such a way, that there is a new value bar, then it gets even better. We just need to recompute the value bar, and we need not care about any other values. This is what we like to call a relative change. We just care about the values that were changed. That is how midPoint works internally. We could say that MidPoint is relativistic system.

This is where delta comes in. Delta is a data structure that describes the change of a single midPoint object. Add delta describes a new midPoint object that is about to be created. Modify delta describes existing midPoint object where some properties have changed. Delete delta describes an object that is going to be deleted.

This is a very powerful mechanism. Just remember that everything in midPoint can be represented as an object: user, account, resource, role, security policy … everything. Therefore, delta can represent any change in midPoint. It may be a change of user password, deletion of an account, change of connector configuration or introduction of a new password policy. If all the changes can be represented in a uniform way, then they can also be handled in a uniform way. Therefore, it is easy for midPoint to record all the changes in an audit trail – including configuration changes. It is easy to route any change through an approval process. And so on. MidPoint can create a relatively simple mechanisms to handle changes, and then those mechanisms can be applied to any change of (almost) any object.

Let’s have a closer look at an anatomy of a delta. There are three types of delta: add, modify and delete. Add delta is quite simple. It contains a new object to be created.

Delete delta is even simpler. It contains just object identifier (OID) of an object to be deleted.

Last one is modify delta. This delta contains a description of modified properties of an existing object. As the object can change in a variety of ways, modify delta is the most complex of the three. Modify delta contains a list of item deltas. Each item delta describes how a particular part of an object changes. For example following delta describes that a new value pirate is added to a user property employeeType.

The item delta may have three modification types: add, delete and replace. Add modification means that new value or values are added to an item. Delete modification means that value or values are removed from an item.

In both add and delete cases, the values that are not mentioned in the delta are not affected. They remain unchanged as they are. However, replace modification is different. This means that all existing values of the item are going to be discarded, and they are replaced with the value or values from the delta.

The deltas are designed to work with both single-valued and multi-valued items. In fact, add modification and delete modification deltas are specifically designed with multi-value items in mind. Those deltas can work efficiently even in cases that there is a multi-valued attribute that has a very large number of values. There is a good reason for this. Multi-valued properties are quite common in the identity management field. Just think about how roles, groups, privileges and access control lists are usually implemented. Everybody that ever managed a large group in LDAP server will surely remember that experience in vivid colors. Fortunately, midPoint is designed to handle situations like these.

Everything in midPoint is designed to work with deltas: user interface, mappings, authorizations, auditing …​ all the way down to the low-level data storage components. Mappings are designed in a relativistic ways. That is one of the reasons why we need to explicitly specify sources of the mapping. Mapping source definitions are matched with items in the delta to control execution of the mapping. Deltas permeate entire midPoint computation. Deltas are input to the mappings, and mapping produce other deltas as output. Therefore, we can have a complete chain: deltas that are result of inbound mappings is applied to the user object, but those deltas are also input to outbound mappings. Everything is relativistic in midPoint.

This might seem to be a bit over-complicated at the beginning. But do not worry. You will get used to it. Clearly, this approach has major advantages.

However, a clever reader does not seem to be impressed. How can this relativistic approach conserve any significant portion of computational resources? We usually fetch the entire account from the resource anyway. Therefore, there is no harm to recompute all the attributes. The computation itself is fast, it is the fetch operation that is slow. Isn’t it? The clever reader is, of course, right. Or rather partially right at the very least. Most resources indeed fetch all the account attributes in a single efficient operation. For those cases, there is no big increase in efficiency if we go with the relativistic methods. However, there are exceptions. For example, some resources will not return all the values of big attributes, e.g. all the members of a large group. Additional requests are needed to fetch all the values – and there may be a lot of requests if the group is really large. Relativistic approach has significant benefit in those cases. The benefits will be even more obvious when we get to the live synchronization in the next section. Yet, performance is not the primary motivation for the relativistic approach. There is one extremely strong reason to go relativistic: data consistency. Consistency is something that brings ugly nightmares to many engineers that try to design distributed system. Identity management solution is in fact a distributed system, managing data in many independent applications and databases. It is also a very loosely-coupled distributed system. There is no support for locking or transactions in the connector. Even if there was some support, the vast majority of resource cannot provide those consistency mechanisms on their identity management APIs. This means that midPoint cannot rely on traditional data consistency mechanisms. That is why relativistic approach is so useful. Relativistic computation has a very high probability of achieving correct result even without locking or transactions. This is more than acceptable for typical identity management deployments. For those rare cases where relativistic computation can fluctuate there is always reconciliation as a last resort. Yet, thanks to the relativistic nature of midPoint, the need for reconciliation is significantly reduced.

That was a lot of long words, but clever reader seems to be satisfied now. At least for a while. For the readers that are still scratching their heads there is quite a simple summary: relativistic approach of midPoint can do miracles. For example, midPoint resource can be both sources and targets, even a single attribute can be both source and target of information. It is the relativistic approach that allows features like this. The principle of relativity is relatively simple. Yet, its effect in midPoint is nothing short of being revolutionary.

MidPoint has a range of synchronization mechanisms. Slow, brutal but reliable reconciliation is at one end. Live synchronization is on the other. Live synchronization is a lightweight mechanism that can provide almost-real-time synchronization capabilities. Live synchronization is specifically looking for recent changes on a resource. When such changes are detected, live synchronization mechanisms process those changes immediately. The synchronization delay is usually in order of seconds or minutes if live synchronization is used properly.

Unlike reconciliation, live synchronization is not triggered manually. That would make very little sense. Live synchronization works in a long-running task, repeatedly looking for fresh changes in short time intervals. If a resource is configured for synchronization, then all that is needed to run live synchronization is to set up a live synchronization task. MidPoint user interface can be used to do that easily.

An example of live synchronization task was provided in the HR feed section above. Live synchronization task wakes up at regular intervals. Each time the task waves up, it invokes the connector. Connectors capable of live synchronization have special operation that is used to get fresh changes from the resource. The connector can support any reasonable change detection mechanism – in theory. Yet, two mechanisms are commonly used in practice:

All live synchronization methods need to keep the track of what changes are "recent", i.e. which changes were already processed by midPoint and which were not processed yet. There is usually some value that needs to be remembered by midPoint: timestamp of last scan, last sequence number in the change log, serial number of last processed change and so on. Each connector has a different value with a connector-specific meaning. MidPoint refers to those values as "tokens". The most recent token is stored in the live synchronization task. That is how midPoint keeps track of processed changes. There are (hopefuly quite rare) cases when resource and midPoint token get out of alignment. This may happen in cases such as the resource database is restored from a backup, if network time gets out of synchronization and so on. If that happen, then deleting the token from the live synchronization task is usually all it takes to get the synchronization running again.

Live synchronization is fast and very efficient. However, it is not entirely reliable. MidPoint may miss some changes. This is quite a rare situation, but it may happen. Reconciliation will surely remedy the situation in such a case. Just remember, all the synchronization mechanism share the same configuration. It is perfectly acceptable to run live synchronization and reconciliation on the same resource at the same time. But of course, it would be a good idea to run reconciliation less frequently than live synchronization.

Synchronization is one of the most important mechanisms in the entire identity management field. Primary purpose of synchronization is to get the data into midPoint. That is good approach when an identity management deployment begins: feed your midPoint with data first. Get the data from the HR system. Correlate the data with Active Directory. Connect all the major resources to midPoint and correlate the data again. MidPoint does not need to make any changes at this stage. In fact, it is perfectly good approach to make all the resource read-only at this stage. The point is to let midPoint see the data. Why do we need that?

This is a good start. But even if this is all that you do in the first step of the deployment it is still a major benefit. You will get better visibility, and with that comes better security. You have the data to analyze your environment, and plan next step of the identity management deployment. You won’t be blind any longer. That is extremely important. It is indeed a capital mistake to theorize before one has data.

MidPoint is designed as a schema-aware system. For every bit of data that passes through midPoint we have a complete definition. We know whether this is string, integer or timestamp. We know whether it is single-valued or multi-valued. We know whether it is optional or mandatory. We know whether this is a sensitive piece of data that requires extra protection. We know whether it is part of technical meta-data that we usually do not want to show by default. And often we also know what label we should use when we are presenting the data and how that label translates to other languages. We know quite a lot about the data that we work with. All the objects that midPoint works with are completely defined by the schema. There is a schema for user, role, org, resource, system configuration and everything else.

Such awareness of the schema brings significant advantages to midPoint. The most obvious advantage is in data presentation. We know that we need to render a calendar selector because that particular data property is timestamp. We know that we need to render a text field with a plus button to add values because that particular property is a multi-valued string. And we know that some fields should be disabled because those properties are read-only. This behavior is not hard-coded. Vast majority of midPoint user interface is rendered by interpreting midPoint schema.

This approach is absolutely crucial for any serious IDM system. One of the reasons is that the IDM system works with data that are retrieved from other systems (resources). It is just not possible to hard-code midPoint user interface for all the various attributes that all the possible resources could have. A different strategy is needed here, a strategy that is much more dynamic.

When midPoint connects to a new resource for the first time it attempts to retrieve resource schema. The schema specifies what object classes the resource supports, which attributes the object classes have, what types are those and so on. MidPoint transforms this schema to its own native format and stores that in the resource definition. This means that midPoint has the schema available anytime it is needed for dynamic interpretation. That schema is used to display resource data in the most natural and user-friendly way. It is also used by automatic data type conversions, which makes mapping configuration easier.

MidPoint schema is not just a nice way how to create user, role or organizational structure. It has a much deeper meaning. The primary purpose of a schema is integration, data translation and unification. A clever reader would certainly remember that we have already talked about star topology or hub-and-spoke integration pattern. MidPoint is like a hub of the wheel and all the resources connect to midPoint as spokes. MidPoint is actively discouraging direct resource-to-resource communication. Everything in midPoint is built for resource-to-midPoint and midPoint-to-resource communication. MidPoint is always the center – and for a good reason. All resource data need to be translated to and from a midPoint “data language”. Thus, midPoint creates a common language that everybody can understand. And this is exactly the purpose of midPoint schema. The schema of user, role, org and service is designed to contain properties that are often used in identity-related integration scenarios. Therefore, an engineer who is designing a mapping is quite likely to find a suitable property in midPoint schema that is prepared to be used.

MidPoint schema forms a lingua franca, a common language that can be translating to various data dialects used by the resources. But it also provides a basic framework that can be reused for many midPoint deployments. Therefore, an engineer starting a new deployment does not need to start on a completely green field. The basic schema will always be there to provide a starting point.

When it comes to identity management field, there is one concept that is at the center of everything: a concept of user. MidPoint is no exception. User is undoubtedly the most important object in the entire midPoint schema. Therefore it is worth to have a closer look at how this object looks like. This is going to be a really educative lesson, as it will explain several fundamental principles of midPoint.

User is represented by schema datatype identified as UserType. Adding the Type suffix to data types is a common convention in midPoint. There is UserType, RoleType, OrgType, ResourceType and so on. This convention is partially historic, partially given by XML Schema conventions, partially a convenience to developers. Regardless of the origins, this convention is used for all the data types in midPoint schemas. You will get used to it eventually.

UserType is what we call an object definition in midPoint parlance. This means that UserType data structure specifies a complete midPoint object with all the things that any self-respecting object needs. There is object identifier (OID), name that can be presented in different forms and languages, free-form description and so on. All midPoint objects have those things.

The UserType data structure has many additional properties, containers and references. Property is a primitive data item such as string, integer or a timestamp. Container is a complex data structure that contains a bunch of properties or other containers. Reference is a pointer to another midPoint object.

Definition of UserType is summarized in the following table:

This is a basic outline of the schema for UserType. This description is slightly simplified. Not all the items that are defined for UserType are shown in the table above. Deprecated items are not shown at all. Only some operational properties are shown. Some items are simplified or entirely omitted for clarity.

Following example illustrates the use of midPoint UserType schema:

Most of the items in midPoint schema are quite ordinary and they behave as expected. Such as the fullName property. The property can be set and changed by using midPoint user interface. But then there are some extraordinary items. Those are automatically determined and controlled by midPoint core. Those items are essential for correct operation of midPoint. Therefore they are called operational items. Operational items are usually not directly displayed in the user interface. They are either completely hidden, displayed indirectly or displayed only when user chooses to display them.

MidPoint schema has grown and evolved over time. And it is still evolving. Therefore, it is quite expected that the schema will slightly change over time. However, we do not affect midPoint deployments by incompatible schema changes. Therefore items are usually not removed from midPoint schema without a warning. An item that we do not want is marked as deprecated first. At that point such item is still working as before. However, it is not displayed in the user interface to discourage use of that item. Deprecated items are removed in one of the subsequent midPoint releases. This gives enough time for midPoint users to adapt to schema changes.

There is also another kind of schema evolution. Development of most midPoint features is quick and straightforward. But then there are features that are quite complex or features that involve some degree of exploration to implement. Those features cannot be implemented in a single midPoint release. There are also features that are provided to the midPoint community as a "preview" to gather feedback for further development. All such features are marked as experimental. Those features are not officially supported, but you are free to use them at your own risk. Most new features require extensions of midPoint schema. This is also true for those experimental features. But when going experimental, there is a fair chance that something will change in the future. Therefore, we are explicitly marking parts of the schema as experimental. This is a warning that those parts are likely to change. We are not promising any kind of compatibility for experimental parts of midPoint schema. They may change any time, they may even completely disappear. And there will be no deprecation or any other warning. Simply speaking: if you are dealing with experimental features, you are completely on your own. Do not come crying when those things stop working. You have been warned.

Time is cruel and everything that we do is in some way temporary. Except perhaps for stupidity, which seems to be utterly endless. But all other things have a beginning and an end. Employees have hiring date, contracts have end dates, users can be disabled, roles may get replaced and so on. We use the term activation to encompass all those things that deal with the questions of digital life and death of the objects.

The activation in itself is multi-dimensional and quite complex. It is composed of several properties that may change in somehow independent and somehow inter-dependent way. Following list provides a quick summary of activation properties:

The best way to explain how activation works is to describe the meaning and behavior of individual properties.

Administrative status defines the "administrative state" of the object (user), i.e. the explicit decision of the administrator. If administrative status is set, this property overrides any other constraints in the activation type. E.g. if this is set to enabled and the user is not yet valid (according to validity below), the user should be considered active. If set to disabled the user is considered inactive regardless of other settings. Therefore this property does not necessarily define an actual state of the object. It is a kind of "manual override". In fact, the most common setting for this property is to leave it unset and let other properties determine the state. If this property is not present then the other constraints in the activation type should be considered (namely validity properties, see below).

If the administrative status is not present and there are no other constraints in the activation type or if there is no activation type at all, then the object is assumed to be "enabled", i.e. that the user is active.

Validity refers to state when the object is considered legal or otherwise usable. In midPoint the validity is currently defined by two dates: the date from which the object is valid (validFrom) and the date to which an object is valid (validTo). When talking about users these dates usually represent the date when the contract with the user started (hiring date) and the date when the contract ends. The user is considered valid (active) between these two dates. The user is considered inactive before the validFrom date or after the validTo date.

It is perfectly OK to set just one of the dates or no date at all. If any date is not set then it is assumed to extend to infinity. E.g. if validFrom date is not set the user is considered active from the beginning of the universe to the moment specified by the validTo date.

The validity is overridden by the administrative status. Therefore, if administrative status is set to any non-empty value then the validity dates are not considered at all.

Activation is also influenced by object lifecycle. Object lifecycle specifies phases of object’s life, such as draft, proposed, active and deprecated. There are some lifecycle states in which the object is considered to be active. And there are other states when the object is considered to be inactive. The later states are important, because object lifecycle can completely override activation. This makes perfect sense. E.g. when an object is in draft state, it is just being prepared for use. Such object may have validity dates or administrative status that would normally activate it. But we do not want draft objects to be active yet. Such object may need a review and approval to transition to active lifecycle state. Only then it will really become active. This is just a rough overview of the lifecycle functionality that only scratches the surface. We will deal with object lifecycle details later in this book.

Activation is quite a complex matter that is spread out in several dimensions. Therefore it may not be entirely obvious which objects are active and which are not. For that reason midPoint provides an operational property effectiveStatus which shows the computed "effective state" of the object. Simply speaking it is a read-only property that tells whether the user should be considered active or inactive. The effective status is the result of combining several activation settings (administrative status, validity dates, etc.).

The effective status holds the result of a computation, therefore it is an operational property that is recomputed every time the status changes. The effective status should not be set directly. The effective status can be changed only indirectly by changing other activation properties.

The effective status is the property that is used by majority of midPoint code when determining whether a particular object is active or inactive. This property should always have a value in a normal case. If this property is not present then the computation haven’t taken place yet.

Similarly to effective status, there is yet another operational property validityStatus. This property reflects the state of validity constraints with respect to current time. The values are before, in and after, meaning the states before the validity intervals started, inside the validity interval and after the validity interval ended respectively.

Lockout status defines the state of user or account lock-out. Lock-out means that the account was temporarily disabled due to failed login attempts or a similar abuse attempt. This mechanism is usually used to avoid brute-force or dictionary password attacks and the lock will usually expire by itself in a matter of minutes.

This value is usually set by the resource or by midpoint internal authentication code. This value is mostly used to read the lockout status of a user or an account. This value is semi-writable. If the object is locked then it can be used to set it to the unlocked state. But not the other way around. It cannot be used to lock the account. Locking is always done by the authentication code.

Please note that even if user of account are in the normal (unlocked) state they still be disabled by administrative status or validity which will make them efficiently inactive.

There is also an informational property lockoutExpirationTimestamp that provides information about the expiration of the lock. However, not all resources may be able to provide such information.

There are several operational properties in the activation data structure that provide operational data about user activation:

Those properties are operational, therefore from the user point of view they are read-only. The values are automatically computed by midPoint and stored in the database.

Let’s see how that works on some examples. The simplest example is perhaps not even worth mentioning. A user without any activation data structure is considered to be active (enabled). When such user is stored in midPoint repository, midPoint will automatically compute effective status:

Administrator can disable such user by using administrative status property:

Once again, when such user object is stored after the modification, midPoint computes the value of effective status:

The use of administrative status is usually quite harsh. MidPoint deployments are often using validity constraints instead. For example, an employee that has employment contract for a year would look like this:

Given that this chapter was written in 2019, such user will be active. It will automatically switch to inactive state after the last day of 2019. However, if there is ever a need to explicitly disable the user, administrative status can still be used:

In this case the user is still in its validity interval. Hence the in value of validityStatus. But the administrative status is explicitly set to disabled. Therefore the resulting effective status is also disabled.

The concept of activation is not limited to users. Many midPoint objects have activation. Roles can expire, organizational units can be disabled and so on. Activation is a concept that has a very broad application in midPoint. Even assignments have activation, which is a crucial element in some configuration (e.g. multi-affiliation). Assignments are often used to model employment contracts, student affiliations, service contracts and similar concepts that have time boundaries. This is usually achieved by a clever use of assignment activation.

So far we have talked mostly about the use schema (UserType data type). However, the entire midPoint schema is quite complex. There are many types of objects and there are thousands of data types overall. It would be almost impossible to manage such a big schema directly in midPoint code. Therefore the schema is defined in special definition files that are used by midPoint in several ways. It is used by the user interface to automatically render form fields. It is used by midPoint expression engine to automatically convert data types. It is even used by midPoint build process (compilation) to make sure that the schema is properly used in midPoint code. MidPoint is completely schema-aware system from top to bottom.

Schema obviously plays a crucial role in everything that midPoint does. Therefore it may be interesting to have a look at schema definition. This can be particularly useful for engineers that are deploying midPoint professionally and that often needs to extend and customize the schema.

MidPoint schema is specified in XML Schema Definition (XSD) format. MidPoint schema is defined in several parts, but the most important is the "core" schema definition. The schema files reside in midPoint source code in schema component in the infra subsystem. Therefore schema files can be found in the resources part under the infra/schema subdirectory of midPoint source code. Schema files are also included in midPoint distribution package for convenience.

Every deployment engineer that takes midPoint deployments seriously should be aware of the schema. Hardcore engineer will surely open the XSD files in their favorite text editor in the terminal and analyze the definitions line-by-line. Developers could open the XSD files in their IDEs and have a nice organized look at the schema. But even an ordinary engineer could benefit from learning the basics of XSD and having a look at a few important data types in midPoint schema.

Schema definition is not just about the properties, containers and data types. Crucial part of the schema definition is in-line documentation. Most of the data types and items are documented by using XSD in-line documentation mechanism. Therefore a huge amount of details about midPoint can be learned by exploring the schema. We have tried to make that process easier by developing schemadoc mechanism. Schemadoc is a process that takes raw midPoint schema and generates HTML documentation out of that. This task is part of midPoint build process and generated documentation is a result of midPoint build. Schemadoc is also available online. Just search for "schemadoc" in midPoint docs.

Schema is not just a description how midPoint works. MidPoint schema is part of midPoint itself. It is used when midPoint is compiled. It is parsed when midPoint starts. It is used by midPoint core and user interface. MidPoint is complex and even the experts can be sometimes wrong. MidPoint documentation is quite extensive, therefore it may be misleading or out of date at places. But not the schema. Schema is always right. Otherwise midPoint won’t work. Schema is the law.

MidPoint schema is quite rich. Many of the properties that are frequently used in IDM deployments are already part of midPoint schema. But reality has always a way to bring unexpected things. Therefore, midPoint deployments won’t get far if midPoint schema cannot be extended.

Vast majority of midPoint schema is available at compile-time. This means that such schema is used during compilation (build) of midPoint. That "static" part of schema is somehow hardcoded into midPoint itself and it would be very difficult to change. Therefore we have developed a mechanism to extend the schema at deployment-time. Small parts of the XSD definition can be provided when midPoint is deployed. MidPoint will read those definitions when it starts up. The static part of the schema is extended with those definitions. From that point on the extensions are part of midPoint schema. The extensions will be used by midPoint user interface, expression-processing code and all other parts of midPoint.

Our ExAmPLE company was quite happy with the progress of IDM deployment so far. Mappings were used to synchronize values of user names and all other common attributes. There is plenty of suitable properties for that in midPoint schema such as givenName and fullName. Even employeeNumber came very handy. But now they need to customize midPoint schema to better suit their very specific needs. The company management decided that the people look really cool in fancy hats. Therefore they will provide a hat for every employee. Which means that the IDM system needs to track hat size for all users. Hat size is not used in the IDM deployments very often, therefore it is not a part of standard midPoint schema. But fortunately, it is easy to extend the schema.

First step to extend midPoint schema is to prepare a small XSD file:

This file defines a new data structure UserTypeExtensionType. The name of this data structure does not really matter. What matters is that it is bound to an extension of UserType in the annotation part of the type definition. When midPoint reads this file, it will extend the definition of UserType with this data type.

The extension data type specifies just a single property: hatSize. This is an optional single-valued string property. Every user in midPoint will have this property. User interface will automatically display text input field for this property.

MidPoint administrator puts this XSD content into example.xsd file. Name of the file can be chosen arbitrarily as long as it has .xsd file extension. Administrator copies that file to schema subdirectory of midPoint home directory and restarts midPoint. From that point on the schema extension is active.

The users can now be extended with custom property:

There is a couple of important remarks to be made here. Firstly, all the extension properties are always placed in a special extension container in the objects. Even though the properties are placed inside a container, the user interface will present them in the same way as the static (native) midPoint properties.

Secondly, a clever reader surely noticed that we have used XML namespace here. We have omitted XML namespaces from the majority of other examples as they are not that important when working with midPoint objects. But schema is different. Namespaces are handled quite a strict way when working with the schema. Namespaces must be declared and namespace prefixes must be properly used in all XSD definitions. The most important namespace in this case is the target namespace of the extended schema. The URI for this namespace should be chosen in such a way that it is globally unique. The use of your DNS domain is the recommended technique.

Namespaces also should be used when working with extension container in users and other midPoint objects. This requirement is not that strict as midPoint can usually figure out the namespace. However, this may be a problem in case that several schema extensions are combined. Such combinations are possible in midPoint. MidPoint will simply parse all the XSD files in the schema directory and apply all of them as extensions. The namespace is used to differentiate between them. Therefore, if there is an expectation that several schema extensions will be used in the same deployment then the use of namespaces in object extension is more than recommended.

MidPoint schema does not just specify the "core" data model. MidPoint schema goes a bit further, and it can also specify the details of data presentation. This means that the schema can specify a label that should be used for particular data item, help text and so on. The XML Schema (XSD) cannot do this. But fortunately, XSD schema can be extended by annotations. Those annotations can be used to define the presentation properties of the items:

This works fine if your system works for just a single localization environment. But this is not enough in case that you need more than one language. MidPoint was born in Europe and we know quite well all the pain that comes with multi-language environments. MidPoint is designed to be localizable. Therefore you can simply use localization keys instead of actual text:

The actual text to be used for the label can be looked up in the localization catalog. However, using localization catalogs is a matter of its own. It will be covered by later chapters.

Majority of midPoint schema is pretty standard stuff. When you walk through the jungle of midPoint schema definition you can see all the usual wildlife: strings, integers, booleans, timestamps and binary values. But there are few species that are quite strange. However strange they might look, they are immensely useful. Their names are PolyString and Protected String.

PolyString is the stranger one of those two. Its name came from polymorphic string, which means a string that can take a variety of forms. In its simplest form PolyString is just a simple string that can be normalized. Normalization means that we convert the original string into some standard form, e.g. we are removing leading and trailing whitespace (trimming), we are converting all letters to lower case, simplifying national characters and so on.

Many ordinary midPoint properties are PolyStrings. Object name and user’s givenName, familyName and fullName and all PolyStrings. And yet not even a clever reader haven’t noticed anything suspicious about them so far. The reason for this is that normalization is almost transparent in midPoint PolyStrings. But now it is a time to have a peek inside. Let’s import a user that looks like this:

What is really stored in midPoint repository is this:

This all happens transparently. PolyStrings are displayed as strings in the user interface. They are handled (almost completely) as strings in the mappings. Ordinary midPoint user has no idea that the normalization happens at all. But why we bother to normalize strings at all? PolyString normalization has many practical uses. However, two of them are embedded quite deep in the way how midPoint works.

Firstly, normalization is used to provide reliable uniqueness mechanism. Usually we do not want a user with username semancik and another user with username Semancik or even Semančík. This may lead to confusion. As midPoint has uniqueness constraints on both the orig and norm parts of the name then such situation is completely avoided. All those usernames have the same normalized form, therefore the uniqueness constraint on norm part of the name will prohibit the use of all those forms at the same time.

Secondly, normalization is simple and elegant way how to conveniently search for objects. When PolyStrings are searched, the value from the query is normalized. Then the norm part of the PolyString is searched. Therefore, whether the query contains semancik, Semancik or semančík, it will always find the user entry above.

Default normalization algorithm in midPoint should be a good fit for most environments. But there are always deployments that are different. For example, characters such as hyphens (-) are usually not considered to be significant. But some deployments will consider aliceanderson and alice-anderson to be two different usernames. The default midPoint normalization mechanism will remove hyphens, therefore attempt to have two such users will end up with an error. But fortunately the normalization algorithm is customizable. There are several algorithms to choose from and they can even be parameterized. In the extreme case there is a way to develop a completely custom algorithm. Therefore the PolyString normalization should fit pretty much every deployment scenario.

But PolyString still has more tricks to do. The normalization is not much of a polymorphism yet. PolyString becomes a real shape-shifter in fully localized environments. PolyString is designed to store values that can have individualized representations in national environments. E.g. in multi-national deployments we may want to provide localized role names. Like this:

This is a mechanism to display midPoint to end users in their own language, complete with localized content of midPoint. This functionality is only partially implemented in midPoint 4.0 and it is considered to be experimental (i.e. unsupported). But this is a glimpse of how the future of midPoint schema may look like.

The other strange animal in the midPoint jungle is protected string (ProtectedStringType). Identity management systems often work with sensitive data such as user passwords. All the identity-related data usually need protection, but those sensitive data items need even better safeguards. This usually means that some kind of cryptographic technique needs to be employed. E.g. we do not want to store passwords in the cleartext form. Want them to be either hashed or encrypted. And that is what protected string is for. Protected string is basically just a simple string, but it has extra cryptographic protection.

If you have ever had something to deal with cryptography, you will probably know that cryptography is not simple. Even such a seemingly simple thing as password hash is quite complex when it comes to all the details. E.g. we do not want simple hash as that would not provide sufficient protection. We want salted hash. Which means that the salt value needs to be stored together with the string. Many algorithms are parametric and the parameters used during the hashing also need to be stored. And most importantly, we do not want to hard-wire midPoint to any specific algorithm. Cryptographic algorithms often do not age well and they need to be replaced. Therefore we also need to store algorithm identifiers with the value. If the value is encrypted, we also need to store key identifier, as several keys may be active at the same time. And so on. The cryptographic devil is in the tiny and often counter-intuitive details.

Protected string is a data structure that is designed to handle all those pesky cryptographic details and still pretend that the content of the data structure is just a string. Similarly to PolyString, the basic usage is quite simple. Data can be imported into midPoint by using clearValue element:

The data are automatically protected when the object is imported into midPoint:

Protected string data type supports cleartext representation, encryption using a symmetric algorithm and hashing. However, the data type itself is just a mechanism for storing the data. Whether specific protected string in the schema gets encrypted or hashed and at which point that happens is not controlled by the protected string itself. It is controlled by midPoint configuration and policies. For example, whether user password is encrypted or hashed is determined by midPoint security policy.

This section describes schema concepts that goes deeper into midPoint mechanisms and implementation. Awareness of those concepts will provide insight into how midPoint works. However, we have already talked about the schema quite a lot. And this chapter was quite low on practical examples. Feel free to skip the rest of this chapter if you want to get your hands dirty as soon as possible. But please make sure to come back later. You will have to learn those schema concepts eventually to get the best of midPoint functionality.

So far we have presented midPoint schema as a simple set of data types. There is UserType for users, RoleType for roles and so on. However, all the midPoint objects have something in common. For example, all of them have object identifier (OID), name, description and so on. We could simply copy definitions of those properties to all the data types. But that is not the best way how to do data modeling. The proper way is to create a type hierarchy. Therefore, there is an ObjectType data type that specifies all the items that all the object types share. However, midPoint schema is substantial and one common ancestor won’t be enough. MidPoint type hierarchy was evolving during midPoint development and now it forms quite a rich structure.

Following table is summarizing midPoint data types and their purpose.

Type hierarchy is a principle that is used in many software systems. This principle will be quite obvious to all software developers, but it may need some time to get used to for other engineers. However, the basic idea is quite simple. E.g. AbstractRoleType has all the items that are needed for an object to behave like a role. RoleType, OrgType, ServiceType and ArchetypeType are subtypes of AbstractRoleType. Therefore RoleType, OrgType, ServiceType and ArchetypeType can all behave like a role.

This may sound quite strange, why would we want an organizational unit to behave like a role. But the answer is quite obvious. Membership in an organizational unit may imply some privileges. Other IDM systems need complex rules in a form "if user belongs to organizational unit A then he will also have role X". But that is not needed in midPoint. Organizational unit is a role, therefore it can simply include all the roles that are needed. This means that the Role-Based Access Control (RBAC) principles can be applied to several object types. And this is a very typical trait of a midPoint philosophy: reuse of generic principles. We reuse existing principles instead of complicating the system by inventing a new single-purpose mechanism. As you will see later, this makes midPoint both elegant and powerful.

MidPoint configuration often needs to reference a particular item in a particular object. For example, mapping sources and target are references to properties and containers. However, midPoint data structures can be quite complex. For example, password is stored in property value that is located in container password which is in container credentials defined in UserType data type. It may be difficult to find a way in this little maze. And there may be even some unambiguous situations. For example, user status is controlled by property administrativeStatus that is in the activation container. But assignment also has an activation container and there is an assignment administrativeStatus. Therefore referencing an item by a simple name would not be enough. We need something more sophisticated here.

MidPoint is using the concept of item path to reference items in the schema. In its simplest form, item path is just a sequence of item names concatenated by slash characters. For example the path of user administrative status is

whereas the path of assignment activation status is

Item path provides an unambiguous reference to a specific item in midPoint schema. The path can be used in all the places where there is a need to reference a particular item. It is often used in mappings to specify sources and target. But it is also used in other places that we will mention in later chapters. The concept of path is deeply embedded in all midPoint operations. For example, modification deltas are using item path to precisely pinpoint the places in the object that are modified.

The path is used to locate a particular item in midPoint schema. But it is also used to reference a specific value in midPoint objects. In that case the path often looks exactly the same. As long as we are dealing only with single-value containers, the path can unambiguously point to a specific item. But we may get into trouble in case that multi-valued containers are used. And those are used in midPoint quite often. Assignment is one of those multi-valued container. User can have many assignments. If we want to disable one particular assignment, how do we do it? If we would use the path above then it is not clear which assignment should be disabled. Therefore, in case of multi-valued containers the path is extended with a container identifier in square brackets:

This path is unambiguously referencing administrativeStatus property in an activation container in a very specific assignment - an assignment container with identifier 123. This form of the path is used mostly in the deltas and user should not need to ever enter those paths manually. However, this form is often recorded in midPoint log files and other diagnostic output. Therefore it is very useful to be familiar with it.

You might wonder why there is an identifier for assignment but there is no identifier for activation. Both are containers, aren’t they? However, the clever reader already knows the answer. assignment is a multi-valued container. Therefore, identifier is needed to pinpoint a specific value of that container. But activation is a single-valued container. There is no danger of ambiguity. Therefore the identifier is not needed in this case.

This form of item path works fine if need to identity an item in a particular object. But sometimes we have a lot of objects and other data structures to choose from. For example a mapping can have several sources. And then there are expression variables. Therefore using simple paths would be ambiguous. In such case the path can start with an optional variable identifier:

The path above explicitly states that is should be applied to the content of variable focus. Therefore there is no danger that this path could be applied to a shadow object which also has the activation container. This form of item path is often used in path expression evaluators.

Clever reader is surely wondering about QNames now. The XML schema defines the elements in a form of QNames, which basically means "names in a namespace". Therefore element names and QNames. And path should use QNames as well. But so far all the names in the path looks like simple strings. Yes, they are simple strings. But they point to elements in the schema. While the path is correct and unambiguous, midPoint does not need the namespaces. Simple string (known as local part of QNames) are enough to navigate through the schema and automatically determine the namespaces. This is the same principle used for parsing XML, JSON or YAML document without namespace definitions. However, there may be ambiguities in case that several custom schema extensions are used. Those extensions may have elements with conflicting local parts. In that case an alternative form of item path can be used:

Clever reader may have also noticed that there are two types of namespaces that are often used in midPoint:

Indeed, the schema is divided into two big parts:

This is all about midPoint schema that you need to know right now. There is still much more to learn, as the entire midPoint schema is big and complex. And understanding of midPoint schema is absolutely crucial, as the schema is a foundation of everything that midPoint does. But the best way to do the learning is to do it on the go. You will learn more about midPoint schema as you will explore midPoint functionality.

Previous chapters were focused on account provisioning and synchronization. Which means that the primary focus was an account (or a similar resource object). This is what we call reality in midPoint way of thinking. Accounts are objects that exist in the databases and files on the resources. In that aspect they are almost tangible things. Existence of an account allows user to access a particular system, to execute operations and so on. Therefore, we consider an account to be something real.

But how do we know whether an account should exist or it should not exist? The situation would be quite clear if midPoint is the only source of truth. In that case if there is a linked shadow then account should exist. If there is no shadow, then account is illegal. But reality is almost never that simple. In real deployments MidPoint is not the only source of truth. It is usually human resource (HR) system that is the source of the truth – but only for some types of users, usually employees. Then there are external users, temporary workers, special personas for user administrators and so on. Some of them may have their own source systems similar to HR database. But for some users it may still be midPoint which is the ultimate source of truth. And that "truth" may be in fact only partial or compiled from several sources. To keep long story short: reality is messy and complicated. And it is often quite difficult to figure out which accounts particular user should have and which he should not have. Yet, for an IDM system this distinction is absolutely crucial. Various IDM systems came with broad range of mechanisms to handle this problem, and sadly, those mechanisms are often not very good. Fortunately, midPoint was designed from the beginning with a full awareness of this problem. Therefore, there is a clean distinction between reality and policy in midPoint.

Accounts, shadows and links are what we refer to as reality. Those describe what exists, what is. And there is a separate mechanism to describe policy. Policy, in midPoint parlance, means definition of what should be. In the ideal world reality and policy should be in accord. They should describe the same state of things. But we do not live in ideal world. Perfectly good accounts may be deleted by mistake, illegal accounts may be created, entitlements may get mixed up, attribute values destroyed – there are many dangers in the big wild world out there. And then there are scenarios when we actually want reality to be different than policy for some period of time. Those may be migration scenarios when a new system is being connected to midPoint and the data needs to be cleaned up. Reality and policy do not match exactly in practice. We all know that only too well. Therefore, midPoint is designed in such a way that it can graciously handle the differences between reality and policy.

When it comes to policy, the most important concept is an assignment. Simply speaking, assignment is a data structure which specifies that a particular user should have something. The simplest case is account assignment. This type of assignment states that the user should have an account on a particular resource. When such assignment is added to a user, there is suddenly a discrepancy between reality and policy. The assignment states that a user should have an account. But there is no such account yet. It is a nature of midPoint to align policy and reality as much as possible (unless it is told otherwise). Therefore midPoint will try to create missing account. Once that account is created reality and policy are aligned once again.

The mechanism that midPoint uses to define that a particular user needs an account, entitlement or other resource object is called construction. The simplest case is a construction that specifies to create an account (a.k.a. account construction):

The term construction means that object on that particular resource should be constructed. In this case the object on OpenLDAP resource should be constructed for this particular user. If no construction parameters are specified then a default account will be constructed. Which means that outbound mappings in the OpenLDAP resource definition will be used to set up the account.

Construction can be quite a complex data structure describing object types, object classes, attributes and so on. However,it is unlikely that they will be placed directly in assignment like this. But more on that later. What is important for now is that assignments specify policy.

After the assignment is added to a user and all the processing and provisioning takes place the situation looks like this:

Assignment is a definition of policy which states that an OpenLDAP account should exist for Alice. However, there is no such account. Therefore, MidPoint aligns reality and policy by creating that account. As for any other account there is a shadow and a link to track account ownership.

This may look like a very complicated method to do something simple. But this kind of thinking is really necessary to handle complex cases. There may be several assignments that mandate the same account. There may be assignments for the same accounts, but each assignment mandates different attributes or values. The account that the assignments mandate may exist already, e.g. it may be linked by previous reconciliation with the resource. There may be several accounts for the same user on the same resource (e.g. "ordinary" account and "testing" account). And so on. We will deal with various cases in this book. But the basic principle is the same: assignments are policy and midPoint is trying to align reality to match the policy.

There is much more in the concept of an assignment than just the very simple account assignment that was introduced above. Assignment is a generic mechanism that is used in midPoint for wide variety of cases, from simple account provisioning to really complex identity governance policies. But one specific assignment type is particularly interesting with respect to the topic of this chapter: role assignment.

The basic idea of Role-Based Access Control (RBAC) is simple: Instead of assigning account to users directly, let us group all accounts that a particular group of users need into a role. Then assign the role to users. Later on you may add new application to your system, and you probably want all the users to have account there. In that case all that is needed is to add that account to a role and recompute the users. All the users that should have the account will get the account. This principle is reused for many purposes in midPoint: accounts, privileges, authorizations, policies …​

Role is a special type of object in midPoint. Yet, as all midPoint objects, role has a very familiar structure:

Role object has its OID and name. The rest of the role usually specifies the privileges that the role gives to the users. But how do we give this role to users? That is what role assignment is good for:

User alice has role Business Analyst assigned. The assignment is using the familiar style of object references in midPoint, referring to the role by its OID. This is very useful, as the assignment stays the same in case that the role or the user are renamed - and both of those events are much more frequent that one would think.

Provisioning is the bread and butter of identity management. Therefore, it is quite understandable that the most natural usage of roles in midPoint is to automate provisioning. Provisioning roles are usually combining several construction statements. The idea is that a provisioning role should specify all the privileges that users of that role need. Therefore a Business Analyst role may look like this:

The usual case is that every employee need to have basic access to company functionality. In our case that access is granted by an account in central OpenLDAP directory. In addition to the basic LDAP account, business analysts need access to the CRM system. The role combines all the accounts that a business analyst needs. Assign that one role and the user has all that is needed to do the job.

But, what is that mysterious inducement thing? Think of inducement as indirect assignment. Assignments give privileges directly to the object in which they are placed. The assignment in the previous section gave account to the user because it was placed in the user object. However, here we do not want the accounts to be created for a role. We want accounts to be created for all the users that have the role. That is one order of indirection down the line. Therefore, we (usually) do not want to use assignments in roles. We want to use something that reflects this indirect relation. And that is exactly what inducement is. Inducement is very similar to assignment - in fact it has exactly the same structure. But while assignment is direct, inducement is indirect.

MidPoint user interface can show a nice summary of the inducements:

It is perhaps worth explaining what happens if this Business Analyst role is assigned to a user. When that role is assigned to a user, midPoint will process all the parts of role definition. MidPoint will take the inducements from the role and apply them to the user. In fact, midPoint will behave in almost the same way as if those construction statements were specified directly in user’s assignment. And then we have the familiar principle: policy mandates that two accounts should exit, but in reality there are no such accounts. Therefore midPoint creates the accounts. MidPoint also creates appropriate shadow objects and links them to the user.

Many applications implement at least some aspects of RBAC, as RBAC is a very useful way how to organize the privileges. However, almost all the applications limit the applicability of RBAC to the application itself. I.e. roles in an application can contain only those privileges that apply to that particular application. The roles cannot have privileges from other application. But IDM systems are different. IDM systems such as midPoint are reaching out to many applications (resources). Therefore, a single midPoint role can give access to many applications at once. No other application can do that.

We have already seen how outbound mappings can be used to set up account attributes. Roles can also contain outbound mappings, therefore they can be used for a similar purpose:

When the above role is assigned to a user, an account on OpenLDAP server will be created. The account will be provisioned in a usual way. All the outbound mappings from resource definition will be applied to set up the account. But there is one difference. The role specifies one additional outbound mapping for the account. This mapping will be included in the set of usual account mappings when the account will be provisioned. Therefore, the account will have attribute title set to Business Analyst.

This is a very typical way how midPoint deployments are set up:

At the time when midPoint is about to provision an account, all the mappings are merged and processed together. It is quite common that more than one role has a construction for the same account. All those constructions from all such roles are merged together, and they are added to the mappings specified in resource definition. All those mappings are used to compute final values of account attributes.

Most account attributes are single-valued. Attempt to set more than one value for such an attribute will end up with an error. Therefore, it does not make sense to specify more than one mapping for such an attribute. The mapping can be specified in a resource or in the role, but only one of those should be active at the time. Mapping conditions and strength can be used to selectively deactivate some mappings in more complicated cases. But it still means that only one mapping is active at a time.

However, some attributes are multi-value. In that case midPoint will merge the values from all the mappings. In that case several roles may contribute to the final set of attribute values, as can the mapping in resource definition. This is the usual case of attributes that specify privileges, such as permissions, authorization codes, access control list (ACL) entries and so on.

Merging of multi-value attributes is an easy way how to manage simple privileges in resources. However, midPoint contains a whole sophisticated mechanism for managing entitlements such as groups. There is an entire chapter in this book dedicated to entitlement management.

Ability to group privileges into roles is quite useful. But it is still not good enough unless your access control policy is extremely simple. Most practical policies require placing roles into roles, thus creating role hierarchy.

Let’s consider two work positions: clerk and supervisor. Clerk has some basic set of privileges. Supervisor can do everything that a clerk can do, but supervisor has some additional privileges. A naive way would be to simply copy all the clerk’s privileges in supervisor’s role. However, privileges are seldom static. Access control policies tend to change and evolve as much as the environment changes. It is likely that a clerk’s privileges will change. In that case we will need to update the supervisor’s role as well. This would be a maintenance burden. Now imagine hundreds or thousands of related roles that need constant maintenance. Any person maintaining such a structure will need superhuman precision and patience to do that.

A more natural idea would be to include clerk’s role into a supervisor’s role. If clerk’s privileges change, then also supervisor’s privileges are automatically updated. Maintenance is much easier. And that is the basic idea of role hierarchy. Basic privileges are placed into low-level roles. Low-level roles are combined to create a higher-level roles. Then those roles are combined as well. The process is repeated until there are all the roles that are needed for assignment to users.

Creating role hierarchies in midPoint is quite easy. A clever reader would already expect that this has something to do with inducements. And clever reader would be absolutely right. Role hierarchy is nothing more than a set of inducements between roles:

The inducement includes Clerk role in Supervisor role. When midPoint evaluates the Supervisor role, it will get all the inducements from both the Supervisor and Clerk roles. This process is almost transparent, it works almost as if the clerk’s privileges were copied in the supervisor’s role. All the constructions in all the inducements in both roles are processed. Therefore, supervisor will get all the accounts that a clerk would get, plus few extra privileges.

Both Clerk and Supervisor roles are likely to have construction for the same account. This is quite natural, as both clerk and supervisor would probably work with the same applications. However, their privileges will be different. This is where the merging mechanism becomes very useful. When a supervisor role is processed then privileges of clerk are merged with privileges of supervisor:

When supervisor’s role is processed, midPoint figures out that those two construction statements are referring to the same account. Therefore they will be merged together. Supervisor will get an account that will have priv attribute set to values read, create, approve, modify and delete.

MidPoint roles are very useful kind of animal, because they are used for almost everything in midPoint. MidPoint role be used as:

All those aspects can be combined into a single role. Therefore, such role can specify everything that is needed for the role holder to live a complete digital life: access to systems (accounts), entitlements, access to midPoint itself (e.g. for self-service), apply policy constraints and so on. Everything in one place.

Role universality may seem mundane and completely natural, but in fact it is quite unique and incredibly powerful idea. As you will see later, roles can be driven through approval process, lifecycle management can be applied to role, roles can be subject to policies, role compliance can be evaluated and so on. All of this applies to provisioning roles. But the same mechanism can be applied also to roles that govern the administration of midPoint itself. And even to meta-roles that specify high-level policies. Which means that in a strange post-modern way midPoint can be applied to itself. MidPoint can be its own manager.

There are many ways how a role hierarchy can be structured. One way is to create all roles as "end user" roles that are supposed to be directly assigned to user. The clerk-supervisor example above is that case. However, there are other approaches that are often used. For example, the low level roles are often modeled as application roles. Those roles deal with access to a single application, but they are not meant to be assigned directly to users. They are supposed to be abstract, to be the base "material" used to create other roles. Those higher-level roles are often called business roles, as they reflect the needs of the business, such as specific job or responsibility in a business process. Those roles are assigned to users. However, those are just two of many conventions and recommendations on designing a role structure. We will not dive deep into role modeling topics in this book. There is a plenty of literature on that topic already. We will rather focus on a technical implementation of role hierarchy in midPoint.

However, there are few cases that are frequent pain points in RBAC deployments. The philosophy of midPoint is to make IDM deployment easier, therefore it is quite natural that midPoint tries to address those particular issues.

One of the big troubles are application roles. There is usually a huge number of them, and they need to be maintained manually. It is usual practice that there is one application role for every privilege in the target system (resource), for every group for every organizational unit and so on. Application roles duplicate the information that is already present on the resource side. And as application roles are maintained manually, it is almost certain that this information will become inconsistent. This approach is sometime even recommended as a best practice. But the reality it is a maintenance nightmare.

This behavior is motivated by several factors. But perhaps the strongest factor is that it was very difficult to set up the privileges in older (first generation) IDM systems. Setup of an application role often required intimate knowledge of the entire IDM configuration as various tricks were used to implement entitlement management. Heavy connector support and customization were often necessary to provide even the very basic entitlement management. However, midPoint is different. There is a clean concept of construction, which is designed in such a way that a system (resource) administrator can understand. E.g. the construction refers to the native (non-mapped) names of resource attributes, it is referring to native object classes that are used on the resource, native identifiers and so on. The construction is built to use the language of the target system (resource), not the language of midPoint. Therefore, there is a good chance that system administrators can set up constructions easily. In addition to that, midPoint has a native support for entitlements such as groups and resource-side roles. Those are designed to be easy to use in constructions. Therefore there is very little need for application roles in midPoint. Higher-level roles can contain the constructions directly.

However, application roles may still be needed and in some cases they may even be useful. For example, there may be a common combination of privileges that is always assigned together. In that case it makes a lot of sense to create an application role. There may still be a need for application roles if strong governance or role lifecycle management is required, often as a means to govern the entitlements such as groups. Application roles might even be a legacy from a previous IDM system. As always, the best recommendation would be to analyze the RBAC policies and to use pragmatic thinking. Be careful about generic RBAC recommendations and be even more careful about recommendations that are meant for older IDM systems. MidPoint is different. Of course, midPoint can implement all those old-fashioned RBAC models. However, it will be a pain to maintain them. MidPoint can do better. Try to understand how midPoint works with the roles first, then apply those mechanisms to your RBAC policies. You might be surprised how midPoint can simplify the implementation of the policies.

In case that application roles are still needed, you may consider automating the management of such roles. MidPoint synchronization mechanism is really powerful. It can synchronize users and accounts, but it is designed to synchronize almost anything with anything. Therefore, it can be used to automatically create application roles from all the LDAP groups. While this approach still have some drawbacks, it automates the most painful parts of application role’s maintenance.

There is yet another practice that is quite common, but mostly wrong. Many IDM deployments create "login roles" or "default roles" for each application (resource). Those roles are supposed to define basic properties of the account. And in some cases they are supposed to keep account in existence when such account is unassigned. While this practice is very common, it is complicated, messy and very difficult to maintain. MidPoint was specifically designed tn such a way that this practice is not necessary in midPoint deployments. Default account attributes are easy to set up by using resource definition. And even the ability to keep unassigned account is directly supported in midPoint by using existence mapping, which will be described later. Therefore, such "login roles" are not needed at all in midPoint deployments, and they are generally considered to be a bad practice.

At the first sight, the concept of assignment may seem quite mundane, maybe even over-complicated. In fact, it is a very powerful concept, and it has been a crucial part of midPoint design from the very beginning. Assignment is so much more that just a simple user-role connection:

For example, assignment validity period can be used to assign a role only for a temporary period:

As assignment and inducement are in fact the same data structure, similar approach can be used to disable parts of role hierarchy:

Many types and variants of assignments can be combined in a single user. Assignment validity periods may overlap, there may be disabled assignments and enabled assignments for the same role at the same time, there may be several assignments to the same role with various parameters and so on. All reasonable combinations are supported, which allows modeling very complicated schemes such as multi-affiliation, multiple employment contracts and so on. Assignment is a crucial data structure, and we will be dealing with it in almost every chapter in the book.

RBAC is a nice and elegant way how to create and maintain access control policies. However, there is a serious danger: roles can be quite explosive. The role structure can easily get out of control and the roles may start to multiply. This is known as role explosion, and it is one of the nastiest drawbacks of access control system based on static roles. It is not uncommon for an organization to have much more roles than it has users. This creates a recurring maintenance nightmare. Fortunately, midPoint has a very powerful support for dynamic roles that can significantly reduce or even completely eliminate the impact of role explosion.

To understand dynamic roles, we first need to understand what is wrong with static roles. Many organizations have jobs that are very similar, they just differ is some small detail. For example, all bank tellers are similar, the difference is just their branch office. Similarly, all the assistant jobs are pretty much the same. The difference is the department or section that they work for. Therefore, there is Sales Assistant, Engineering Assistant, Logistics Assistant - and a hundred or so similar roles. Almost all the privileges in those roles are the same. Of course, we can create an (abstract) role Assistant that will have all the common privileges. But we still need those hundreds of specific assistant roles. And then it gets even worse, because there may be Senior Sales Assistant, Trainee Sales Assistant, Senior Engineering Assistant, …​

The key to the role explosion is a realization that those "exploded" roles are created in an algorithmic way. Maybe we do not need Sales Assistant, Engineering Assistant and Logistics Assistant roles at all. Maybe we need just one Assistant role. The organizational unit (sales, engineering or logistics) is just a parameter to that role. Then the number of roles can be significantly reduced. This is what we call parametric roles.

Parametric roles are not your ordinary garden-variety static roles that just contain a set of privileges. Parametric roles need to be much smarter. E.g. the Assistant role need an algorithm, that takes the organization unit as an input, and it outputs a privileges that are appropriate for that organizational unit. This may be a simple expression that determines correct group name based on organizational unit name. But it may also be quite a complex code that determines most efficient location of home directories and other resources based on office location. There is no free lunch. The algorithm that was used to generate the number of "exploded" roles will not magically disappear. In case of parametric roles that algorithm needs to be placed in the role itself. But it still may be much easier to maintain a couple of expression than to maintain thousands upon thousands of roles.

The usual problem with parametric roles is, quite obviously, the presence of the parameters. The parameters cannot be stored with the role, as they are different for each assignment of the role. The parameters also cannot be stored directly with the user, as the user may have the same role assigned with a different set of parameters. Fortunately, midPoint was designed with this problem in mind and this was one of the big motivations to create a concept of assignment. Assignment is the right place to store the parameters, as it is the data structure that associates user with a specific role.

ExAmPLE is a very progressive company. Similarly to other corporations they have functional organizational structure. But their employees are also organized in teams. Each team can have a manager and ordinary members. The team membership is represented by custom attributes in LDAP server. Each user has two custom multi-value attributes: exampleTeamMember and exampleTeamManager. Both attributes expect team name as their value.

The naïve way to handle this would be to create two roles for each team. But there are hundreds of team and that would be a maintenance nightmare. A smarter solution is to use parametric roles. There will be two roles only: Team Member and Team Manager. Those roles will take custom property teamName as parameter. But where does this property comes from? It comes from assignment extension. Each time the team role is assigned there needs to be a parameter in the assignment:

This is the first part of the solution. The second part are the roles. The roles need to be a bit smarter to use the teamName parameter:

Resulting LDAP account looks like this:

This setup is illustrated in the following diagram:

This is the basic mechanism of parametric roles. It is incredibly powerful mechanism. Unfortunately, current implementation of parametric roles in midPoint is quite limited. While midPoint was designed with parametric roles in mind, the implementation is not yet finished. Therefore support for parametric roles is quite limited. MidPoint core supports parametric roles quite well. Assignment parameters and mappings should work perfectly. However, the support for assignment parameters in midPoint user interface is very limited. In fact, the production-quality support is limited only to the couple of hardcoded parameters (orgRef and tenantRef) and even that leaves a lot to be desired. While we would like to improve support for parametric roles in midPoint, we have to listen to what midPoint subscribers are saying. Our development priorities are influenced by midPoint platform subscribers. So far platform subscribers prioritized other features and therefore there was not sufficient funding to completely finish user interface support for parametric roles.

MidPoint roles are usually applied to users. But midPoint roles are universal. The roles can be applied to almost any midPoint object. Roles can be applied to users, organizations, services and even to roles themselves.

Simply speaking, meta-roles are roles applied to other roles. Ordinary role applies its characteristics to a user. Meta-role applies its characteristics to another role. This is perfectly possible in midPoint, as role can be applied to almost any midPoint object. Then why not apply a role to another role? This may seem like a pretty useless exercise, but the truth is that meta-roles are tremendously useful.

History is repeating, they say. The fact is that repetition is daily bread in almost all IDM deployments. E.g. many business roles have something in common. For example, the business roles have similar approval process. There may be role classes that have similar exclusion policies that are part of global segregation of duties (SoD) policy. There are roles that are tied to entitlements in a systematic way and so on. Roles, organizational units, services and other role-like objects tend to be quite similar. Therefore, applying meta-roles to them can be very useful.

So far all the roles that we have seen were composed exclusively from inducements. That made perfect sense, as all those things that were in the inducement did not apply to the role itself. Privileges specified in inducements applied to users that the role was assigned to. However, in this case, we want to apply meta-role to a role. The effects of a meta-role should apply to the role, not to the user. Therefore assignment is used instead of inducement:

This may seem similar to a role hierarchy. However, it is a completely different animal. The crucial difference is that the meta-role is applied to the role, and not to the user. The inducements in the meta-role often contain policies such as approval policy, or construction clauses that create groups or organizational units. We usually do not want to create a group for each user. Yet, we often want to create a group for a role. That’s what meta-role can do.

A clever reader would probably notice that meta-roles can be used to set up different types of roles. We could have meta-role for application role, business role and so on. And clever reader, as always, would be right. However, given just the metaroles, there would be few bits still missing here to create a full-featured type system. Those missing bits are implemented in a form of archetypes. Simply speaking, archetypes are meta-roles, with some optimizations and improved user experience. More on that later.

It may be difficult to understand the concept of meta-roles from such a short and very abstract description. But do not worry. As meta-roles are often used in midPoint, we will get back to the meta-roles on several occasions. Meta-roles often allow simplification of complex problems by creating a very elegant solutions. For now, it is enough to remember that roles can be applied to almost anything in midPoint, including themselves.

This section is where we will get all thoughtful and philosophical. The people that are bored with philosophical questions should skip the rest of this chapter. We will also throw some dirt on almost every access control model in existence. Therefore, the people that maintain dogmatic beliefs about IAM mechanisms should skip this section as well. On the other hand, open-minded people are quite likely to enjoy it.

Role-based Access Control (RBAC) is just one of many access control models. There many variants of RBAC and there are other access control models that are based on a completely different paradigm. One such popular model is Attribute-Based Access Control (ABAC). ABAC is based on an idea that access to the systems can be determined dynamically just based on "attributes". Simply speaking, we can imagine ABAC as a one big algorithm that takes "attributes" as an input and decides whether access should be allowed or denied.

ABAC is very popular in the access management (AM) community because of its simplicity. And it all makes much sense as it is much simpler and faster to evaluate one expression than to sift through a mountain of roles. The problem with ABAC is manageability. ABAC assumes that all access control decisions could be based on algorithms and that they can be made anytime a decision is needed. However, that is almost never the case in larger practical deployments.

Many professionals responsible for identity management dream about complete automation of access control. It would be a marvel if an IDM system could automatically determine the privileges of every person simply based on the organizational unit and work responsibilities of that person. It would be perfect to get that information from an HR system, process it through a set of algorithms and automatically provision correct privileges to everybody. That is a very nice dream. But reality has a different idea. Such automated approach never really works in practice.

First problem is at the very start: HR data are almost never correct. There is very little motivation for the HR data to be completely correct. It is not a big issues if someone has a wrong job code or organizational unit code in the HR system. The business goes on, the salary is paid, everybody is happy. There is no really efficient feedback loop that would force corrections in HR data. Until the IDM system is deployed, that is. But it takes years or decades for a typical company to get to deployment of IDM system. At that point the HR data are beyond repair. The corrections that need to be done in the HR system are substantial. Even in small organizations it is very difficult to correct HR data manually. Bigger deployments absolutely require proper tooling to do that job. But even with good tooling it can take a lot of time. Many IDM deployments were significantly delayed or even canceled because of data quality problems. This method does not work very well.

The fundamental problem here is in the overall approach. IDM system should not fail when the input data are wrong. There should be procedures how to correct those data. And the IDM deployment should not be delayed because of wrong input data. That would be like refusing to use your reading glasses because the text you are reading is wrong. IDM systems are essential tools that help you to clean up the data. The IDM system should be deployed and it should be used to manage data quality on day-to-day basis. It is naive to think that once the data are cleaned up they will stay clean. The processes that lead to data errors will continue, therefore data errors will appear all the time. The crucial insight is to accept that there will be data errors and to design the mechanisms to detect and correct them.

There are many manual and ad-hoc decisions that need to be made in practical IDM deployments. And not just during the deployment. Many ad-hoc decisions must be made during routine operation. Privileges need to be assigned manually to compensate for missing input data. Privileges need to be corrected, input data need to be temporarily overridden, policy exceptions has to be made. There are many things that need to do manually. Such decisions are made almost on day-to-day basis. For ABAC and similar systems this would mean that a policy needs to be updated on a day-to-day basis. And ABAC is not designed for that.

This leads to another big problem of ABAC and similar "flexible" access control models. Even if HR are data are correct, the data usually do not provide all the information needed to completely provision the user with privileges. The HR data are often limited to organizational unit and formal code of the work position. However, this is often miles away from the job that user really does. The usual solution to this problem is that the user requests the privileges that are needed to to a job. Such request is then routed through appropriate approval process. And that request is a big problem for ABAC. What should the user request to get the privileges? Should the user request a change in ABAC policy? That would not be practical. Should the user request a new value for an attribute? Which attribute? And what value that should be? Can be somehow create a catalog of the things that a user can request? Once again, ABAC is not designed for this. But all those problems are very easy to solve in RBAC. User is expected to request a role. And it is quite easy to create a role catalog. But as ABAC does not have roles, there is nothing that a user can get a grip on. There is no "handle" that would allow the user to make sense from the ABAC policies.

This is all a consequence of yet another ABAC problem. While ABAC policy may be easy to set up, it is quite difficult to analyze and maintain. Which users are affected by this particular policy statement? How many users will be affected if I make this change to ABAC policy? ABAC systems would need a complex simulation algorithms to answer those questions. However, it is all quite trivial in RBAC. Policies are encapsulated into roles. Therefore, only the users that have those roles are affected. The roles also divide the policy to a smaller, manageable pieces. Each of the roles can have its own state and lifecycle. Therefore, it is not that difficult to work with two versions of the same role at the same time. Old version is still assigned to some users, but we are deprecating that and slowly migrating to a new version. Such continuous processes are difficult to do in ABAC.

And there is still one crucial problem when ABAC is used in provisioning scenarios. ABAC policies often benefit from the fact that complete data about the user accessing the system are available when the access control decision is made. The crucial part of that data is called context. This includes data such as time of day, network location of the user, recent events related to the user, real-time estimate of the risk and so on. However, such data are simply not available in provisioning scenarios. Accounts are usually provisioned long before the first access to the account is made. Therefore, many of the advantages of ABAC are useless in identity management scenarios that rely on provisioning.

However, ABAC is not a complete failure. ABAC is very useful in customer-oriented identity and access management (CIAM). Customer identities are usually "lightweight" and the policies are simple. But when there is a need to manage employees, teachers, contractors and similar "heavyweight" identities then ABAC almost always fails.

The fact that ABAC fails in complex practical IDM deployments does not mean that RBAC is ideal. Quite the contrary. RBAC has problems of its own and the applicability of pure RBAC in practical IDM deployments is very limited. Many of the problems of RBAC model motivated engineers to develop ABAC and similar models. In fact, the "algorithmic" idea of ABAC is not entirely bad. Only if we had a way how to combine ABAC and RBAC …​ Oh, but there is a way! We did it already.

MidPoint combines RBAC and ABAC by putting expressions into roles. We have seen that already. When role is assigned, the expressions in the role gets evaluated. And there can be any complex algorithm in the expression, even a complete ABAC policy. At least in theory. MidPoint expressions do not make access control decisions, because it is not the job of an IDM system to make such decisions. IDM system should set up the account. It provides the "material" for an authorization system to make a correct decisions. Therefore, midPoint goes as close to ABAC as a provisioning system can go. In an extreme case the entire ABAC policy can be implemented in outbound expressions in resource definition. But there is a good reason nobody does that.

Dividing the policy into smaller parts brings substantial advantage. Therefore many midPoint deployments are very RBAC-like. There are many roles and rich role hierarchies. Role expressions are used in moderation. But there are also deployments that are using parametric roles and role expressions extensively. In such cases there is a smaller number of roles, almost no role hierarchy, but the roles are smarter. Those are more ABAC-like deployments. But roles are still there. The roles act as "handles" for users to understand the policies, to give names to relevant parts of the policy. This combined approach works surprisingly well.

Of course, this idea is not new. Many RBAC-like systems use some kind of "smart" behavior inside the roles. However, so far we haven’t seen anything as comprehensive as midPoint role-based access control model. Therefore we had to invent an impressive marketing name for our creation. Due to a momentary lapse of imagination we dubbed it Advanced Hybrid RBAC. But whatever you choose to call it, the fact is that this approach is very useful in practice. It can transform apparent chaos into something that can be efficiently managed.

Data that come to midPoint are seldom complete and clean. Quite the contrary. The data that come from the "feeds" are often incomplete, they are not very precise and sometimes several sources may not even agree on a value for a particular data item. Inbound mappings can be used to sort out some of these problems. However, inbound mappings are designed to work in isolation. They work only for one particular resource. However, it is often needed to gather data from several resources and then look at all of them at once. Inbound mappings cannot do that very well, as they are tied to a particular resource. However, they can be used to gather all the relevant data in the user object. Then we can use some kind of mechanism to have a look at user object when those data are gathered together. That is what object templates do.

Simply speaking, object template is a set of mappings that is applied to a particular midPoint object. For example, user template is applied to all user objects. The mappings in the object template can produce new values for the object. For instance, a very typical use of object template is computation of user’s full name:

The mapping above will compute the value of user’s fullName property from givenName and familyName by using a simple Groovy expression. It is a weak mapping, therefore it will compute the full name only in case that it is not present already.

Object template can be used to do variety of things to all kinds of midPoint objects. This chapter will cover the most important functionality of object templates.

Importing an object template definition into midPoint will not do much. The template will not be used just by being imported. There can be several object templates for different types of objects, archetypes and even organizations. MidPoint will not know how to use the template. Therefore, the use of the template needs to be specified in a configuration. The simplest and most common way to use an object template is to configure its use in the system configuration.

This configuration activates the object template for use by all object of UserType type. Therefore, this template will be applied to all midPoint users.

Alternatively, user interface can be used to activate an object template. Navigate to Configuration  System  Object policy, create a new policy using the btn:New[] button. Specify reference to your template, select User in type field and click btn:Save[].

User template is applied every time an object is changed or explicitly recomputed (e.g. on reconciliation). User template is applied after all the inbound mappings are processed. Inbound mappings often copy important data to the focal objects (e.g. user objects). Therefore, the template can work on a data that are summarized from all the resources.

We have already seen how object template can be used to apply mappings on particular items of midPoint objects. But object template can also do other tricks. We will have a look at some of them here.

The processing of an object template is almost always focused on particular items of an object. Therefore, almost all the object template functionality is located in item element that references a particular item by its path:

The most common use of object template is to run mappings on items, such as the mapping to determine user’s full name above. Target of the mapping is automatically set to the item for which it is specified. Sources of the mapping need to be defined explicitly. But the basic idea is, that the user template should take properties of user as inputs. In other words, user template works on the same user object both as input and output.

Object template mappings often use static (literal) values or a very simple expressions, but mapping conditions are used to control application of the value. The easiest way to explain this is to use a simple example:

This mapping works for description property of the user. The mapping sets a fixed warning text specified as text literal in the mapping by using value expression evaluator. However, the mapping is not setting that value for all the objects. The mapping is applied only for objects that satisfy the condition. The condition is set to trigger for all the users that have hat size larger than 60.

However, mappings are made to be relativistic. This means that mappings react to changes. The same principle applies to mapping conditions. They are also relativistic. Therefore the mapping reacts to changes in the condition state. When user’s head grows and the hat size changes to a value over 60, then the mapping will add the warning. When the user’s head shrinks, then the warning disappears.

It may look that this mechanism is too complicated, if you look at single-valued properties only. But it all starts to make sense in case of multi-value items. Such as the assignments. But more on that in the next section.

While mappings are the things that object template does almost all the time, the template can also do other interesting things. First of all, object template can tweak the schema. MidPoint comes with a rich schema that is prepared to be used. However, the schema is not a perfect fit for all the deployments. Previous chapter described a method to extend the schema. But what we should do if we want to change the built-in schema of midPoint? Yes, object template is the right answer. The item specification can be used to modify the way how midPoint applies the schema:

The "additional name" is a nice and generic term that can fit many cultural environment. However, it is not very usual or intuitive in cultures that are not used to it (which means pretty much all the cultures). Therefore, almost all midPoint deployments that chose to use this property would like to rename it to something that feels more natural. Similarly for "given name" and "family name", which do not fit well in all the cultural environments. We have expected that and object template can be used to modify some aspects of built-in schema.

Object template can also be used to override object multiplicity, especially to change mandatory item into optional. MidPoint insists on having name set for all the users. But we may be able to compute a name from other properties, such as other user names, employee number or other identifiers, if we don’t want to present name item as mandatory in the user interface. We can compute the value of user’s full name from given name and family name. Therefore user may leave full name blank in the user interface. But user interface is driven by the schema. As name is mandatory in the schema, also the user interface will insist that the name field should be filled in. But this can be changed in the object template:

This configuration will make the name optional for the presentation purposes. This means that the user interface will treat name as optional. But core midPoint engine will still require the name to have a value. This gives object template a chance to generate the value for name. However, this means that name will still be present as read-write item in the user interface. We do not want that as name is supposed to be immutable identifier. We want to present name as read-only item. This can also be achieved by object template by using access configuration:

Object template can be used to adjust how midPoint user interface interprets the schema. But perhaps the most extreme measure is to eliminate certain item entirely. In fact, this happens quite often. MidPoint schema is rich and many deployments do not use all the items in midPoint schema. It makes little sense to present the items that are not used, therefore there is a way to tell midPoint that we want to completely ignore an item:

Object template can be used to do further tricks. It can be used to associate value enumeration with an item, e.g. to apply lookup table to a particular item. Object templates can be used to set up a validation expression for items. And a couple of other things. But more on that later.

Object templates are very flexible and they can be used for a lot of different things. But there is one particular usage of object template that appears in almost every deployment. It is an ability to automatically assign roles.

The basic idea is quite simple. An assignment is just an ordinary item. If we use object template mapping to populate that item with appropriate value, we will get automatic assignment of roles. Like this:

The trick here is to set up the mapping correctly. The simplest case is a conditional assignment of a role. Let’s suppose that we want to assign a Hatter role to everybody that has provided a hat size in user profile. We already know what to do, don’t we? Let’s use mapping condition:

Simple, isn’t it? The value part in the expression is an inside of a new assignment to create. And the assignment will be created on a condition that hatSize has a non-null, non-empty and non-zero value (that is a built-in evaluation of booleans in Groovy). The real trick here is the relativity of mapping conditions. Assignments are multi-valued. Therefore it is important to know when to add a value and when to remove one. MidPoint evaluates the condition twice. The condition is evaluated for an object before a change is applied (old object) first. The the condition is evaluated for an object after the change is applied (new object) once again. When the condition changes from false to true, Hatter role is assigned. When the condition changes from true to false, Hatter role is unassigned. Other assignments values are not changed by this mapping. Therefore many assignment mappings can happily coexist.

However, this is a very simple case. Typical midPoint deployments will have many roles. It is theoretically possible to create mappings like this for each and every role that has to be assigned automatically. But that would be a lot of repetitive work. And even worse, it is likely to become a major maintenance nightmare in the future. We are creative people and we do not really like repetitive work. And we really hate maintenance nightmares. Therefore it is perhaps no big surprise that there is a better way to do this.

The most common use case for automatic role assignment is to look up the role using some of its properties. For example, let’s have suppose that our HR system provides job codes for our employees. Therefore we have extended midPoint schema with a custom property jobCode: