Practical Identity Management With MidPoint

The central concept of identity management is a data record which contains information about a person. This concept has many names: user profile, persona, user record, digital identity and many more. The most common name in the context of identity management is user account. Accounts usually hold information that describes the real-world person using a set of attributes such as given name and family name. However, probably the most important part of the account is the technical information that relates to the operation of an information system for which the account is created. This includes operational parameters such as location of user’s home directory, wide variety of permission information such as group and role membership, system resource limits and so on. User accounts are represented in a wide variety of forms, ranging from relational database records through structured data files to semi-structured text files. Regardless of the specific method used to store and process the records, the account is undoubtedly one of the most important concepts of IAM field - and so are the databases where the accounts are stored.

The account databases are as varied as the account types. Most account databases in the past were implemented as an integral part of the monolithic information system using the same database technology as the system used. This is an obvious choice, and it remains quite popular even today. Therefore, many accounts are stored in relational database tables and similar application data stores.

Application data stores are usually tightly bound to the application. Therefore, accounts stored in such databases are difficult to share with other applications. However, sharing account data across the organization is more than desirable. It makes very little sense to maintain account data in each database separately – especially if most the accounts are the same in each application. Therefore, there is a strong motivation to deploy account databases that can be shared by many applications.

Directory servers are built with the primary purpose to provide shared data storage to applications. While application databases usually use their own proprietary communication protocol, directory servers implement standardized protocols. While databases are built for application-specific data model, directory servers usually extend standardized data model, which improves interoperability. While databases may be heavyweight and expensive to scale, directory servers are designed to be lightweight and provide massive scalability. That makes directory servers almost ideal candidates for a shared account database.

Shared identity store is making user management easier. An account needs to be created and managed in one place only. Authentication still happens in each application separately. Yet, as the applications use the same credentials from the shared store, the user may use the same password for all the connected applications. This is an improvement over setting the password for each application separately.

Identity management solutions based on shared directory servers are simple and quite cost-efficient. Therefore, we have been giving the same advice for many years: if you can connect all your applications to an LDAP directory server, do not think too much about it and just do it. The problem is that this usually works only for very simple systems.

Directory servers are just databases that store information. Nothing more. The protocols and interfaces (APIs) used to access directory servers are designed as database interfaces. They are good for storing, searching and retrieving data. While the user account data often contain entitlement information (permissions, groups, roles, etc.), directory servers are not well-suited to evaluate them. I.e. directory server can provide information what permissions an account has, but it is not designed to make a decision whether to allow or deny a specific operation. Also, directory servers do not contain data about user sessions. Directory servers do not know whether the user is currently logged in or not. Many directory servers are used for basic authentication and even authorization. Yet, the directory servers were not designed to do this. Directory servers provide only the very basic capabilities. There are plug-ins and extensions that provide partial capabilities to support authentication and authorization. However, that does not change the fundamental design principles. Directory servers are databases, not authentication or authorization servers. Being databases, they should be used as such.

However, many applications use directory servers to centralize password authentication. In fact, this is somehow good and cost-efficient way to centralize password-based authentication, especially if you are just starting with identity and access management. Nevertheless, you should be aware that this a temporary solution. It has many limitations. The right way to do it is to use an authentication server instead of directory server. Access Management (AM) technologies can provide that.

While directory systems are not designed to handle complex authentication, access management (AM) systems are built to handle just that. Access management systems handle all the flavors of authentication, and even some authorization aspects. The principle of all access management systems is basically the same:

After that procedure, the user works with the application normally. Only the first access goes through the AM server. This is important for AM system performance and sizing, and it impacts session management functionality.

The applications only need to provide the code that integrates with the AM system. Except for that small integration code, applications do not need to implement any authentication code at all. It is the AM system that prompts for the password, not the application. This is a fundamental difference when compared to LDAP-based authentication mechanisms. In the LDAP case, it is the application that prompts for the password. In the AM case, the Access Management server does everything. Many applications do not even care how the user was authenticated. All they need to know is that he was authenticated and that the authentication was strong enough. This feature brings a very desirable flexibility to the entire application infrastructure. The authentication mechanism can be changed at any time without disrupting the applications. We live in an era when passwords are frowned upon and replaced by stronger authentication mechanisms. The flexibility that the AM-based approach brings may play a key part in that migration.

Identity management (IDM) is maybe the most overlooked and underestimated technology in the whole identity and access management (IAM) field. Yet IDM is a crucial part of almost every IAM solution. It is IDM that can bring substantial benefits to almost any organization. So, what that mysterious IDM thing really is?

Identity management is exactly what the name says: it is all about managing identities. It is about the processes to create Active Directory accounts and mailboxes for a new employee. IDM sets up accounts for students at the beginning of each school year. IDM makes it possible to immediately disable all access to a suspicious user during a security incident. IDM takes care of adding new privileges and removing old privileges of users during reorganization. IDM makes sure all the accounts are properly disabled when the employee leaves the company. IDM automatically sets up privileges for students and staff appropriate for their classes. IDM records access privileges of temporary workers, partners, support engineers and all the third-party identities that are not maintained in your human resources (HR) system. IDM automates the processes of role request and approval. IDM records every change in user privileges in the audit trail. IDM governs the annual reviews of roles and access privileges. IDM makes sure the copies of user data that are kept in the applications are synchronized and properly managed. IDM makes sure data are managed according to data protection rules. IDM does many other things that are absolutely essential for every organization to operate in an efficient and secure manner.

It looks like IDM is the best thing since the sliced bread. So where’s the catch? Oh yes, there is a catch. At least, there was a catch. The IDM systems used to be expensive. Very expensive. The IDM systems used to be so expensive, it was very difficult to justify the cost even with such substantial and clear benefits. That time is over now. Identity management is still not cheap. However, the benefits clearly outweigh the costs now.

Identity governance is basically an identity management taken to a higher business level. The identity management proper, sometimes called identity administration, is focused mainly on technical aspects of identity life-cycle such as automatic provisioning, synchronization, evaluation of the roles and computing attribute values. On the other hand, identity governance abstracts from the technical details, focusing on policies, roles, business rules, processes and data analysis. E.g. a governance system may deal with segregation of duties policy. It may drive the process of access certification. It may focus on automatic analysis and reporting of the identity, auditing and policy data. It drives remediation processes to address policy violations. It manages application of new and changed policies, evaluate how is your system compliant with policies and regulations and so on. It may evaluate risk levels, modeling overall organizational risk, identifying risk hot-spots. This field is sometimes referred to as governance, risk management and compliance (GRC).

Almost all IDM systems will need at least some governance features to be of any use in practical deployments. Moreover, many governance features are just refinement of concepts that originated in the IDM field many years ago. Therefore, the boundary between identity management and identity governance is quite fuzzy. The boundary is so fuzzy that new terms were invented for the unified field that includes the identity management proper together with identity governance. Identity governance and administration (IGA) is one of these terms. For us, the governance is just a natural continuation of identity management evolution.

Back in the 2010s, it was a common practice for identity governance features to be implemented by specialized products that are separated from their underlying IDM platforms. Many IDM and governance solutions are still divided into (at least) two products. This strategy brings new revenue streams for the vendors. Yet, it makes almost no sense from customer’s point of view. It perhaps comes without saying that reasonable IDM/IGA solutions should offer both the IDM and governance features in one unified and well aligned product.

Identity professionals, often motivated by marketing needs, like to invent new names and use them to describe the same thing. Therefore, there are many overlapping, overloaded and similar terms in use.

Identity management (IDM) is usually used to describe the low-level parts (technology), while identity governance is used to describe the high-level parts (business). Yet the boundary is very fuzzy and many IDM systems provide governance capabilities, and many governance systems provide low-level functions. Identity governance and administration (IGA) is a term supposed to describe both parts together. Governance, risk management and compliance (GRC) is a terms that was mostly used in the past to represent the high-level identity governance functionality, later known simply as identity governance. Identity security is a marketing term that roughly covers IGA functionality.

Overall, the terminology is very fluid. Vendors use their own terms, often choosing overloaded or confusing terminology. Industry analysts and consultants also adding their own terms and meanings to existing terms. Marketing terms are invented faster than the documentation can adapt, making the situation quite confusing. We have tried to compile the terminology as precisely as we could, while still making the terms understandable. We have chosen to follow established industry terminology when possible, even though many terms are overloaded and ambiguous. However, we did not want to increase the confusion by re-inventing the terminology. We are pointing out the ambiguities in the text as needed. At the very least, we are trying to use a consistent terminology in this book. When in doubt, please refer to the glossary.

A comprehensive Identity and Access Management solution cannot be built by using just a single component. There is no single product or solution that can provide all the necessary features. As the requirements are so complex and often even contradictory, it is very unlikely that there ever will be any single product that can do it all.

A clever combination of several components is needed to build complete solution. The right mix of ingredients for this IAM soup will always be slightly different, as no two IAM solutions are exactly the same.

There are three basic components that are required for any practical IAM deployment:

The following diagram shows how all these components fit together.

This is truly a composite solution. There are several components that have vastly different features and characteristics. When bound together into one solution, the result is something that is much more than just a sum of its part. The components support each other. The solution cannot be complete unless all three components are in place.

However, building a complete solution may be quite expensive, and it may take a long time. You have to start somewhere. If you have resources for just one product, then choose identity management. It is a good start. It is not as expensive to deploy and integration as access management system is. IDM brings good value, even quite early in the IAM program. Going for open source product will also keep the initial investment down. Staring with IDM is usually the best choice to start the IAM program.

Strictly speaking, Identity and Access Management (IAM) does not entirely fit into the information security field. The IAM goes far beyond information security. IAM can bring user comfort, reduce operation costs, speed up processes, and generally improve the efficiency of the organization. This is not what information security is concerned with. Even though IAM is not strictly part of information security, there is still a huge overlap. IAM deals with authentication, authorization, auditing, role management and governance of objects that are directly related to the information security. Therefore, IAM and information security have an intimate and very complicated relationship.

It is perhaps not too bold to say that the IAM is a pre-requisite to good information security. Especially the identity management (IDM) part is absolutely critical - even though this may not be that obvious at the first sight. Yet, the evidence speaks clearly. Security studies quite consistency rate the insider threat as one of the most severe threats for an organization. However, there is not much that the technical security countermeasures can do about the insider threat. The employee, contractor, partner, serviceman - they all get the access to your systems easily and legally. They can legally pass through even the strongest encryption and authentication because they have got the keys. Firewalls and VPNs will not stop them, because those people are meant to pass through them to do their jobs.

Vulnerabilities are there, obviously. With the population of thousands of users there is a good change that there is also an attacker. Maybe one particular engineer was fired yesterday. Yet, he still has VPN access and administration rights to the servers. As he might not be entirely happy about the way how he has been treated, the chances are he might be quite inclined to make your life a bit harder. Maybe leaking some company records would do the trick. Now we have a motivated attacker who will not be stopped by any countermeasures, and who can easily access the assets. Any security officer can predict the result without a need for a comprehensive risk analysis.

Information security has no clear answers to the insider threat. This is no easy issue to solve, as there is obviously a major security trade-off. The business people wants to access the assets easily to do their jobs, to keep the wheels of an organization turning. However, security team needs to protect the assets from the very users that are accessing them. There is no silver bullet to solve this issue. However, there is a couple of things that can be done to improve the situation:

Theoretically all of these things can be done manually. However, it is not feasible in practice. The reality is that information security seriously suffers - unless there is and IDM system that brings automation and visibility. Good information security without an IDM system is hardly possible.

Risk-based approach to identity management and governance is a very good idea. In fact, it is an excellent idea, one of the best ideas in decades. However, as with many great ideas, there are difficulties and drawbacks.

But wait a moment, what is this "risk-based" thing all about? To answer that question we have to make a quick road-trip through information security landscape.

The concept of risk comes from information security theory. Security practitioners have realized a long time ago that it is all but impossible to create a perfectly secure system. As you try to make a system more and more secure, every step is more expensive than the previous one. Every countermeasure is less efficient than the previous one, it is more intrusive, it is less flexible, harder to adapt to business needs. Eventually, the system gets to the state where it is practically useless for business, yet the system is still not completely secure.

Therefore, the security practitioners came with a concept of risk. Risk is a measure of danger that a particular asset is subjected to. An asset such as a customer database can be in risk with respect to particular threat, for example a hacker trying to steal the database to sell it to your competition. The risk tells about the probability of an asset being compromised. For example, keeping the database in a form of spreadsheet on decades old Windows machine connected to an open Internet is obviously quite a risky thing to do. The risk can be addressed using countermeasures. Countermeasures are all the things that we do to make systems more secure, ranging from operating system updates, through access-control systems up to bomb-proof doors and heavily-armed guards.

As it is not practical to completely secure a system, there is always some amount of risk that we have to accept. This is called residual risk, a risk that we are aware of, yet it is not efficient to reduce or eliminate the risk. Even though residual risk cannot be completely eliminated, there may be risk mitigation plans. For example, we may accept that there is a risk of an operating system vulnerability, and no amount of automated software updates, vulnerability database integrations and watchfulness is ever going to eliminate the risk completely. However, we can mitigate the risk by preparing plans to be executed when we are affected by a zero-day vulnerability. The plan may include disabling network access to vulnerable services as soon as we learn about the vulnerability, investigation that looks for traces of an attacker exploiting the vulnerability, emergency communication and contingency plans and so on. Risk mitigation is focused on making the impact of such an attack less painful, reducing the damage.

In an ideal situation we are completely aware of the risk, so we can implement countermeasures and prepare risk mitigation plans. However, we need to know quite a lot about the risk for the countermeasures and mitigation plans to be efficient. Risk is not just a single number, it is a multi-dimensional and often very complex concept. The amount of risk is evaluated in a risk assessment process, which is often very tedious and demanding exercise. Risk assessment process evaluates assets, to determine the value of the data and services. The process looks at threats, such as skills and motivations of an attacker. The assessment looks at vulnerabilities that attackers can use to gain access to our systems. It is also concerned with existing countermeasures, processes, policies and other details.

This means that the risk assessment process deals with big and complex data that cannot be processed by a human mind alone. The data are usually fed into risk models to determine the risk. Risk models are a set of complex mathematical formulas that transform data on assets, threats, vulnerabilities and all the other inputs into a multi-dimensional representation of risk areas. In theory, the risk model can tell us that we have a high risk in network security, especially when dealing with customer data - and we should really do something about it!

The results of risk assessment are meant to drive the implementation of countermeasures and risk mitigation plans. There are too many countermeasures and mitigation strategies to choose from, we cannot possibly implement them all. We want only the really efficient ones. We do not want to waste time and money on sophisticated encryption of data that are just copies of public information, do we? The risk assessment is supposed to tell us what is important and what is not. This is an important principle of efficient and systematic information security process: never guess, never go blind, let the risk guide you. Base your decisions on data. Implement countermeasures exactly where they are needed, where they are in a good position to address real risks.

However, information systems are not the easiest things to analyze. They never seem to stand still! The data are changing, new integration routes are added, systems are re-configured. Yet, the most annoying of all, user accounts and privileges change pretty much every day. By the time the risk assessment is done, the results are already out of date! How are we supposed to evaluate the risk, when everything around us is changing constantly?

The answer is, of course, automation. There are parts of risk assessment that cannot be automated. For example, there is no magic method to automatically assess business value of your data assets. However, some parts of risk assessment can be automated.

This is where we get back to identity management and governance. Almost all organizations are affected by insider threat, a threat posed by people that are already part of your organization. Employees, contractors, support engineers, cloud service providers - they already have access to your data. They do not need to hack anything, they do not need to overcome any countermeasures. They are already inside. All it takes to reveal your trade secrets are a simple copy and paste keystrokes. One file download is all it takes to sell your customer database. The proliferation of cloud services make such "exploits" entirely trivial. There is no technological countermeasure, no perimeter that could stop an insider to use a privilege that he or she already has.

This means that identity data heavily affect outcome of risk modeling. A system where almost all of your employees have unrestricted access to a customer database is very likely to pose much higher risk than all the network-related risks combined. A single person that has administrator-level access to almost every system in your organization is certainly a very attractive phishing target. There are high risks hidden in the identity data of almost any organization. Yet, such risk can be easily reduced by adjusting the access rights. But how to find such risks? Identity data are often complex, system-specific, distributed in many directories, cloud systems and application databases. Any identity practitioner can certainly see where this leads: identity management system, of course.

Identity management system is an ideal place for evaluation of risks related to identity and access data. Essential data are already in the database of identity management system: users, roles, role assignments, role composition, entitlements, everything is there. Entitlements can be assessed to assign a risk score to them. Then the data can be fed into a risk model, evaluating how are the entitlements combined in roles, how are teh roles assigned to users, identifying high-risk roles and users. The model is evaluated by the machine, quickly and efficiently. The efficiency opens up a whole range of possibilities, that form the essence of a risk-based approach to identity governance.

As we can evaluate a risk posed by any individual user, we can easily identify dangerous accumulation of privileges in the hands of a single user. Then we can focus on addressing this risk, analysing why have the privileges accumulated, whether they are all necessary, removing excess privileges to lower the risk. Perhaps we can consider changing business processes to divide the responsibilities among several users, lowering the risk even further. We can evaluate the risk model after each step, checking whether we have reached acceptable risk levels already.

The risk model can evaluate risk of each role. This allows detection of anomalies, such as assignment of a powerful high-risk role to an ordinary user that is supposed to be low-risk. Such role is likely to be assigned by mistake, or perhaps it was a role assigned during an emergency that was never removed. Unassignment of such role may be a quick way to reduce the risk. Smart system could suggest several types of such outliers, where privileges of an individual users stand out from the surrounding.

Once we have the concept of risk established in our system, we can use it in security policies. For example, it makes sense to require stronger password for high-risk users, or even better, automatically set up a multi-factor authentication for them. It may be desirable to re-certify privileges of high-risk users more frequently than those of low-risk users. Assignment of new role to a high-risk user may need to pass through additional approval stage. The policies can take the risk into consideration. This approach is often referred to as adaptive security.

There are even more advantages when the risk-based approach is applied also to other areas of identity and access management field. For example, it may be a good idea to require strong authentication from high-risk users, while allowing weaker authentication for low-risk users. This can be achieved in many ways, the easiest is perhaps for identity management system to propagate risk scores to access management user profiles.

Risk-based identity management and governance is indeed the right thing to do. However, the devil is in the details, and the reality is much harder than the glossy marketing brochures dare to admit. There are hidden dangers and dark corners on this route:

Information security is quite a strange field. You can clearly prove that your system is insecure, for example by successfully attacking your system. Yet, you can never completely prove that your system is secure. This limitation is a major source of confusion, and it also opens up opportunities for charlatans, offering their security snake oil on the market. As you cannot buy information security process, you cannot buy a ready-made risk-based approach to identity governance. You have to build it yourself. Having advanced and smart identity governance platform at your side is unquestionably a great help. However, such a platform is only a tool. Even the smartest and most expensive tool will not do all the work. It will make your work more efficient, but you still need to be the one driving it. One size does not fit all. Your organization is different to other organizations. That is what makes you unique, what gives you competitive edge, what makes you survive on the market. You cannot expect to buy a model or a process that fits your needs perfectly. You have to adapt and develop your own models, policies and processes. Having a starting point that is similar to your needs is a huge advantage. Starting from industry-specific frameworks, templates and samples will save you a lot of time. Go for these, whenever they are available. However, you will have to understand how the frameworks work, you will need to know what you are doing, as you will certainly need to adapt them to your needs.

Similarly to information security, "off the shelf" is mostly just an illusion in identity management and governance. Whatever the bold marketing statements say, you cannot just buy it and run it. No, not even in the cloud. You can buy identity governance platform as a service, but you cannot buy identity governance. You will have to learn a lot of things, you will have to dive deep into policies and models, you will have to do a lot of work yourself. Set your expectations realistically.

There is no single identity and access management solution that would suit everybody. Every deployment has specific needs and characteristics. Deployment in a big bank will probably focus on governance, role management and security. Deployment in small enterprise will focus on cost efficiency. Cloud provider will focus on scalability, user experience and simplicity. Simply speaking, one size does not fit all. Almost all IAM solutions use the same principal components. However, product choice, solution topology and configuration will significantly vary. Do not expect that you download a product, install it and that it will solve all your problems. It won’t. Customization is the key.

We consider identity management to be heart and brain of any IAM solution. This is one of the reasons why we have started midPoint project. The rest of this book will focus almost exclusively on identity management and the use of midPoint as the IDM component. This is the place where theory ends and practice begins.

MidPoint does what any identity management system is supposed to do: it manages identities. The very basic functionality of midPoint is the synchronization of identity data that are stored in various applications, databases, directory servers and cloud data stores. We call all these systems identity resources, or resources for short. MidPoint is using connectors to reach the identity resources. MidPoint can propagate a change that happened in one identity resource to other identity resources. E.g. an employee record appears in the HR system, it is picked up by midPoint, processed, transformed and then new Active Directory and CRM accounts are created. This is the process that we call synchronization. It is at the core of everything else that midPoint does.

MidPoint has a rich graphical user interface (GUI) that can be used to manage the identities. Changes made by system administrators are automatically propagated to all affected identity resources. E.g. security officer disables a user by clicking on Disable button in midPoint user interface. Then MidPoint makes sure that all accounts that belong to the user are immediately disabled.

This is the essence of midPoint operation. It sounds simple, because this description is extremely simplified. The reality is much more complicated. Most of the important things happen inside midPoint before the changes are applied to target resources. For each change that midPoint detects it needs to evaluate:

This is a lot of things to process, evaluate and execute. Some of these steps are quite complex. Indeed, there are many sophisticated algorithms implemented in midPoint. There are algorithms that evaluate role structures, organizational structures, temporal constraints, policy rules, password policies and so on. MidPoint does all that for you. The only thing that is needed is to configure them properly.

However, midPoint does even more than that. MidPoint does not only manage identities, it can also manage any object that is anyhow related to identity management. MidPoint can manage roles, role catalogs, organizational structures, groups, projects, teams, services, devices and almost anything else you can imagine.

MidPoint is also an identity governance system. The job of identity management features is to make sure that the policies are consistently applied through the organization. The governance features assist with the maintenance and evolution of those policies. MidPoint implements access certification process. This is a recurring process that asks managers to confirm that the users still need the privileges that they have previously received. MidPoint contains mechanism to sort roles into hierarchies and categories. That is necessary to maintain order during role engineering and maintenance of role definitions. MidPoint has mechanisms for selective enforcement of role which comes very useful during migrations and when new system is connected to midPoint. MidPoint has support for policy lifecycle, general policy rules and so on. Even more work in that direction is planned in future midPoint versions. We fully understand that it is not enough to simply apply the policies. Policies are living things and they need to evolve.

Last, but certainly not least, midPoint provides visibility. Identity data often look like a huge maze, a labyrinth or users, roles, organizational units, entitlements and policies. It is too easy to get lost there. However, there are treasures in that labyrinth. There are patterns and structures hidden in that huge pile of data. Roles and policies buried there, as well as risks and dangerous anomalies lurking behind the corners. MidPoint provides identity analytic capabilities to uncover them. There are powerful reporting and simulation mechanisms. Recent midPoint versions introduced role mining, a capability to find role patterns in the data. MidPoint gives identity professionals capability to see inside identity data.

This book is about practical identity management. Therefore, we will get very close to a practice by demonstrating midPoint features using a case study. This is a case study of a fictional company ExAmPLE, Inc. The name stands for "Exemplary Amplified Placeholder Enterprise". ExAmPLE is a mid-sized financial company. Its operation heavily relies on information technologies, therefore there is a diverse set of applications and information systems ranging from legacy applications to cloud services. As ExAmPLE has few thousand employees and there is a good potential for growth, the management has decided to start an identity and access management program. The first step of the program is deployment of midPoint as the identity governance and administration platform.

Eric is an IT engineer at ExAmPLE, Inc. He has taken the responsibility to install and configure midPoint. Eric spins up a new container with midPoint in it. Couple of minutes later, midPoint instance starts up. Eric logs in to the midPoint user interface.

MidPoint instance is almost empty after fresh installation. It contains only a couple of essential objects. Luckily, Eric is a smart engineer. He has already read through this book, and he knows exactly what he needs to do.

First thing to do is to populate midPoint with employee data. The primary source of ExAmPLE employee data is an HR system. The HR system is quite big and not entirely flexible piece of software. It is not easy to connect to that system directly. Fortunately, it is quite easy to get a text export of the employee data in comma-separated (CSV) format. Eric plans to use this file to get employee data to midPoint.

MidPoint communicates with all the source and target systems by the means of connectors. Connectors are relatively small Java components that are plugged into midPoint. There is usually one connector for each type of the connected system. There are connectors for LDAP servers, Active Directory, databases, cloud applications, UNIX operating systems and so on. The responsibility of a connector is to translate protocols. E.g. LDAP connector translates midPoint search commands to LDAP search requests. Database connector translates the requests to SQL language. Cloud connectors translate midPoint search commands to HTTPS requests for their respective cloud APIs. The UNIX connector creates an SSH session and translate midPoint create command to the invocation of Linux useradd binary. And so on. Each connector talks using its own communication protocol on one side. On the other side, the connectors translate the information to a common format that is understood by midPoint.

There is no distinction between source and target system when it comes to the connector. The same connectors are used for source and target systems. The difference is only in midPoint configuration.

The connectors are distributed as Java binaries (JAR files). To deploy them to midPoint you just need to place them in the correct directory. MidPoint automatically discovers and examines the connectors during start-up. A handful of frequently-used connectors is even bundled right into midPoint distribution. These connectors do not need to be deployed, they are automatically available.

Connector of a specific type works for all the systems that communicate by the protocol supported by connector. E.g. LDAP connector works for all the LDAP-compliant servers. Connector is a very generic piece of code. It does not know the hostname, port or passwords that are needed to establish a connection to a particular server. The configuration that specify connection parameters for individual server is stored in special configuration object called resource. The term resource in midPoint terminology means identity resource, any system which contains identity data and is connected to a midPoint instance.

Therefore, Eric the Engineer needs to define a new resource in midPoint in order to get ExAmPLE employees into midPoint. This resource represents the CSV file exported from the HR system. MidPoint distribution contains CSV file connector already, therefore there is no need to deploy the connector binary package. All that Eric has to do is to create a new resource definition. There are (at least) two ways to do it. Firstly, there is a resource configuration wizard in midPoint user interface. Eric can use the wizard to configure a new resource from scratch. However, Eric is quite an experienced professional, and he is also a little bit impatient. Therefore, he decides to use the other approach: start from an example. There are examples of various resource definitions in the midPoint distribution package, and even more examples are available on-line. Eric quickly locates an XML file that contains a complete example of a CSV resource. He edits the file to change the filesystem path to his CSV file and adjusts the names of the columns to match the format of his file. The very minimal resource configuration specifies just the resource name, connector and connector configuration. The XML file that Eric creates looks approximately like this (simplified for clarity):

Then Eric navigates to Configuration section of midPoint user interface and imports the XML file into midPoint using Import object tool. Import operation creates new resource definition in midPoint. Eric now navigates to Resources  All resources in Administration section of the midPoint user interface. The new CSV resource is there. When Eric clicks on the resource name a resource details screen appears.

Eric can click on Test connection button to test connection to the resource. As this is a local CSV file, there is no real connection. The test checks that the filesystem path is correct, that the file exists and that it can be opened. The connector reads the CSV file header to discover the structure of the data in the CSV file, and presents the information to midPoint. This is stored in midPoint as resource schema, which describes structure of accounts in this resource. The resource is now prepared for use.

There is not much that Eric can do with the resource yet. We need to explain a couple of essential midPoint concepts before moving forward with our case study.

The concept of user is perhaps the most important concept in the entire identity management field. The term user represents physical person: an employee, support engineer, temporary worker, student, teacher, customer, etc. On the other hand, the term account refers to the data structure that allows the user to access applications. This may be an account in the operating system, LDAP entry, row in the database table and so on. Typically, one user has many accounts – usually one account for each resource.

The data that represent users are stored in midPoint, while the data that represent accounts are stored "on the resource side". Which means that accounts are stored in the connected applications, databases, directories and operating systems. Accounts are not stored in midPoint. Under normal circumstances, MidPoint keeps just account identifiers and some meta-data about the accounts. All other account attributes are retrieved when needed. MidPoint is using the connectors to fetch account data.

Accounts are linked to users that own the accounts. MidPoint knows which account belongs to which user. MidPoint can list all the accounts for any user, it can synchronize the data, it can disable all the accounts of a particular user and so on. This user-account link is almost always automatically established and maintained by midPoint.

MidPoint comes with a built-in data model (schema) for users. It contains properties that are frequently used to describe users such as full name, e-mail address and telephone number. There is a reasonable set of properties that should be a good starting point for most deployments. Of course, as most midPoint objects, the user schema can be extended with custom properties if needed.

However, there is no built-in data model for accounts. Such data model would not be possible. Every resource may support different account attributes. Account attributes may have different names, different types and the values may have different meaning. MidPoint is designed to handle those differences.

Account schema may significantly differ from resource to resource. Yet midPoint must be able to synchronize all the accounts from any kind of resource imaginable. In this case the user schema works as a unifying data model. The schema of each account is mapped to the user schema.

Schema for resource accounts is dynamically discovered when midPoint connects to the resource for the first time. MidPoint interprets the schema and automatically adapts to it. E.g. when midPoint displays information about an account, the user interface fields are dynamically generated from the discovered schema. MidPoint does that all by itself. No extra configuration and no coding is necessary. Except for one thing. We need to tell midPoint which type of account is the right one.

Many systems support variety of object types: accounts, groups, roles, organizational units. We call such types object classes in midPoint. Similarly to account attributes, object classes vary from resource to resource. Therefore, we need to specify which object class is the right one to use for accounts. Resource definition has schemaHandling section which is designed for this kind of specifications:

This is how we tell midPoint that resource supports object type (using objectType clause) that is interesting for us. This object type should be labeled as HR Record in midPoint user interface. The connector will use AccountObjectClass object class when dealing with this type of accounts. MidPoint now knows how to access our HR records in a form of accounts in HR System resource.

Getting back to our ExAmPLE story, Eric has an HR resource configured. Now he can see the "accounts" that the users have in the HR system. Eric opens the resource detail page in the midPoint GUI and navigates to Accounts panel. There is not much to see yet. We would like to see all the HR records here. However, reading all the accounts may be quite a demanding task, therefore midPoint does not do it automatically. Eric has to click on Reload button to initiate the process for the first time. The list of accounts appears:

All that can be seen in this list are just employee numbers, because employee number is set as the primary identifier for the HR system. Click on the employee number displays more details about the respective "account". However, these data are not real accounts in this particular case. These are lines in the CSV file exported from the HR database. However, they describe some aspects of user’s identity, therefore midPoint interprets them as accounts. For midPoint, "account" is a generic term used to describe any resource-side data structure that represents the user.

The user is a central concept for any identity management system, and midPoint is no exception. MidPoint needs reliable information about users to work correctly. The HR system is usually a relatively good source of user information. Eric needs to get that information from the HR system into midPoint. He has already set up a resource that connects to the CSV file exported from the HR system. However, the resource does not do anything interesting by default. It has to be configured to pull the information from the HR file into midPoint. What Eric needs is a set of mappings. Mapping is a mechanism for synchronization of attribute values between user and accounts. In this case, Eric needs inbound mappings to import the data. Inbound mappings synchronize the value in the direction from the resource into midPoint. Eric can open the resource definition in the configuration wizard in GUI, and he can add the mappings there. However, Eric likes to know how things work. He looks at the configuration samples again, and he adds the mappings in the XML form. Inbound mapping looks like this:

This is a mapping that maps the account (HR) attribute firstname to user (midPoint) property givenName. This tells midPoint to always update a value of user’s given name when the mapped HR attribute changes. Eric adds similar mappings for all the attributes in the HR export file. Eric also needs to add synchronization section to the resource definition. The synchronization section instructs midPoint to create a new user for each new account that midPoint discovers. This is exactly what Eric wants: create a user for each HR account. Eric then re-imports the modified XML file into midPoint.

MidPoint is now ready to synchronize the attributes. MidPoint is ready, however it does not do anything yet. We still need a task to pull all the data from the HR system. Eric navigates to the page that shows the list of HR accounts. There is a Tasks button which can be used to manage tasks that manage the accounts. Eric clicks on that button and creates a new import task by Create task option in the drop-down menu. He creates an Import task. All he needs to do is fill out task name (e.g. HR Import) in the task wizard, everything else is pre-set to reasonable default values. When Eric clicks on Save & Run button, the task is started, and it runs for a couple of seconds. After the task is done Eric can look at users in midPoint by navigating Users  All users in main menu:

Eric can see details about the user by clicking on the username:

This page shows all the details about the user that midPoint knows about.

The details are sorted into several sections. We are going to explain all of that later in this book. For now, we only care about first two sections. The Basic section shows user properties as midPoint knows them. These properties are stored in midPoint repository. MidPoint has quite a rich data model that can be used out-of-the-box, but the GUI only shows those properties that are actually used. The "name", given name and family name were imported from the HR resource and that’s what the page shows.

Let’s have a look at the second tab now, clicking on Projections in the details menu:

The Projections panel shows user’s accounts. Currently, there is only one account. It is the HR record that was used to import the data. Account details are displayed by clicking on account identifier:

The data that are displayed here are really fresh. Account details were retrieved from the resource at the very moment that the account was displayed. This is the difference between user data and account data: user data are kept in midPoint repository, while account data are retrieved from the resource as needed.

The user and the account are linked. MidPoint remembers that this user originated from this specific HR account. If the HR account is modified, then the change is synchronized and applied to the user data. The mappings are not just for the import. They can work continually and keep the account and user data synchronized all the time.

The concepts of an account is all about the reality: it shows the data that are there at this very moment. It shows what is there. However, identity management is all about policies. Policies, by definition, specify what should be there. Policies specify what is right. However, as every citizen knows all too well, the things that are and the things that should be do not always match perfectly. We are no idealists. Therefore, we have designed midPoint from the day one to acknowledge that there may be a difference between reality and policy. It is a prime responsibility of midPoint to manage that difference - and align policy and reality as much as possible in the long run.

This kind of thinking is easy to see in midPoint user interface. There is Projections panel in the user details page. It shows the accounts that the user has right now. It shows the real state in which the accounts are. It shows the reality. Then there is Assignments panel. This panel shows the policy. It shows what accounts, roles, organizations, or services are assigned to the user. It shows what user should have.

We need a new resource to demonstrate how the assignments work. Therefore, let Eric connect a new resource to midPoint. This time it will be new, clean and empty LDAP server. Eric once again locates the proper example, modifies the configuration and imports it to midPoint. In a while, there is a new LDAP resource. Eric wants to synchronize all the users to the LDAP server. To do that, Eric has to define mappings once again. They will be outbound mappings this time, as Eric wants to propagate data out of midPoint into the (LDAP) resource. We will cover the details of mapping configuration later, let’s just see the results now. We have two resources now:

How do we create an account on that LDAP resource? The right way to do this is to let midPoint know that a user should have an account on that resource. In midPoint terminology we say, that we are assigning the resource to the user. Eric needs to edit a user (navigate to user list Users  All users, select a user), navigate to resource assignments panel by click on the Assignments  Resource in the details menu, and use unassuming New button on the bottom of empty list to add an assignment for the LDAP resource:

After the click on Save button, a lot of things happen. MidPoint recomputes what the user should have and what the user has. MidPoint detects that the user should have an LDAP account now (because there is a new assignment for it). However, no such account exists yet. Therefore, midPoint creates the account.

When Eric opens the user details again, and navigates to the Projections panel he can see that there are two accounts now:

There is an HR account that was used to create the user in the first place. Then there is an LDAP account that was created as a reaction to a new assignment.

There is reality and there is policy. There are accounts and there are assignments. Ideally these two things should match perfectly. MidPoint will try really hard to make them match. However, there may be exceptions. Careful reader surely noticed that there is HR account, but there is no assignment for that account. Despite that, midPoint has not deleted the HR account. That is because the HR system is what we call a "pure source" system. MidPoint does not write to the HR, it only reads from it. Writes to the CSV export file would be overwritten by the next export anyway, therefore there is no point in writing there. For that reason, the HR resource has an exception specified in its configuration: it allows the HR account to exist even if there is no assignment for it. We can keep the HR account linked to the user by using this method. We can see the data that were used to create the user. This improves overall visibility, and it is a great help for diagnostics of configuration issues.

It would be a daunting task if Eric had to assign every individual account for every individual resource to every user. Typical identity management deployment has thousands of users and dozens of resources. Such deployment would be very difficult to manage using direct resource assignments only.

There is a better way, of course. We can use roles. The concept of role-based access control (RBAC) is a well-established practice. Indeed, the roles are the bread-and-butter of identity management. The basic idea of RBAC is to group privileges into roles. Then the roles are assigned to the users instead of privileges. Let’s create a Webmaster role. Then put all the privileges that a webmaster should have into that role. Let’s assign the role to every user that works as a webmaster. All of them will get the same privileges. This simplifies the privilege management. If there are two webmasters, there is no need to think about the individual privileges that a webmaster should have. Just assign the role, the role has everything that is needed for a webmaster to do his job. It is also easy to change webmasters: unassign role from one user, assign it to another user. It is also easy to adjust privileges if you add a new web server. Just add the privilege for accessing new server into the Webmaster role. All webmasters will get that privilege.

That’s the theory. How does it work for Eric? First of all, let’s add a handful of new resources – to get some material for the roles to work on. Now we have four resources: HR, LDAP, CRM and Portal. That’s a good start. Let’s do some role engineering now.

Many organizations have one role that almost every user has. It is often Employee or Staff role. This role gives access to all the systems that an employee should have access to: Windows domain login, e-mail, employee portal – things like that. The ExAmPLE company is no exception. In this case the basic role should create accounts in two systems:

It is simple to create such role in midPoint user interface. Eric navigates to Roles  New role. Role wizard starts, prompts for a role type. We cannot create neither application nor business role, as we do not have basic structure for that yet. Therefore, Eric selects the last option. He fills in the name of the new role (Employee). Then he navigates to the Inducements  All panel. This is where the role definition takes place. Inducements are almost the same as assignments. However, inducements do not give access to the role itself. Inducements give access to the users that have this role. So they are kind of indirect assignments. Eric clicks on New button, and adds inducements for LDAP resource into the role, by selecting the Resource tab and selecting LDAP resource. He repeats the process for Portal resource as well.

Eric clicks the Save button, and the new role is created. Now it is ready to be assigned to the users. Eric goes on and assigns Employee role to user Bob by editing the user and adding a role assignment:

MidPoint automatically creates all the accounts given by the role after the modification is saved:

There is the HR account that was used to create the Bob user record in the first place. Then there are the two accounts that were created because Bob has the Employee role: portal account and LDAP account.

This operation works in both directions: if Eric unassigns the Employee role, the accounts given by the role will be deleted. Eric can create any number of roles like this: roles for sales agents with CRM access, roles for sales managers with higher CRM privileges, roles for assistants and clerks, roles for everybody. MidPoint is designed to handle large number of roles. Each role can have its own combination of resources and entitlements. MidPoint seamlessly merges the privileges given by all the roles a user has. E.g. if two roles give CRM access to the user, only one CRM account will be created. If one of these roles is unassigned, then CRM account remains there. The account is not deleted yet because it is given by the other role. Only when the last CRM role is removed, that’s the point where the account gets deleted. MidPoint takes care of all that logic.

Of course, there is much more midPoint roles can do:

Roles are really the essence of identity management. We will be dealing with assignments, roles and role-like objects in almost all the parts of this book.

Eric the Engineer has done a few basic steps to configure midPoint as an identity management system for his company. However, this is still a very basic configuration. It is only a beginning. Careful readers have already noticed a lot of things that need to be done. E.g. employee full name is not automatically generated. Employee numbers are used as identifiers. We would like to have something that is more user-friendly instead. We need to automatically assign the Employee role instead of doing that manually. And so on. There are still a lot of things to improve. Fortunately, all of that is very easy to do with midPoint, once you know where to look. We will be dealing with all these things in the rest of this book. New functionality will be administered to the ExAmPLE solution in small doses in each chapter – together with a proper explanation of midPoint principles. MidPoint is a very flexible and comprehensive platform, and there are still a lot of things to learn.

Now you probably have some idea what midPoint is. However, it is also very important to understand what midPoint is not. Identity and Access Management (IAM) field is a combination of many technologies, and it may be quite confusing sometimes. That is perhaps the reason why the midPoint team occasionally gets questions about midPoint functionality that simply do not make much sense.

First of all, midPoint is not an authentication server. MidPoint is not designed to validate your username and password. Yes, midPoint maintains data about users (including passwords). However, the data model that midPoint maintains is quite complex. It is not meant to be exposed to applications directly. That would not be very efficient.

If you want midPoint to manage users, but you also want your applications to have a centralized authentication service there is a solution: publish the data to the LDAP server. Connect LDAP server to midPoint as a resource, and let midPoint populate and maintain the LDAP sever data. The application will not talk to midPoint directly. They will talk to the LDAP server. This is better for everybody: LDAP is a standard protocol, well-supported in many applications. LDAP servers are also extremely fast and scalable. Therefore, use the combination of midPoint and an LDAP server of your choice. That’s what people usually do and it works perfectly.

Of course, we said that LDAP is not authentication server either. That’s right, it is not. However, LDAP can still work for simple authentication scenarios. If you need more than that, connect a proper authentication server to LDAP. That is what most authentication servers expect anyway.

As midPoint is not an authentication server it obviously is not a single sign-on (SSO) server either. If you want SSO, you will need a dedicated SSO server. There are plenty of SSO servers to choose from in both the closed-source and open-source worlds. You will also need a scalable directory system (LDAP) to store the data for the SSO server. MidPoint will be pleased to manage that LDAP server for you.

One of the things that seems to be shrouded in a lot of confusion is authorization. To get the record straight from the beginning: midPoint is not an authorization server. It is not a policy decision point (PDP), and it definitely is not a policy enforcement point (PEP). You cannot rip authorization out of your application and just "use midPoint for that". That does not work.

You can think of midPoint as a policy administration point (PAP). MidPoint has a lot of sophisticated authorization-related logic inside its core. However, that logic is not designed to answer questions such as "Is subject S authorized to execute operation O on object X?". MidPoint logic is different. MidPoint is not concerned with making authorization decisions. It is concerned about managing the authorization policies. MidPoint sets up the authorization policies in target applications. The applications evaluate these policies themselves. This is a much more efficient and more reliable method. Unlike authentication, the authorization decisions are done all the time. Authorization is evaluated at least once per every request. If the application makes these decision internally, then there is no need to a round-trip to the authorization server. Performance is significantly increased. Also, there is no single point of failure. MidPoint failure will not interrupt authorization flow, because the application has all the data inside. One less component to cause a failure. Yet, the policies are centrally managed by midPoint. When a policy changes, midPoint updates all the affected applications. You get all the benefits without the usual drawbacks.

MidPoint does what it is supposed to do: it manages identities, entitlements, organizational structures and policies. MidPoint does not do things that are not necessary. It does not do the things that other technologies already do well. MidPoint does not reinvent the wheel. There is no need for this. MidPoint is not the wheel. MidPoint sits above all the wheels. MidPoint is the chauffeur.

MidPoint will run on almost any machine. Mid-range laptop is fine for personal use of midPoint. MidPoint needs approximately 4GB of RAM, which is perhaps the only real limiting factor.

Theoretically, MidPoint is platform-independent, it can run on any environment where Java is running. However, devil is in the detail. Individual environments and operating systems have their peculiarities. Therefore, only some environments are supported by Evolveum.

Linux is an operating system of choice for production use of midPoint. However, midPoint will be happy in Windows or macOS environment on your laptop for non-production use. Some midPoint releases have production-grade support for Windows environment as well.

If you look for more formal system requirements definition, then you will find that in midPoint docs (see Additional Information chapter at the end of the book).

There are several ways to install and use midPoint.

It is strongly recommended to start with the Docker method. The descriptions and examples in this book assume that Docker Compose installation is used.

Docker provides a popular method to distribute and run containerized software. Evolveum is building and publishing docker images for every midPoint version. In addition to that, Docker Compose definition file is provided for easier start. Docker Compose file describes how the necessary containers are assembled, creating a simple environment with all the necessary components.

MidPoint installation described in this chapter is a very basic one. It is ideal for initial exploration of midPoint, development of midPoint configurations, simple testing, demonstrations, exploration and similar purposes. It is a very convenient installation, and it provides all you need to get started.

Following steps are applicable to Linux and macOS environment. Windows users may use Docker Desktop running on Windows Subsystem for Linux (WSL). Please refer to Evolveum documentation regarding Docker Compose for more details.

You will need:

Once the Docker is installed and the script is downloaded, there is still some preparation to do:

Once the environment is started, midPoint user interface can be accessed as a localhost service at port 8080.

+ http://localhost:8080/

+ You can start working with midPoint now. Log in as administrator. The script has generated a random initial password (remember: secure by default), which can be used to log in.

In case that midPoint does not start, the problems are indicated in the logfile. The logfile (midpoint.log) is located in midPoint home directory. In our little Docker Compose environment, the logfile can be found at the following path:

midpoint-home/log/midpoint.log

If there is any problem, it is likely that detailed problem description can be found at the end of the log file. When desperate, please refer to the Troubleshooting chapter.

When not needed, the containers can be shut down:

./midpoint-quickstart.sh down

When shut down, midPoint keeps all the data and state in the repository. The data will be available when the container is started again.

In case that there is a need to purge all data and re-start on a clean playing field, persistent database data can be deleted by using the clean command passed to the script:

./midpoint-quickstart.sh clean

Standard Docker Compose definition for midPoint contains just the basic components: midPoint server and PostgreSQL database. However, for the purposes of this book, we are going to need a little bit more - to have some fun with midPoint. We are going to need an LDAP server, CSV files and some additional database tables to play the role of resources. Of course, the stock docker compose definition can be extended to include all that. However, the primary goal of this book is to learn midPoint. Messing around with docker may be a distraction. Therefore, there are already containers configured to run the environment suitable for trying examples provided in this book. The containers are located in the docker directory in book samples (see Additional Information chapter for details). There are two configurations, first one corresponds roughly to the beginning of chapter 5 (Synchronization), the other corresponds to end of chapter 10 (organizational structure). There are README files in both directories, explaining the details.

The containers have convenient book-example.sh script, which can be used to set up, start and re-set container state. This script is meant for Linux, as primary author of this book is a hopeless Linux enthusiast. There are no Windows or macOS versions. However, this is open source world, contributions are more than welcome.

If you want to have just a quick look at midPoint, there is on-line midPoint demo located at the following URL:

https://demo.evolveum.com/

The demo has some basic configuration to demonstrate midPoint features. There are several resources configured for synchronization, role-based access control (RBAC) is set up, there is organizational structure, some policies, and many other things. Link to the demo documentation is provided at the login page.

When local midPoint instance is started, use the following URL to access midPoint user interface:

http://localhost:8080/midpoint

Log in with the following credentials:

Username: administrator
Password: randomly generated at first start, displayed by the start script

Now you are logged-in as the administrator. This user has superuser privileges, therefore you can see everything, and you can do anything in the midPoint user interface.

MidPoint user interface is structured. It has the same layout and controls for all the user interface areas:

Primary tool for user interface interaction is the menu. MidPoint user interface is functionally divided into three parts, therefore there are also three parts of the menu:

The menu can be hidden by clicking on the button at the top of the screen. The top bar also contains the title of the current user interface page and breadcrumbs. Breadcrumbs show where you currently are in the user interface and how you got there. The breadcrumbs can be used to "find your way home" and back to the previous page. The use of browser Back button is not recommended. Please use the Back buttons that are present in midPoint user interface, or use breadcrumbs.

MidPoint user interface is quite rich. Following list provides short description of the most important parts of the user interface.

MidPoint user interface is using the same concepts and controls in all its parts. For example all the lists of all the objects (users, roles, …​) look like this:

Each row represents one object: user, roles, service, task, etc. There is also a color-coded object icon. The search bar at the top can be used to look for a specific object or to filter the object view. Right side is reserved for action buttons. Buttons in the table header trigger actions that apply to all selected objects. Buttons in each row trigger actions that apply only to that individual object. The buttons in the bottom-left corner execute global actions, such as creating or importing new object, exporting objects and refreshing the view. Paging controls are in the bottom-right corner.

MidPoint has a unified color-code that makes the navigation easier. Users, roles and other object types have their specific color and icon. This indicates the object type, and it is used whenever possible: menu, information boxes, lists, box title accents and so on. The primary colors and icons are shown in the dashboard:

All user-related controls are red, all controls that deal with organizational structure are yellow. Roles are green. And so on. This color code is applied consistently through the midPoint user interface.

Similar color code applies to object icons when displayed in user lists. However, the color that is used there does not indicate object type, but rather an archetype. Archetypes are subtypes that are often used to distinguish similar objects. For example, archetypes can be used to sort users to employees, contractors, customers and so on. Look and behavior of "archetyped" objects is configurable. Default midPoint configuration contains just a couple of archetypes. Those archetypes apply red color to system users and roles.

When a user clicks on a name of any object in the object list then object details page appears. The detail pages for common midPoint objects, such as user or role, are very similar to each other. They have the same layout and controls. E.g. user details page looks like this:

The screen below the information area is divided into several panels. There is a summary panel at the top of the page. This is an information area which shows user photo (or icon) and provides some basic information such as user identifier and title. It also shows where the object belongs in the organizational structure. There is also a couple of "tags" that show interesting details about the object: whether the object is enabled, whether it has special privileges and so on.

Below the summary panel, there is a panel with operation controls. There are buttons to save the changes, go back to the previous page, controls to set operation options and so on.

The data are displayed in details panel. However, midPoint data structures are often quite complex. Displaying the data in their full complexity would make it very difficult to understand. Therefore, the data are divided into several parts, each containing a portion of the data. Navigation menu is displayed left to the main details panel. It allows easy navigation through data structures, each displaying a specialized panel with object details.

The first item is perhaps the most interesting one. It contains fields that show basic object properties: the attributes of the object. Properties are displayed, and they can be edited – depending on the authorizations of currently logged-in user. The other panels shows more complex information about the object. E.g. activation panel shows whether the object is enabled or disabled, it shows the activation dates and other activation details. The password panel provide password management functionality. And so on.

The view control buttons present in the details panel can be used to adjust the way the information is presented. These buttons control sorting of the fields, can toggle metadata display and there may be additional control buttons for more complex fields. MidPoint shows only some fields by default, to make the presentation easier to read. For example, most empty fields are not shown by default. There is a small Show empty fields button under the fields than can be used to display empty fields.

The content of the details panel changes its form, adapting to the type of displayed information. Basic properties are shown as a series of editable fields. Content of some panels is similar to the Basic panel, displaying a set of fields. Other panels provide lists of more complex data structures such as projections, assignments and personas.

MidPoint user interface often needs to present objects that are internally quite complex. It does not make sense to present all the details at once. These objects need to be presented in quite a compact form that can be expanded to show the details. This applies to list of user’s accounts, assignments, role inducements, etc. The objects are initially displayed as in a form of a simple list, displaying only the basic data:

The list above shows user’s projections. Those are usually accounts that are linked to user object. Click on account name shows account details:

Account details display is shown in place of user details. This may be slightly confusing. However, account details can be often complex, therefore all the available screen space is needed to display them. The Cancel and Done buttons at the bottom can be used to return to list of projections. Click on Done button will not start the operation yet, it only changes the view. Therefore, do not confuse those buttons with Back and Save buttons located in the control panel.

Each panel provides the information in an editable way, provided that the user has adequate privileges to edit the information. When the data are edited, the changes can be confirmed by using the Save button in the operations panel. Saving the changes is a universal way how to start almost any operation: change of user properties, assignment of roles, change of password, user disable, etc. When you make edits in any of the panels on the details page, then nothing really happens yet. MidPoint just remembers what you are editing. The operation is executed only when you click the Save button. This is our method to execute several changes in one operation. It may require some time to get used to it. Just do not forget to click the Save button when you are done editing.

Operation options are used to modify the behavior of the operation. These options may force execution of operations that fail to pass midPoint internal checks. There is an option to reconcile the data even if midPoint thinks that reconciliation is not needed. And so on. Checking or unchecking these options influences the way how midPoint executes the operation.

The principle of midPoint configuration is quite different from what would a typical system administrator expect. There are almost no configuration files in midPoint. MidPoint is storing the vast majority of its configuration in its configuration database. There are several reasons for this:

Therefore, midPoint has a completely different approach to configuration. The configuration is stored in a form of well-defined structured objects in the midPoint database. We call that database midPoint repository.

Everything is an object in midPoint. Every piece of configuration is represented as a structured object and stored in midPoint repository. Object may look like this:

Every object has its identifier. We call that identifier OID, which stands for "object identifier" (it has nothing to do with LDAP or ASN.1 OIDs). OID is usually randomly-generated universally unique identifier (UUID). OID value has to be unique in a whole system. This identifier is persistent – it is assigned to the object, and it is never changed. OID is used for internal purposes, and it is almost never displayed to midPoint user.

Every object has a name. Name is human-readable, and it can change any time. The value of name is usually displayed to users. This is the values that ordinary users understand as an identifier.

Then there are other object properties - or rather items. Each type of midPoint object has a slightly different set of these items. That’s what we call schema. The items may be simple properties such as string, integer or boolean values. There may be complex structures, and references between objects. MidPoint data model is quite rich. It is in fact so rich that its description will take better part of this book, because description of the data model is also description of midPoint features.

You can see midPoint configuration objects in midPoint user interface by navigating to Configuration  Repository Objects  All objects and selecting object type. The following picture shows objects of type "Role":

The objects are stored in the midPoint repository in a native form which is hidden from midPoint users. However, the objects also have a human-readable representation. They can be represented in XML, JSON and YAML forms. All the objects can be imported into midPoint in any of those forms. They can be exported from midPoint in any of those forms. They can be edited directly in midPoint using embedded editor, in any of those forms. Just click on any object in the Repository objects page:

The ability to export, import and edit objects in XML/JSON/YAML form is absolutely essential, because:

Therefore, the midPoint configuration has the best of both worlds. It is stored in repository, therefore it can be processed efficiently, it can be made available and so on. Yet, it also has a text form, therefore it can be easily managed.

The XML, JSON and YAML forms are considered to be equivalent. Objects can be written in any of these forms.

Most of the examples in this book are in XML notation. The XML form is almost always simplified for clarity: there are no namespace definitions, no namespace prefixes and so on. The complete files with all the details can be found in midPoint distribution package, midPoint source code or in other places. See Additional Information chapter for more details.

When it comes to maintenance of midPoint configuration, there are two practical methods.

First method is to maintain the configuration in midPoint: use midPoint wizards and user interface tools to create new objects and modify them. Export the objects in regular intervals to backed them up. This is an easy method to start with.

However, sooner or later you will probably figure out that you need the ability to copy and paste parts of the configuration, compare differences and review configuration history. You will need to share parts of the configuration with other team members. As we all know, no user interface is ever as efficient as an experienced engineer with a good text editor.

Then there is a second method: maintain the configuration in text form: XML, JSON or YAML. Import the configuration to midPoint as needed. The objects can be imported in midPoint user interface by going to Configuration  Import object page.

It is much easier to maintain a proper version control and a good teamwork using this method. It also seems to be more efficient once you get used to midPoint: pieces of configuration can be copied from samples, documentation or from other projects. This makes the work efficient. Although work with midPoint is "just" configuration, and there is usually almost no programming, this method of work is quite close to the methods that software developers use. We know that these methods work quite efficiently, as we are using them every day.

However, maintaining configuration in files and importing them individually using midPoint user interface may look a bit uncomfortable. However, there is a better way. There is MidPoint Studio, an integrated development environment (IDE) based on IntelliJ IDEA. MidPoint Studio allows you to maintain the configuration files in a form of a development project. You can edit the files with all the usual luxury of IDE, such as syntax highlight and autocompletion. Studio allows easy download and upload of changed configuration files to midPoint instance. As the IntelliJ platform has good integration for version control systems and other development tools, this seems like an ideal approach, especially for large and complex projects. If you plan to use midPoint professionally, MidPoint Studio is certainly worth trying.

MidPoint needs its own database to work. We call that database midPoint repository. The database is used to store the configuration, users, resource definitions, account links, audit trails and a lot of other things. Proper relational database is needed for midPoint repository, such as PostgreSQL database. Docker Compose deployment makes sure that PostgreSQL is available, properly set up, initialized with database schema and ready to be used as midPoint repository.

MidPoint 4.8.5 supports several database engines. However, only PostgreSQL database fully supports all midPoint features. Support for database engines other than PostgreSQL is currently deprecated. The future of generic repository support (as we call it) is uncertain. Choose PostgreSQL directory (a.k.a. _native repository) to get reliable support for all midPoint features.

While almost all of midPoint configuration is stored in the database, there are few things that cannot be stored there - such as connection parameters to the database itself. For that purpose midPoint has a small configuration file called config.xml. MidPoint also needs a place where to store other data that cannot be in the database, such as cryptographic keys, connector binaries and so on.

MidPoint needs a special directory on a filesystem for that purpose. We call it midPoint home directory. When a new midPoint instance starts for the first time, new midPoint home directory, populated with default configuration. Location of midPoint home directory slightly varies, it depends on your deployment method:

MidPoint home directory contains several interesting items:

When midPoint starts for the first time, it starts with an empty database. MidPoint populates the database with a minimal set of configuration objects. We call them initial objects. This set contains objects such as the Superuser role and user administrator. These objects get imported automatically, because if they are not there, you will not be able to log into the new midPoint instance. These objects are imported only if they are not already present in the database. If you modify them later, midPoint will not overwrite them.

Logging is perhaps the most important mechanism to diagnose any issues with midPoint. Logging should be the thing that comes to your mind anytime you cast a puzzled look at midPoint user interface. We try to make midPoint user interface convenient to use, and we pay a lot of attention to good error reporting. However, there are always some practical limits what the user interface can do. The error that the user interface displays may be just a result of a long chain of causes and effects. Error messages in the user interface may not directly point to the primary cause. Perhaps there is no error at all, just midPoint does not do what it is supposed to do. That is the point where logging comes to the rescue.

MidPoint is using Java logging facilities to log its messages. MidPoint log file name is midpoint.log, and it is stored in the log subdirectory in midPoint home directory (midpoint-home/log/midpoint.log for docker-based deployment, /opt/midpoint/var/log/midpoint.log for package-based deployment). The default logging level is set up more-or-less to suit normal midPoint operation. This means that the messages on level INFO and above are logged while the finer levels are not logged. If you want to diagnose midPoint issues you will need to switch the logging levels to DEBUG, or in extreme cases even to TRACE. The logging levels can be adjusted in midPoint user interface. Navigate to Configuration  System  Logging.

MidPoint is not a simple system. There are complex interactions, there is usually a lot of custom configuration, customizations, expressions and so on. Diagnostics of midPoint issues is no easy task. Therefore, there is a dedicated Troubleshooting chapter in this book. That chapter is your best friend when things do not go as expected.

We have small and simple docker-based midPoint installation now. It is not much, yet it is more than enough to explore almost all the midPoint concepts and mechanisms. Therefore, let us go ahead and have some fun.

Identity resource is one of the most important concepts in midPoint. Any system connected to midPoint is an identity resource. Identity resources (or just resources for short) are typically target systems where midPoint manages accounts. Moreover, source systems, such as HR databases, are also considered to be identity resources. There is no strict distinction between the source and target resource in midPoint. Both source and target resources are defined in exactly the same way. Resource can even act as both source and target at the same time.

MidPoint needs a way to communicate with the identity resource. MidPoint has to know communication protocol, hostname, keys and passwords, and all other communication parameters. For that purpose midPoint has resource definition objects. These are midPoint configuration objects stored in midPoint repository. Resource definition usually contains:

Resource definition looks like this in its XML form (simplified):

Resource definition is a very rich (and powerful) configuration object. It is maybe the richest configuration object in the entire midPoint platform. Creating resource definition from scratch is usually no easy task. There is a lot of things to consider: connector configuration, identifier conventions, mandatory attributes and attribute value formats to name just few. There are two practical ways to create resource definitions:

However, you need to understand how the resource definitions work to do this efficiently - even if you start with resource wizard. Next few sections will explain the structure and function of resource definitions.

Every resource needs a connector to work. Connectors are small pieces of Java code that are used to communicate with the source and target systems. Few popular connectors are part of midPoint distribution package (bundled connectors), other connectors can be downloaded and deployed in midPoint home directory.

MidPoint automatically looks for available connectors. MidPoint creates new configuration object for each connector that it discovers. The list of discovered connectors can be seen in midPoint user interface in Configuration  Repository objects  All objects, selecting Connector in the type field located at the top of the screen. The connector objects look like this:

The resource definition needs to point to the appropriate connector object. Therefore, select the right connector from the connector list and remember its OID. Then use the connector OID in the resource configuration like this:

This is a straightforward way how to link connector and resource. However, it is not the most convenient one. MidPoint creates connector objects automatically. Therefore, the OIDs of the connector objects are not fixed. Every midPoint instance will have different OID for the discovered connectors. Therefore, if we want a resource that is always using the LDAP connector in all the midPoint instances, we cannot do that by just using OIDs. There is another way. You can use search filter instead of fixed OID:

The detailed explanation of the search filters will come later. For now, it is important to know just few basic principles. When this resource definition is imported, midPoint notices that there is no OID in the connectorRef reference. It also notices that there is a search filter. Therefore, midPoint executes that search filter. In this case it looks for an object of ConnectorType type that has property connectorType with value com.evolveum.polygon.connector.ldap.LdapConnector. Therefore, midPoint finds LDAP connector, regardless of the OID that was generated when midPoint discovered that connector. Then midPoint takes the OID of the object that it has found. The OID is placed to the connectorRef reference, so midPoint can find the connector directly, and it does not need to execute the search every time the resource is used.

This is the method that is frequently used to bind resource definition to a specific connector type. It has the advantage that it works in all midPoint deployments. Therefore, it is also used in the configuration samples.

Each type of resources needs its own connector. There is an LDAP connector that supports all the common LDAP servers. There are connectors that work with generic database tables. These connectors are quite generic. However, most connectors are built for a specific application or software system: Linux servers, SAP, Zoom, etc.

There is a handful of connectors that are so generic that they are used in almost all midPoint deployments. These connectors are bundled with midPoint. It means that they are part of the midPoint application package, and they are always available. These three connector bundles are part of midPoint:

These connectors are always available in midPoint. Other connectors must be deployed into midPoint. Connector deployment is a very straightforward process:

MidPoint periodically scans the connid-connectors directory. It discovers any new connectors, and creates a connector configuration objects for them.

Connector needs a configuration to be able to work with the resource. This configuration usually consists of connection parameters such as hostname, port, administrative username, password, connection security settings and so on. The connector configuration properties are specified in the resource definition object. In a simplified from it looks like this:

There may be a very broad range of configuration properties - and every connector has its own set. While working just with the XML/JSON/YAML representation of the resource definition, you will need to find out the names of the configuration properties by looking at the samples, connector documentation or maybe even connector source code. It may look difficult, but this is a perfectly viable approach. However, there are other ways. Firstly, there is the resource wizard. The wizard knows all the connector configuration properties, and it will present the properties in a configuration form. The wizard takes the definition of the configuration properties from the connector schema. The connector schema is a definition of the properties that the connector supports: their names, types, multiplicity and so on. The connector schema is stored in the connector configuration object in the schema element. Therefore, even if you are working only with the XML/JSON/YAML files, you can have a look at that schema to figure out what connector configuration properties are supported.

The connector schema also defines the connector namespace. Generally speaking, namespaces in midPoint are used to isolate schema extensions that might have conflicting element names. The use of namespaces is optional in almost all parts of midPoint - but not in all the parts yet. Connector configuration is one of the few parts where namespaces must still be used. It also makes some sense, as namespaces are used here as an additional safety mechanism. To keep a long story short, the configuration properties should be properly namespace-qualified:

The use of namespaces will be completely optional in later midPoint versions. For now, just copy the namespace URIs from the samples. You do not have to completely understand what is going on. Just one thing: the namespace of the configuration properties should be the same as the namespace defined in the connector object. This is a long URI that is composed of connector bundle name and connector name.

For example: http://midpoint.evolveum.com/xml/ns/public/connector/icf-1/bundle/com.evolveum.polygon.connector-ldap/com.evolveum.polygon.connector.ldap.LdapConnector

If the namespace does not match, then the connector will refuse to work. This is a safety mechanism that prohibits accidental use of configuration from one connector in another connector, where the configuration properties may have the same name but a completely different meaning.

Minimal resource definition has just the name, connector reference and connector configuration properties. After that, the resource should show the first signs of life. Therefore, go ahead and select a suitable sample file now. Strip it down to the minimum, modify connector configuration properties and import the resource into midPoint. You should be able to see your resource in the list in Administration  Resources  All resources. The icon next to your resource is most likely black - not green and not red. Green icon means that the resource is working, red icon means that there is an error, black means "I do not know yet". Click on the resource label. The resource details page appears. There is a Test Connection button at the top of the page. Click on that button. It may take a while now. MidPoint is initializing the connector with the configuration properties that you have specified. Then the connector is used to check connection to the resource. If the parameters were correct, and midPoint can reach the resource, you will see the green lights:

If there are any errors during connector initialization, configuration or network connection you will see the errors here. In that case, correct the configuration properties, and try again. If everything works well, then the resource icon turns green. Now we have a very minimal working resource.

The test connection procedure is testing whether connector can connect to the resource. However, can the connector access the data as well? We would like to see some data now, to make sure that everything works fine. We cannot do that just yet. We have to talk about the schema first.

The only thing that early identity management systems dealt with was an account. The world have evolved a lot since the early days of identity management in the 2000s. Today, identity management systems need to manage many different types of resource objects: accounts, groups, organizational units, privileges, roles, access control lists and so on. In midPoint, these are the object classes: types of resource objects that are made accessible to midPoint by the connector. A minimal resource supports at least one account object class, but a typical resource supports more object classes. Each object class may have a completely different set of attributes: different attribute names, different data types, some may be mandatory, some optional, single-valued or multi-valued.

The collective definition of the object classes and their attributes is what we call resource schema. Obviously, resource schema is different for every resource. Even resources that are using the same connector may have different resource schema. For example two LDAP servers with different custom schema extensions or two business systems with different customizations. MidPoint is a smart system, and it is capable of automatic resource schema discovery. MidPoint reaches out to the resource and retrieves the schema when the resource is used for the first time. Retrieved resource schema is stored as the schema element in resource definition object. You can have a look and examine the schema there. But beware, the schema may be quite rich and big.

Resource schema is an absolutely crucial concept. MidPoint takes advantage of resource schema whenever it needs to work with resource objects such as accounts or groups. MidPoint uses resource schema to validate mappings. The schema is used for automatic type conversions. Most importantly of all: resource schema is used to display resource objects in user interface. MidPoint adapts to resource schema automatically. Not a single line of custom code is needed to do that.

We would like to have a look at resource data before going on with configuration. It would be nice to make sure that the connector is configured correctly, that there are appropriate access rights in place for the connector to access the data and that everything works fine. However, some resources are very flexible and generic. LDAP servers are a prime example. They have lots of object classes to choose from, their resource schema is quite bit. We need only a couple of object classes from that huge LDAP schema. However, LDAP connector is very generic, it does not know which object classes are the right ones. Therefore, we have to tell midPoint which object class to choose from the schema.

We will learn about the schemaHandling section later. For now, this declaration of object type tells midPoint, that there are accounts on this resource. The accounts are using inetOrgPerson object class.

Now, as midPoint knows what "Normal LDAP Account" means exactly, we can have a look at such accounts. Navigate to the Accounts panel in resource details page. This is the place to browse accounts on this resource. However, the list is empty! It is empty, as midPoint have not tried to access the accounts yet. Click on Reload button does the trick. MidPoint is using the connector to list all the accounts (objects with inetOrgPerson object class) in our LDAP server.

Now you can click on any object to see the details. This is a very useful feature for several reasons. By looking at several objects, you can get a basic overview of how the data are structured: what attributes are used and what are the typical values. You will appreciate that information later on when we will be setting up mappings.

MidPoint topology is a star (a.k.a. "hub and spoke") with midPoint at the center. This is both physical and logical topology of midPoint deployments.

This means that the account A can be synchronized with midPoint user and then midPoint user can be synchronized with account B. However, account A cannot be synchronized directly to account B. This is a deliberate decision that was made very early in midPoint design. We have very good reasons for it.

Accounts and user that represent the same person are linked together. This link is a relation that midPoint creates and maintains. Therefore, midPoint knows who is the owner of a particular account. MidPoint also knows which accounts the user has. That is how midPoint knows which account needs to be synchronized with which user. It is critical for the links to be correct, otherwise midPoint cannot reliably synchronize the data. For that reason, midPoint takes great care to maintain the links. That is not always an easy task. There are strange corner cases, such as renamed accounts, or accounts that were deleted by mistake and re-created. Yet, midPoint is built to handle such cases. The links are always maintained. It is the link that allows midPoint to list all user’s accounts in the user interface.

The user in midPoint is known as focus in midPoint terminology. The accounts are known as projections. You can imagine a light projector that sends many light beams from its focal point to create a projection on the screen. This is the metaphor that we have chosen when developing midPoint. For the lack of better words, this terminology remains in use even today. We will get back to the concept of focus and projections many times in this book. For now, you just need to remember that projection means an account.

MidPoint knows which account belongs to which user by following links that it maintains. However, how does midPoint know which attributes to synchronize? How to transform the values? Which side is the authoritative one? Mappings take care of that. Mapping is like a flexible data replication recipe. MidPoint allows to define mappings for each attribute in any direction. The mappings are used to control the synchronization on a very fine granularity.

Perhaps the best way to summarize synchronization principles is to illustrate them using a couple of examples. The first example is a modification of user properties in midPoint user interface. When the Save button is pressed, midPoint user interface sends the modification to midPoint core engine. The synchronization code in midPoint core follows the links to find all the accounts that belong to this specific user. Then the mappings are applied to synchronize the changed user properties to the accounts. Account changes are propagated to the resources, and user changes are stored in midPoint repository.

The second example is slightly different. This case starts with a change of account data. This may be a change of an employee record in HR system. MidPoint detects that change, and reads the changed account. MidPoint follows the link to find the user to which the account belongs. Then it follows other links from the user to find all the other accounts that may be affected. Similarly to the previous case, the mappings are applied. The mappings from the HR account to the user are applied first. The result is a modification of user properties. Then a process identical to the previous case takes place. User modifications are automatically applied to all affected accounts.

Those two cases might look to be quite different. First case is a manual change of data by system administrator. Second case is an automatic data feed from the HR system. However, as you can see, the principles that are used to implement those two cases are almost exactly the same. This is the consequence of midPoint philosophy: radical reuse of functionality and generic application of principles. You define what you want to do (the policy) by setting up the mappings. MidPoint takes care that it is done when it needs to be done.

Resource schema is a very important concept. It defines what object classes are supported by the resource and how they look like. Yet, it is important to know not only how the objects look like. It is also important to know what to do with them. That is what the schema handling is all about.

Schema handling is a part of the resource definition object. It specifies which object classes from the resource schema are actually used by midPoint. Most importantly of all, it specifies how they are used. This is the place where mappings are stored. This is the place where account-group associations are defined. This is the place where schema can be augmented and tweaked. Consequently, this is the place where most of the resource-related configuration takes place.

Schema handling section contains definition of several object types. Each object type refers to one "thing" that midPoint works with: default account, testing account, group, organizational unit and so on. Let’s start with something simple. Let’s define just one object type now: default account. It looks like this:

This may seem trivial, but even such a minimal definition is important for midPoint. This definition tells midPoint that default account on this resource has inetOrgPerson object class. Resources such as LDAP servers may have dozens of object classes. Most of them are not used at all. There are often several alternative object classes that can be used to create accounts. It is important to tell midPoint which object class is the right one. That’s what this definition does. Once this definition is in place, the accounts appear on the Accounts panel of the resource details page. This is a sign that the definition works correctly.

A clever reader surely noticed definition of kind in the above example. Setting kind to account indicates that this object type definition represents (quite surprisingly) an account. MidPoint supports many types of objects. However, two types have a special place: accounts that represents the users and entitlements that give privileges to the accounts. MidPoint can handle the objects in a smart way if it knows that it is either account or entitlement. The kind definition tells just that. There is also optional intent setting that can be used to define subtypes - but more on that later.

The schema handling section can also be used to augment (or even override) some parts of the resource schema. E.g. following example sets a display name for this object type. The display name will be used by the user interface when it displays the account.

However, the most powerful feature that is used in the schema handling is the ability to deal with attributes. Following sections are all about that.

Resource objects such as accounts or groups are mostly just a bunch of attributes. Almost all the IDM magic is about setting the correct attribute to the correct value. The schema handling section of the resource definition is the place where that magic happens.

The object type definition contains sections that define behavior of each attribute that we care about:

There is an attribute element for every attribute that we need to handle. The attribute elements are used to set up the attributes that a typical user account has. They are used to assign identifiers, set up full name, set description and telephone number attributes and things like that. Lot of details can be defined here: display name of the attribute for use by the user interface, limitations, override settings and so on. However, the most important things that go there are the mappings. MidPoint evaluates the mappings in attribute elements to populate account attributes with the correct values. In the simplest form, a mapping looks like this:

The mapping specifies that the value of the cn attribute will be taken from the fullName property of the focal object (which is typically a user). This a very simple mapping, there is no value transformation, no condition – nothing complicated at all. This is how a lot of mappings look like. However, mappings can also be very powerful and complex. That will be described in next section.

Mapping is a very flexible mechanism that takes one or more input properties, transforms them, and puts the result in another property. Mappings are used all over midPoint. However, perhaps the most important use of mappings is in the schema handling part of the resource definition, where they are used to set up account attribute values. We have already seen a very simple mapping that simply copies the values from one place to another. Now it is the time to look at mappings in their entirety.

Mapping consists of the three basic parts:

The three parts of the mapping, as well as the basic principle, is illustrated in the following diagram:

The diagram shows a mapping that takes employeeNumber user property and transforms it to description account attribute by using a simple Groovy script expression.

The source part of the mapping specifies that there is a single source, which is based on employeeNumber user property. Source definitions are important for the mapping to correctly process relative changes (deltas), resolve mapping dependencies, etc. The source definition tells mapping that the value of employeeNumber user property should be passed to an expression.

The expression part contains a simple Groovy script that prepends the prefix emp# to the employee number value, specified by the source definition. The expression part of the mapping is very flexible. There is a lot of ways that can be used to transform a value, generate new value, use a fixed value, pass a value without any change and so on.

The target part defines how the result of the expression should be used. In this case, the result is to be used as a new value for description account attribute. The target definition is necessary, so the mapping can locate appropriate definition of the target property, and make sure that the expression produces a correct data type, and that other schema constraints are maintained (e.g. single vs multiple values).

This mapping can be expressed in XML:

Not all parts of the mapping are mandatory. If the expression is not present, then "as is" expression is assumed. Such expression simply copies the source to target without any transformation. Some parts of the mapping may be implicitly defined by the surrounding context. E.g. target or source is implicit if the mapping is used to define attribute behavior in the schema handling section. Therefore, it is usually sufficient to define either source or target for mappings in schema handling.

The following example specifies a mapping in element outbound. There is explicit definition of mapping source, specifying that mapping input is familyName property of a focal object (which is usually user). The mapping has no expression defined, therefore it defaults to the "as is" expression. Because the mapping is specified in attribute element, the mapping has an implicit target, which is the sn attribute.

In this case, the mapping notation can even be shortened even further. It is quite clear that the mapping source will be one of the properties of the focal object (user). Therefore, the $focus prefix can be omitted:

Those examples are still very simple. Mappings can do much more – as you will learn in a while. However, there is one more thing that we need to explain before going on. Mappings are designed to work with more than just a single source. Following diagram illustrates a mapping that takes two arguments: given name and family name. The mapping produces full name by concatenating these value with a space in between. This is the kind of mapping that is frequently used to construct user’s full name from its components. While the mapping may seem simple, there are some sophisticated mechanisms hidden inside.

The mapping is represented in the XML form as follows:

There are two sources, specified by the source definitions: user property givenName and another user property familyName. The mapping is using script expression to combine the values into a single value, which is used to populate user’s fullName property.

This example also illustrates that the mappings are quite smart. The mapping may be evaluated only if one of the sources changes, or if a full recompute is requested. In case that neither givenName not familyName changes, there is no need to re-evaluate that expression. This is one of the reasons for requiring explicit source definition in the mappings. Without such definitions it is not (realistically) possible to reliably determine when and how the expression should be re-evaluated.

Mappings are used all over midPoint, in many places and situations. Sometimes a mapping needs to be really authoritative. It has to enforce the value to the target. Yet sometimes, we want to provide a default value, and the mapping should never change the target value once it is set. Therefore, mapping can be set to various levels of strength: from weak to strong. Following table describes how that works:

The strength can be specified in any mapping, by using the strength element:

When it comes to mapping strength, the following rule of the thumb may be useful: If you want to enforce policy, use strong mappings. If you just want to set a default value, use weak mapping. If you are not sure what you are doing, then normal mappings will probably work just fine.

Expression is the most flexible part of the mapping. There is approximately a dozen different types of expressions ranging from the simplest as is expression, through the scripting expressions, all the way to a special purpose expressions that search through midPoint repository. Expression type is determined by the element that is used inside the expression part of the mapping. We refer to those elements as expression evaluators. You can find detailed description of expression evaluators in midPoint documentation. We are going to deal only with few popular types:

The simplest expression evaluator is asIs. It simply takes the source, and copies the value to the target. It obviously works only if there is just one source. It is also the default expression evaluator. If no expression is specified in the mapping, then asIs is assumed. It is used like this:

In the example above, the asIs expression was specified explicitly. As asIs is the default expression evaluator, we can save some typing and shorten the notation:

Literal expression evaluator is used to place a constant value in the target. This expression does not need any source at all. It always produces the same value. Following code sets the attribute o to fixed value ExAmPLE, Inc.:

The generate expression evaluator is used to generate a random value. As such it is used almost exclusively to generate passwords. We will deal with that expression later when we will be dealing with credentials.

The most interesting expression evaluator is undoubtedly the script expression evaluator. It allows execution of arbitrary scripting code to transform the value. Basic principle is simple: values from source properties are stored in the script variables. Script is executed, and it produces an output. Return value of the script is stored in the target.

We have already seen a mapping that has a scripting expression:

There are two sources: givenName and familyName. The values of these user properties are placed in variables used by the script. The variables have the same names as the sources: givenName and familyName. Then the script may do whatever it needs to do. It may use input variables, or it may use any other data available in the platform. At the end, the script has to return a value. The script above is written in Groovy, therefore the return value is the value of the last evaluated expression. In this case it is the only expression in the script, which concatenates the two variables with a space in between. Script return value is placed in the target, which in this case is fullName user property.

Scripts are often used to transform the values before they are stored in account attributes. One very common case is construction of LDAP distinguished name (DN). The DN is a complex value in the form of uid=foobar,ou=people,dc=example,dc=com. However, it is easy to construct such value using a simple script:

Midpoint supports three scripting languages:

All three languages can be arbitrarily mixed even in a single midPoint deployment - although, quite understandably, such a practice is not recommended. The language can be selected for each individual expression by using language URI:

Scripting expressions can do almost anything. There is still more to them that meets the eye. This section provides only the very basic description to get you started. Will get back to the scripting expressions many times in this book.

In midPoint, the term activation is used to denote a set of properties that describe whether an object is active. This includes properties that describe whether the user is enabled or disabled, since when he should be enabled, to what date he should be active, and so on. The simple binary enabled/disabled flag might have been sufficient in the 1990s. That was a long time ago. We need much more than that. Therefore, midPoint activation is quite a rich data structure. We are going to describe just the basic idea now, the details will follow later.

The most important activation concept is administrative status. Administrative status defines "administrative state" of the object (user), i.e. the explicit decision of an administrator whether the user is enabled or disabled. Except for administrative status, there are also validity times, lockout status, various timestamps and metadata. We will get to that later.

The important thing to realize is that both user and the accounts have activation properties - and they are almost the same. The user and account activation are using the same property names, meaning and data formats. This is important, because you would probably want account activation to follow user activation. E.g. if user is disabled, then also all his accounts should be disabled. This is very easy to do in midPoint, because the user and account activation are compatible. Therefore, all it takes is a very simple mapping. There is a special place in the resource schema handling section for that:

It is as simple as that. Just an empty mapping represented by empty outbount element. User has administrativeStatus property, account has administrativeStatus property, therefore midPoint knows what is the source and target of the mapping. We do not need to specify these. The values of the administrativeStatus property has the same type and meaning on both sides. Therefore, the default asIs mapping is just fine. We do not need to specify that either. All that midPoint needs to know is that the mapping exists at all - that we want to pass the value from user to account. That is a reason for having outbound element there, even an empty one. MidPoint will fill in all the details.

When this mapping is in place and the user gets disabled, the account will be disabled as well. When the user gets enabled, the account will follow suit.

Credential management is important part of identity management. There are many systems in an organization, almost all of them require a password. Users set up a password, then they forget the password, then they request password reset, setting a new password, which they forget as well. The endless cycle continues. While there is probably no magic formula to completely solve this problem (perhaps except for removing passwords altogether), the situation can be made less painful.

The usual method is to synchronize the passwords among all the systems in one organization. This does not make the problem go away. However, it reduces many set-forget-reset-forget cycles to just one. Moreover, as users are using that one password quite often, they are less likely to forget it. Of course, having the same password on many systems is not exactly the best security practice. However, security is all about trade-offs. Having the same password set for all systems in one organization is an acceptable risk in vast majority of cases.

MidPoint is designed to easily synchronize credentials to many accounts. Similarly to activation, credential data structures of user and account are aligned. Therefore, all that is needed to synchronize password to an account is a simple empty mapping:

When the user password in midPoint is changed, the changed password will be propagated to all the resources that have a mapping like this.

This section describes a complete working example of connection to the LDAP directory. The configuration below is used to automatically create accounts in OpenLDAP server. Entire configuration is contained in a single resource definition file. Following paragraphs explain individual parts of the file. Simplified XML notation is used for clarity. Complete file in a form directly usable in midPoint can be found at the same place as all the other samples in this book (see Additional Information chapter for details).

Resource definition begins with object type, OID, name and description. These are self-explanatory:

Connector reference comes next. We want to point to the LDAP connector. Here we use dynamic reference that is using search filter to locate the connector:

The reference is resolved when this object is imported to midPoint. The resolution process takes the search filter, and it looks for connector object with the connectorType specified in the filter.

Connector configuration goes next. This block specifies connector configuration properties such as hostname, port, passwords and so on.

These parts alone should already define a minimal resource. If you define just the name, connector reference and connector configuration you should be able to import the resource to midPoint. The connection test should pass. However, there is absolutely no IDM logic or automation yet. That is what we are going to add next.

Connector configuration is usually followed by schema element. However, if you look at almost any file that contains resource definition, you will find no such element. The schema element is automatically generated by midPoint when midPoint connects to the resource for the first time. Therefore, there is no need to include schema element in the definition.

What we have to include in the definition is the configuration that tells midPoint how to handle the schema. This is defined in schemaHandling section. Our schemaHandling section contains just one objectType definition. We are going to define how to handle ordinary user accounts on our OpenLDAP server.

This is the place where we define the kind of objects that we are going to handle. In this case it is account. This object is default account. Which means that it will be used in case that the account type is not explicitly specified. There is also specification of a display name. Display name is not used in automation logic. It is used by the user interface when referring to this definition. Finally, there is specification of the object class. The inetOrgPerson object class will be used to create new accounts. The object class specification determines what attributes the account have.

The objectType definition also includes a specification of attribute handling. There is one section for each attribute that we want to handle in automated or special way. It starts with the most important attribute: LDAP distinguished name (DN):

The ref element specifies name of the attribute that we are going to work with. In fact, this is a reference to automatically-generated schema part of resource definition. Definition of display name follows. Display name is used by the user interface as a label for the user interface elements (fields) that work with this attribute. This definition sets a nice "Distinguished Name" label instead of cryptic "dn" which would be used by default.

Let’s skip the limitation definition now. We will come back to that later.

The outbound mapping definition follows. This is where the automation logic is specified. This is the place where the DN value is computed. The name property of the user object is the source for this mapping. The name property usually contains username (login name). Its value is used by scripting expression in the mapping. The expression is supposed to create a DN in the form:

uid=username,ou=people,dc=example,dc=com

This expression is a clever one. It does not do the work all by itself. It invokes a library function to compose the DN. It may look like a good idea to use simple string concatenation to construct a DN. However, that fails in case that the DN components contain certain characters that need to be escaped in the final DN. The composeDnWithSuffix library function takes care of that, and it creates a proper DN.

The outbound mapping is evaluated whenever we need to construct a DN. This obviously happens when a new object is created. However, the same mapping is used when a user is renamed (i.e. his username changes). This is the reason that the mapping needs specification of source. Rename is often quite tricky and complicated operation. It may not be cheap, and in some cases it may not be entirely reversible. We definitely do not want to trigger DN changes unless they are really needed. The specification of the mapping source tells us when the DN change is needed. In this case, it tells us to change the DN if the name property of the user object changes.

Now it is the right time to go back to the limitations section. The dn attribute is defines as mandatory attribute by the schema. Strictly speaking, that definition is perfectly correct: LDAP object cannot be created without a DN. As midPoint is using schema for everything, when midPoint displays a form to edit this LDAP account, it will require that DN has a value, because it is a mandatory attribute. However, normally we do not want users to enter the DN in the user interface forms. We want to compute DN automatically - which is exactly the point of the outbound mapping above. Yet, midPoint does not know when the expression computes a value and when it does not. The expression is a generic piece of Groovy code, there is no telling what it does until it is executed. As far as midPoint can see, the expression can produce any value, including empty one. Therefore, even if there is an expression, midPoint sticks to the schema, and it still requires that DN value is entered by the user. However, we have written the expression, and we know that it will produce a value for any (reasonable) input. Therefore, we want to tell midPoint that the DN is no longer mandatory – that the user does not need to enter DN value in user interface forms. That is exactly what the limitations section does. This section overrides the automatically generated schema, and it turns the dn attribute from mandatory to optional.

Now we have defined the behavior of the dn attribute. We can use similar approach to define the behavior of other attributes as well. E.g. the handling of the cn attribute has similar definition:

In this case there is outbound mapping, but it has no explicit expression. Which means that the value is taken from the source without any change ("as is"). Therefore, the attribute cn will have the same value as user property fullName.

It is also possible to define an attribute without any mapping:

This means that midPoint will not provide any automatic handling for the entryUUID attribute. This definition is used just to set a user-friendly display name for the attribute.

Mappings and expressions have almost unlimited flexibility. E.g. the following definition sets a static value for the description attribute:

This mapping has no source, because the source does not make any sense for literal expressions. Static values are always the same, regardless of the source. You can also notice that this mapping is weak. It is used to set the description attribute only if that attribute does not have any value already. It does not overwrite existing values.

The inetOrgPerson object class has much more attributes than those defined in the schemaHandling section. Those attributes will be automatically displayed in the user interface. MidPoint uses the generated resource schema to determine their names and types. MidPoint displays these attributes, the user can change them and midPoint executes those changes. However, apart from that, midPoint does not do any special handling on those attributes. It is all right not to enumerate all the attributes in schemaHandling section. You only need to define those attributes which you want to handle in a special way.

There are two more definitions to describe, before our example is complete. First definition is the activation definition. It is very simple:

This is a definition that specifies handling of the activation administrative status. This status property specifies whether account is enabled or disabled. Activation properties are somehow special in midPoint. MidPoint understands the meaning and the values of activation properties. MidPoint also expects that user activation and account activation are usually mapped together. Therefore, it is enough to tell midPoint that you want such mapping. MidPoint already knows the source (user activation) and the target (account activation). If the user is disabled then the account will get disabled. If the user is enabled than the account will get enabled.

The last thing that we need for the resource to work well is to define a mapping for credentials. In this case it is a password mapping:

Similarly to activation, the credentials are handled in a special way. MidPoint understands how credentials work, what their values are, and how they are used. MidPoint also expects that user credentials, such as passwords, are usually mapped to the account credentials. Therefore, all that midPoint needs to know is that you want to do that mapping. It can automatically determine the source and target. The account will have the same password as the user. User’s password is used when a new account is created. When user changes his password, the change is also propagated to the account.

That is it. Now you have your first (almost) working resource. You can import the definition to midPoint and test it. Simply assign the resource to a user. The OpenLDAP account will be created - the DN and all the essential attributes will be automatically computed. When midPoint creates an account for a user, it remembers who is the owner of that account. Therefore, it can easily delete the account when needed. Unassign the resource, and the account will get deleted. This is how automated provisioning and deprovisioning works. No hardcore programming is needed, just a declarative specification, and a line or two of very simple scripting. Such configuration can be done in a couple of minutes. It is essentially the same process for all the applications, just the connector is different. The connector has different configuration properties, the attribute names are different - but the principles and the tools are the same. It is easy to connect many heterogeneous applications in this way. The connectors and the mappings are hiding the differences. In the end, all the "resources" look the same to midPoint. The same principles are used to manage them. Therefore, the management can be done efficiently, even at a large scale.

Yet this configuration is still extremely simple. We are just scratching the surface of what midPoint can do. There is much more to see in the next chapters. However, we still need to explain one more fundamental midPoint concept before getting there.

Linking users and accounts is one of the basic principle of any decent identity management system. However, it is surprisingly difficult to implement such link. There are numerous methods to reliably identify accounts, and they vary from system to system. Some systems identify accounts only by username - which makes reliable detection of rename operations quite difficult. Other systems improve on that by introducing another identifier, an identifier that is persistent. Identifier value is assigned by the resource, and it never changes. However, username is still used as secondary identifier, and it still has to be unique. Yet another system may have compound identifiers that consist of two or more values. Some system have globally-unique identifiers, while other systems have local identifiers that are only unique in their own object class. Some systems have hierarchically-structured identifiers, others have flat unstructured identifiers. Some identifiers are case-sensitive strings, others are case-insensitive, some identifiers follow complex normalization rules, and yet another identifiers are binary and completely opaque. To make long story short: reliable identification is really complicated.

We do not want to pollute user object with all the delicate details of account identification. Therefore, we have created a separate midPoint object that hides all the resource-related details and identification complexities. We call it shadow, because it behaves as a shadow of the real account. When midPoint detects new account, a shadow is automatically created to represent the account in midPoint repository. When midPoint detects that account has changed (e.g. it was renamed), then midPoint automatically updates the shadow. When the account is deleted, midPoint deletes the shadow. Technically, shadow is still an ordinary midPoint object. Therefore, it has object identifier (OID). Other objects can simply point to the shadow using ordinary object reference. That is exactly how user-account links are implemented:

The shadow objects contain all the data that are needed to reliably identify an account - or any other resource object such as group or organizational unit. In addition to the account identifiers, shadow points to the corresponding resource definition, to make the identification complete. Shadows are multi-purpose objects, and they have many uses in midPoint. Shadows record meta-data about resource objects. They are used to hold cached values of the attributes (this functionality is still experimental in midPoint 4.8.5). Shadows can be used to hold the state of the resource objects for which midPoint does not have on-line communication channel and the operations are executed manually (a.k.a. "manual resources"). Therefore, shadows are quite complex objects. Following picture provides more substantial example of a shadow.

Do not worry if that picture looks a bit scary. Shadows may be complex, but they are almost always invisible to midPoint users. Shadows are automatically and transparently maintained by midPoint core engine. Under normal circumstances, MidPoint does all that is needed to maintain the shadow, and no special configuration is needed for that. We are describing the mechanics of the shadow objects mostly for the sake of completeness. There are situations when this knowledge may be useful. These are usually situations when midPoint was mis-configured, and the shadows were created incorrectly. In that case you may need to update the shadows, or even purge all shadows and start over.

MidPoint is configured as a simple provisioning engine now. We can use midPoint to create, modify and delete accounts on many diverse systems. However, we are still at the beginning. MidPoint is pushing the changes to other systems, but it is not listening for any changes yet. Similarly to people, systems that talk all the time and never listens do not make good companions. Therefore, the next task is to make midPoint a better listener, by configuring synchronization mechanisms.

Synchronization is one of the fundamental mechanisms of midPoint. Synchronization mechanisms are integral part of midPoint design from its very beginning. Many of the things that midPoint normally does are in fact just different flavors of synchronization. There are obvious cases such as reconciliation process, synchronizing account attributes with data in midPoint repository. However, there are also less obvious cases, such as ordinary provisioning operation when midPoint needs to create a new account for a user. Even that case is in fact a synchronization: midPoint user properties are synchronized with a new empty account on the resource. Majority of midPoint operations are directly or indirectly using the synchronization principles.

MidPoint synchronization provides a continuous functionality spectrum that can be tweaked and tuned to match specific needs. Yet, the synchronization mechanisms can be divided to several broad and slightly overlapping categories:

Individual mechanisms differ in a way data inconsistency is discovered: live synchronization actively looks for new changes, reconciliation compares the data one-by-one and opportunistic synchronization discovers inconsistency by chance. Yet, all the mechanisms react to inconsistency in the same way. There is only one policy that specifies how to fix the data. Of course, there may be slight deviations in the behavior. For example, we usually want import to behave in slightly different way than reconciliation. MidPoint allows that. Yet, there is just one common synchronization mechanism. This has a very good reason. It does not really matter how the problem was discovered. What really matters is that the problem gets fixed. We do not want to maintain four separate configurations for every delicate variation of the functionality. Having one policy is much better. MidPoint knows which part of the configuration need to be applied in each specific situation, and it does it automatically. This unifying approach significantly simplifies the configuration of midPoint synchronization mechanisms. That is also a reason why the boundaries of individual synchronization mechanisms are quite fuzzy. In fact, it is just one big mechanism with several facets.

The tale of idealistic identity management deployment starts with a human resources (HR) system. The HR system is supposed to have records for all the identities, therefore it is authoritative source system. Identity management system pulls in all the data from HR database, recomputes them and creates accounts on target systems. And they lived happily ever after.

Now, let’s get back to reality. The HR database is indeed an authoritative source of data in many real-world cases. However, it is a limited source. It contains data about employees only, and it has only partial information about them. For example, there is no username in the HR record. Username has to be generated by IDM logic. There is no initial password. Organizational structure assignment is often incomplete or unreliable. Therefore, HR database is only a partially-authoritative source. There may be additional authoritative sources for contractors, partners, suppliers, support engineers and other identities that need to access our systems. These are additional source systems. Then there is a directory system, which is often Microsoft Active Directory. It should be a target resource, in theory. Yet, there may be pieces of authoritative information in here. For example, an algorithm to generate a username may be based on the usernames that are already used in the Active Directory. The data in Active Directory may also be needed to create a unique e-mail address. Directory systems are also used as a semi-authoritative sources for telephone numbers, office numbers and similar identity attributes. Therefore, such systems are both target and source systems. Then there are "pure" target systems. These are not supposed to be authoritative in any way. Identity management system will only write to such systems. Or …​ will it? What happens when a conflicting account already exists on such system, and therefore we cannot create a new account for a new employee? How do we check if there are no accounts that are not supposed to be there? It turns out that even the "pure" target systems contain valuable sources of information after all.

The reality brings a wild mix of source, target, semi-source, target/source and quasi-target systems that are almost impossible to put into a pre-defined boxes. Therefore, midPoint does not bother to define a concept of "source" or "target" resource. Any resource can be both source and target, and the authoritativeness of each attribute can be controlled on a very fine level. Almost every real-world situation can easily fit into this model.

We are still using the terms source system and target system. However, these terms refer to the business purpose of such systems in the identity management architecture. The terms describe how we think about the systems, what is the usual direction of data flow. However, midPoint will read data from target system when needed, and it will be happy to write data to source systems if necessary.

MidPoint is firmly based on the principle of reuse. Previous chapter explained that behavior of attributes during provisioning is controlled by mappings. Therefore, it is perhaps no big surprise that the behavior of attributes during synchronization is also controlled by mappings. In fact, provisioning is just a special case of synchronization. Following picture explains the combined mechanism.

There are two types of mappings:

The mappings themselves are almost the same regardless whether they are inbound or outbound. They have sources, targets, expressions, conditions, etc. Just the sources and targets are reversed:

That is it. Think about the mappings that were used in previous chapter, just flip the direction. Now the mapping will take data from the account and the results will be applied to user object. Like this:

This mapping takes the value of lastname attribute from the resource and stores the value in familyName property of midPoint user.

The rest is the same as outbound mappings. All the expressions and evaluators can be used for inbound mappings in the same way as for outbound mappings. For example, a Groovy expression can be used to sanitize the value before it is stored in midPoint:

The same approach can also be taken for activation, and even for password mappings. However, there is one difference for password mappings. Password is usually write-only value. When the password is written, it is usually hashed, and the original value cannot be retrieved any longer. Then there are resources such as HR systems that do not store employee passwords at all, because those are not really accounts that we are reading. Those are just regular database entries that the connector presents as accounts. Inbound password synchronization is almost never easy, and it often requires a lot of planning and ingenuity. However, there is one method that is used quite often. The initial user passwords are usually randomly generated. As this is a very common case, midPoint can do this easily:

This mapping generates random password for a user. Both the mapping and generate expression evaluators are quite smart. The mapping knows that the target is user password, without any need to explicitly specify that. In addition to that, generate expression evaluator will take password policy into consideration. It does not make sense to generate any random password. If we do not consider password policy, then we can generate password that is too short, too long, too weak or too strong to be useful in any way. Therefore, generate expression looks for password policy, and generates a random password that just matches requirements for password length and complexity.

There are more important details to see here. The inbound password mapping is weak. There is good reason for this. We do not want existing midPoint password to be replaced by randomly generated password. We only want to set a random password in case that it is an initial password, the first password ever. That is exactly what a weak mapping does: it sets new value only if the target does not have any existing value. Therefore, this mapping will not overwrite passwords that are already set.

It is quite easy to import all HR records into an empty midPoint: we have to set up inbound mappings, start import task, wait a bit, and all is done. However, practical situations are much more complex. Synchronization usually does not run on a green field. Live synchronization and reconciliation are supposed to work with pre-existing midPoint users. Import may not be entirely trivial either, for example we may import data from an additional data source into a running midPoint deployment. Some users in the import data set are new, but there may be accounts describing existing users. We need to tell the difference between brand-new account and an account that belongs to an existing user. We need to handle them in a different way. Of course, midPoint has a solution for this: correlation mechanism.

Correlation is a method to connect newly-discovered accounts and existing users. Whenever midPoint discovers new account it will try to link that account to an existing user. MidPoint looks for an appropriate user to represent newly-discovered account. It does so by matching selected account attributes to appropriate user properties. Two things are needed for this to work: correlation rule and corresponding inbound mapping.

Correlation rule, better known as correlator, specifies which properties are used for correlation, and how they are used. In its simplest form, the correlator points to a single user property:

Correlator, as shown above, specifies that the personalNumber property should be used to correlate accounts and users. If a newly-discovered account has a value that matches the personalNumber of a user, it is assumed that such user is an owner of the account. Such account and user are correlated, and they will be linked.

Now we know that personalNumber property of midPoint users should be used for correlation. However, which account attribute should we use to match values of personalNumber property? That is where the mapping comes in. MidPoint looks for inbound mapping that maps a value of an attribute to personalNumber property, such as this one:

MidPoint knows that it needs to take value of account attribute empno to produce value for user property personalNumber. This is what midPoint normally does when it synchronizes account data to user data. The same mapping is used for correlation. MidPoint takes the value of empno attribute, passes it through the mapping, and compares the result with the personalNumber property of all users. The user that has a matching value is correlated to the account.

The correlator and the mapping always work together. Mapping transforms the value, correlator matches the value.

We have seen only very simple correlator so far. However, correlator can be quite complex. Correlation may be set up to match several items, correlation may be based on custom search filter or expression, and several rules may be used in composition, using specified weights to tune the process. Correlation can use approximate (fuzzy) matching with human assistance, submitting probable matches for approval by administrator.

Even though correlation mechanism is very rich, the humble item-based correlator remains the most popular one. Therefore, there is simplified method to configure it. Definition of correlation item can be placed directly in account attribute definition:

The correlator clause in the mapping marks the empno attribute for correlation. It will be matched with the mapping target, which is personalNumber user property. This notation is all that is needed to set up simple correlation scheme.

We have dealt with correlation for source resources so far, such as the HR systems in our ExAmPLE scenario. However, there is a slightly surprising fact, that correlation is usually not necessary for source systems with a single authoritative source. In such cases, the users are created from the accounts originating in the source systems. The users are automatically linked to the originating accounts at the moment of creation, no correlation is necessary there. The correlation is necessary in case that there is another source of data which overlaps with the primary source, such as second source system or manual entry of user data using user interface. Even though correlation mechanism may not be necessary in some cases, it is a best practice to set it up for production deployments. Having it set up from the beginning lowers the probability that it will be missing when new data source is added, or in case that system administrator manually creates user during HR system outage. It makes the deployment more resilient.

Apart from source resources, correlation is absolutely essential for handling target resources. It is a very rare occasion to connect an empty target system to midPoint. Almost all the target systems that are connected to midPoint contain existing data, existing accounts that belong to users in midPoint. When such target system is connected to midPoint, we have to correlate the account to midPoint users to correctly set up the links. The correlation mechanism is the same, regardless whether it is used for source or target systems - except for one little difference. While we would have a suitable inbound mapping for source systems, we may not have that for target system. Target systems often use outbound mappings only. Therefore, we need to add the missing inbound mapping to be used to correlate target systems:

In this case, there are two mappings for employeeNumber attribute. The usual outbound mapping is used for ordinary provisioning, it populates the target attribute employeeNumber with value taken from personalNumber property of midPoint user. The other inbound mapping is used for correlation. It takes the employeeNumber account attribute to correlate it with personalNumber user property. However, we do not want that inbound mapping to be used to copy attribute values, as that could overwrite the authoritative information of personalNumber user property with incorrect data. Therefore, we are setting up this mapping to be correlation-only, using the evaluationPhases clause. The clause specifies that the mapping should be used to provide data for correlation (including beforeCorrelation phase), however it should not be used for ordinary synchronization processing (excluding clockwork phase).

It is important to realize, that correlation is always a database search. We have value of account attribute on one hand, and we have the entire user database on second hand. We have to find one specific user in the entire database, based on the value of correlation attribute. Looking for each individual user one-by-one is not a scalable method, we have to utilize database search capabilities to make it efficiently. Therefore, when choosing a correlation property, make sure it is a searchable (indexed) property.

Corrlators are efficient method to automatically link large number of accounts and users. However, correlators rely on reliable and clean data. If employee numbers were consistently recorded for every account, correlators can do all the work. If usernames were applied consistently across all systems, correlators are going to be a great tool. However, we all know that there is an exception to every rule. Practical data are not perfect, there are omissions, typos and all kind of errors. Therefore, in practical scenarios, there are always some accounts that cannot be correlated automatically. Manual action is necessary to correlate them. The usual approach is to use Change owner button in account list to manually set up owner for an account.

Correlation mechanism can be used to find an owner for a new account. That is a part of the solution, but it is not a whole solution yet. If the owner is found, then the action is quite obvious: link the account to the user and proceed as usual. However, what to do if the owner is not found? This resource may be an authoritative resource, and therefore we want to create a new user based on the account. On the other hand, this may be a reconciliation with a target resource, and we have just found an illegal (orphaned) account. We probably want to deactivate such account in this case. Moreover, what to do if more than one owner is found? There are many scenarios, and the situation can become quite complicated. Therefore, midPoint has a concept of synchronization situations to make all the possible cases understandable and manageable.

Whenever midPoint deals with a change on an account, the situation of that account is determined. The situation reflects whether this account is already linked to the user, whether we know the candidate owner, whether we cannot determine the owner at all, and so on. Individual situations are explained in the following table.

After synchronization situation is determined, midPoint continues by figuring out what a proper reaction is. The reaction may be quite clear for some situations (e.g. unlinked), but there is a lot of variability for other situations (e.g. unmatched and deleted). This variability is a reason that midPoint allows to set a reaction for each situation individually. There are several pre-defined reactions:

If no reaction is explicitly configured for a situation, then midPoint does nothing. The situation is recorded in midPoint repository, but no other action is taken. This is part of midPoint philosophy: not to change the data unless an action was explicitly configured.

The reactions can be defined in the synchronization section of object type definition in resource configuration:

This is a typical configuration for an authoritative resource. Most of the configuration is perhaps self-explanatory:

Perhaps the only thing that deserves special explanation is the synchronize reaction. In the linked situation, there is not much to do. Everything seems to be in order. However, there may still be attributes that are not set correctly. We may need to fully apply all the inbound and outbound mappings, to make sure that all the data are correct. This is what the synchronize reaction does, among many other things. In fact, we want to do the "synchronize" thing as part of all other reactions too. MidPoint does that implicitly, there is no need to add synchronize to every other reaction. However, we need to explicitly add synchronize to the linked situation, as it is the only reaction that there is. If we leaved the linked situation empty, midPoint does nothing. Putting synchronize action there tells midPoint do to the usual thing.

Synchronization reactions are used to configure resource for a variety of scenarios. The addFocus reaction is used to configure authoritative resources, other reactions may be used to configure target resources, semi-authoritative resources and so on. Synchronization reactions provide control on account level, while mappings provide control on attribute level. Together they can be used to implement variety of synchronization scenarios.

Correlation and synchronization actions are just two pieces of much bigger synchronization puzzle. Connector provides account data, correlation looks for an owner, synchronization situations and reaction decide what to do with the account, and so on. Every mechanism has its place in a comprehensive synchronization flow.

The figure above illustrates the usual sequence of events during inbound synchronization:

When the data are reflected to user, the usual midPoint recompute process starts. This also includes evaluation of object templates, roles, policies and all the other things that will be covered in following chapters. This is the main part of midPoint processing, internally referred to as "clockwork". It is essentially the same processing as if user was modified by an administrator using midPoint user interface.

Now we know how the inbound synchronization works: midPoint reads the account, then correlation is applied, situation determined and reaction executed. However, we have not yet discussed the details of the very first step: how does midPoint actually gets account data? Nothing happens without a reason, therefore there must be some active component in midPoint that looks for the new, changed and deleted accounts. That component is a synchronization task.

MidPoint task is an active process that runs inside midPoint server. This is the first time that we encounter the concept of a task, but it is definitely not the last one. Tasks are used for numerous purposes in midPoint. They are used to track long-running operations and actions that work on large sets of objects (bulk actions). There are tasks that execute cleanup jobs, compile reports and provide variety of other functions. The concept of a task is a very powerful and flexible one. Tasks can be used to track execution of a short one-off operations. Tasks can be used to execute scheduled actions in regular intervals. Tasks can be used to track long-running processes. We will be using tasks in almost every chapter of this book.

Tasks are used as an active component to "run" almost all synchronization mechanisms:

Each type of synchronization task is detecting changes using a different mechanism. However, once the task detects the change or reads the account, then the processing is the same for all tasks. All the tasks lead to the same algorithms based on the same configuration and policies. Therefore, it does not matter whether it has all started in reconciliation or live synchronization task, it will all end up in the same correlation-situation-reaction-mapping flow.

The tasks are necessary to initiate the synchronization. They are the active part, the spark that starts the synchronization process. Without the tasks the synchronization does not really work. There are ways the synchronization can "happen" even without a task, e.g. as a reaction to user interface operation, or if a new account is discovered during an unrelated operation. Despite that, practical deployments need at least one synchronization task to work properly. This task takes care of the vast majority of synchronization cases.

Strictly speaking, a task is quite a strange kind of animal. Tasks have their data and configuration as most other midPoint objects have. Unlike other objects, tasks are active, they run. Therefore, there are CPU threads associated with the tasks when the tasks are running. There are mechanisms to monitor task progress. The tasks need to be cluster-aware, they have to fail over to a different midPoint node if one node fails. The tasks are quite complex, and they may be a bit tricky to handle. Fortunately, midPoint is making task handling reasonably simple. Tasks are represented as ordinary midPoint objects. Therefore, tasks can be imported to midPoint in XML/JSON/YAML form as any other object. Tasks can be easily edited in their XML/JSON/YAML form to change the scheduling, modify the parameters, and so on. Of course, there are some special functions that only the tasks have (such as suspend and resume). Such functions cannot be directly controlled using the XML/JSON/YAML format. However, the vast majority of task management can be done using the very same methods that are used to control other midPoint objects.

Tasks can be created by taking the XML/JSON/YAML file and importing that to midPoint. That is the way synchronization tasks are often managed. When an XML-formatted resource definition is created, then there is often an associated synchronization task. Which means that both resource and all the necessary synchronization tasks can be imported together. Synchronization tasks can also be created from midPoint user interface. They are usually created by using special-purpose Defined tasks menu item in resource detail pages.

Once the synchronization tasks are created, they can be managed in the same way as other tasks are managed: in the Server tasks part of the midPoint user interface.

This section describes complete working example that feeds HR data into midPoint. The HR system used by ExAmPLE company is an old and complex thing. Therefore, the easiest integration method is to use structured text exports. The system can export employee data in a form of a comma-separated text file (CSV). The HR system is set up to export fresh data every night. MidPoint takes this export file, crunches the content, and updates the data about users.

This configuration is done in three steps. First, we create a simple setup to import the data into midPoint. This is an operation that is executed only once. Then the configuration is updated to run scheduled reconciliation task. Reconciliation compares all the data records every time, and it makes any necessary updates. Even though this method would be perfectly acceptable for a company of this size, we are still going set up a live synchronization task, to make synchronization lighter and faster.

The core of the configuration is contained in a single resource definition file. Following paragraphs explain individual parts of the file. There are few additional configuration files for reconciliation and live synchronization tasks. Simplified XML notation is used for clarity. The complete file in a form that is directly usable in midPoint can be found at the same place as all the other samples in this book (see Additional Information chapter for details).

The resource representing HR system is configured as data source. It is used to "pull" the data inside midPoint. However, as we have described previously, there is no fundamental difference between source and target resources in midPoint. Therefore, this HR resource starts in entirely ordinary way. There is a reference to the CSV connector and the connector configuration:

The next section is schema handling configuration. That is where it becomes slightly more interesting. The schema handling section contains inbound mappings for HR account attributes:

The schema handling section starts with the usual definitions of account object type. This is a definition of Default account, using the only object class that our CSV-based HR connector provides. We have seen that before, in previous chapter. However, there is a new part, definition of focus. The delineation and most other parts of schema handling definition refer to account or other resource objects. In midPoint parlance those are seen as projections, the "spoke" part of hub-and-spoke synchronization topology. However, the focus part is different. It refers to the objects in midPoint repository, objects that are at the center of the synchronization topology, the focal object. In our case, user is our focal objects, as we are going to synchronize HR accounts with midPoint users. The other part of the focus definition specifies an archetype, which defines finer characteristics of focal object. In this case it points to Person archetype. User objects that are going to be created during synchronization will have this archetype automatically assigned. We will learn more about archetypes later, in Archetypes chapter.

Next part of the definition specifies properties and mappings of account attributes:

The account attribute empno is mapped to midPoint user properties name and personalNumber. Account attributes firstname and lastname are mapped to givenName and familyName properties respectively. This is perhaps self-explanatory.

The next part of the configuration specifies mappings for activation and credentials:

Activation is a very specific concept in midPoint, controlling whether accounts is active (enabled or disabled) among other things. MidPoint knows activation attributes and their meaning, they are pre-defined in midPoint schema. Therefore, there is no need to specify a lot of details. That makes activation mapping a very simple thing. The configuration specifies that the administrative status should be mapped in the inbound direction. That is it.

However, the mapping for credentials needs a bit of explanation. What midPoint sees as HR accounts are not exactly accounts, they are usually just records in the HR database. Nobody is using these HR records to log into the HR systems. Therefore, there is no password associated with them. Yet, we need a password for the users in midPoint. Therefore, we are going to generate the passwords. For that purpose, we are going to use the weak mapping with generate expression that was explained above.

Next section specifies correlation settings. The ownership of the accounts that are not already linked is determined by the correlation mechanism.

In this case, personalNumber user property is used for correlation. MidPoint knows that personalNumber is mapped from empno account attribute, therefore midPoint compares values of personalNumber user property and empno account attribute. If the values match then the user is considered to be an owner of the account.

The mappings are undoubtedly important part of synchronization configuration. The mappings specify how are the account data reflected to midPoint user. However, the mappings do not specify whether the accounts should be created or deleted. Mappings control the data, but they do not control the lifecycle. It is the next configuration section that makes this resource really authoritative:

Given the information in this chapter, this configuration should be quite easy to read. This a typical configuration for authoritative resource. If there is a new account on the resource, and we do not have an owner (situation: unmatched), then we create a new user (action: addFocus). If there is a new account for which we can find existing owner (situation: unlinked), then simply link it (reaction: link). If the account is linked already (situation: linked), then we just synchronize the data. In fact, we synchronize data for all the other situations as well. Except the last one. If the account is deleted in the HR system (situation: deleted), then we want to delete midPoint user as well (reaction: deleteFocus). As the user gets deleted there is no point in synchronizing the data. MidPoint knows that, and it skips application of mappings.

There is one more little detail in this resource definition:

This is a setting that adjusts the behavior of midPoint assignments. As was already mentioned, all resources in midPoint are created equal. The source resources must follow the same rules as target resources. One of the fundamental rules of midPoint is that there should not be any account without a specific reason to exist. In midPoint terminology, every account exists because there is an assignment that justifies its existence. While this approach is exactly what we want for the vast majority of (well behaving) resources, it is not exactly ideal for resources that are pure data sources. Those resources work the other way around. The HR account is in fact a cause for midPoint user existence, not its effect. Therefore, there is really useful assignmentPolicyEnforcement setting that controls the behavior of assignments. This setting is used in a variety of scenarios, mostly for data migration, and to tame resources that just won’t behave in a civilized manner. However, in this case, the setting is used to turn off the assignment enforcement for this resource entirely. As this resource is an authoritative source, the assignment enforcement does not make much sense. Behavior of this resource is defined by the synchronization section of resource configuration.

Resource configuration is complete now. This configuration sets up the connector, mappings, correlation and synchronization policies. The configuration is the same for all the synchronization flavors: import, reconciliation and live sync - they will all use the same settings. When it comes to configuration, the only difference between those synchronization flavors is the way how the synchronization tasks are set up. If an import task is set up, then import of resource accounts will be executed. If reconciliation task is set up, the reconciliation will be executed. It is all in the tasks. Synchronization tasks can be easily set up using those convenient buttons in the user interface. However, we like to make our lives a bit painful in our part of the world. Therefore, we are going to go hardcore, and we import the tasks in the XML form.

First task is an import task. This task lists all the accounts in the HR CSV file. The task pretends that each of the accounts was just created. If the task is executed for the first time, then resulting situation of the accounts is going to be either unmatched or unlinked. Therefore, the task creates new midPoint users, or links the accounts to existing users.

This is a very basic structure of a task. Similarly to all midPoint objects, a task has a name. Then there is an assignment of Import task archetype. We will describe archetypes later. For now, it is only important to know that this classifies the task as import task, so the user interface can place the task in proper categories. Task needs definition of an owner. The owner is a user that is executing the task. This is important, because authorizations of task owner determine what the task is allowed to do. This is also the identity that will be recorded in the audit log. In this case administrator is owner of this task. Task execution state tells whether the task is running, it is suspended or finished. Tasks are often executed periodically, therefore they need a schedule. In this case, we start with task that is executed just once, to test the configuration. Hence the single recurrence, which specifies that the task runs only once. Then there is definition of activity. Activity specifies what the task really does. In this case the activity specifies that this is a synchronization task which imports accounts from the resource. The resource is specified by the resourceRef reference in the activity specification. This points to our HR resource.

As task execution state is set to runnable, midPoint tries to execute the task when the definition is imported. That means that import of accounts from the HR resource starts immediately. Progress of the task can be monitored in the Server tasks section of midPoint user interface. The import task is not a recurring task, it will run only once. If you need to re-run the task, you can do that from midPoint user interface. However, the task will not get executed again, unless you explicitly tell midPoint to do so. This is very typical behavior for import tasks, they are usually executed only when a new resource is connected to the system. Once everything is set up, correlated and linked, then the import task is not needed any more.

Import task gets the data from the resource into midPoint. As import is not a recurring task, it does not keep the data synchronized. Import tasks are not designed to do so. Fortunately, there are other tasks that are designed for continuous synchronization. Reconciliation task is one of these. Reconciliation task lists all the accounts on a resource and compares that with data in midPoint.

Definition of a reconciliation task is almost the same as the definition of import task. However, there are crucial differences. First of all, there is different activity. This is what makes this task a reconciliation task. Then, the task is recurring. This means that midPoint will repeat execution of the task. Therefore, there is also execution schedule, so the server knows when to execute the task. Reconciliation tasks are usually resource-intensive, therefore we usually want to execute them at a very specific off-peak times. For that reason the execution schedule is defined using a cron-like pattern. UNIX-friendly readers will be surely familiar with this. The format is:

seconds minutes hours day-of-month month day-of-week year

The string 0 0 1 ? * SAT means that this task will be executed every Saturday at 01:00:00am. There is also definition of misfire action. Misfire is a situation when the server is down at the time when the task is supposed to run. In this case, if the server is down in the early hours of Saturday, this task will be executed as soon as the server starts up.

Reconciliation is a real workhorse of identity management. It can be used for almost any resource. It is very reliable. It is often used to fix data problems, apply new policies, look for missing accounts, detect illegal accounts and so on. It is indeed a really useful tool. Yet, it has its downside. Reconciliation iterates through all the accounts, it recomputes all the applicable policies for every account, one-by-one. Therefore, it may be quite resource-intensive. It may be even quite brutal if the policies are complex, user population is high and the resources are slow. This can take hours or even days in extreme cases. Even for smaller deployments, reconciliation is not entirely easy. The problem is not in midPoint. MidPoint can be usually scaled up to handle the load. However, listing all the accounts often may put unacceptable load on the resources, the source and target systems. Therefore, reconciliation is not executed often. Daily, weekly or even monthly reconciliation seems to be a common approach.

Reconciliation is reliable, but it is not entirely what we would call "real-time". Of course, midPoint has a faster alternative.

Live synchronization is the way to go for real-time synchronization - or rather almost real-time synchronization. Practical latencies for live synchronization are in the range of seconds or minutes, which is fast enough for most practical cases. Live synchronization is also quite resource-efficient. Overall, it is much faster and much lighter than reconciliation. Unfortunately, live synchronization is not available for all resources. Live synchronization depends on the ability to get recent changes from the resource in a very efficient way. Therefore, it is only available for resources that record the changes. The specific mechanism to record the changes may vary from resource to resource. It may be as basic as a simple modification timestamp, or it may be a complex real-time change log. The mechanism has to be good enough for the connector to discover recent changes, and it must be efficient enough for the connector to do that every couple of seconds. If such mechanism is available, and the connector knows how to use it, then setting up live synchronization is easy. All that is needed is live synchronization task.

This task definition should be easy to understand by now. There is a different activity that makes this a live synchronization task. There is also a different type of scheduling. We do not want to execute this task at a specific time. We rather want to execute it all the time, at regular intervals. In this case the interval is set to 10 seconds. That is all. We have live synchronization running. If the HR CSV file is changed, the changes will get automatically processed by midPoint.

Setting up synchronization flavors such as reconciliation or life synchronization is just a matter of setting up the tasks. The rest of the configuration is the same for all flavors. Therefore, it is very easy to run both live synchronization and reconciliation for the same resource - just create two tasks. In fact, this is quite a common setup. Live synchronization is used to get the changes quickly and efficiently. Reconciliation is used to make sure all the changes were processed and that the policies are applied consistently.

Now we have the HR feed up and running. However, there are still few issues. A clever reader would surely notice that this is not a very good HR resource. MidPoint users created from this HR feed have given name and family name, but the full name field is empty. Do not worry. We will sort that out in later chapters, with the help of object template. Also, the users have employee number as their username. This may be in fact a very good approach for some deployments as it avoids the need to rename accounts. However, it is not a very user-friendly approach. Therefore, most deployments would like to generate more convenient usernames. This is easy to do with midPoint, and we will also address that later. There is still a lot of things to learn before we get to a complete synchronization set up.

All resources are created equal in midPoint. However, source resources almost always have a slightly special standing. Even though midPoint mechanisms are the same for all resources, the data coming from the sources often have significant impact on the entire solution. There is this traditional computer engineering wisdom: garbage in, garbage out. An error in data feed may cause a lot of problems everywhere. Therefore, it is important to get the data sources right. This is usually one of the first steps in an identity management project.

Unfortunately, source data feed is usually quite difficult to set up correctly - and it is almost impossible to get it right at the first try. There may be good old configuration problems, which are usually easy to fix. There may be data compatibility problems, such as presence of non-ASCII national characters where they are not expected. Worst of all, source data may be of poor quality, there may be inconsistencies, typos, the data may be out of date, not reflecting the reality very well. These problems are the most difficult to correct, as the right way to correct them is to modify the data at its source - in the source database, HR system or spreadsheet exchanged by e-mail. That takes time, meetings, mail messages, management decisions, processes, excuses, delays and huge amount of patience. Therefore, setup of a data source is usually an iterative process. The process usually goes like this:

If the initial identity management deployment step includes an HR feed, we strongly recommend to start with that HR feed. It is significant benefit to have authoritative HR data in the midPoint to start with. It is usually easier to correlate other resources to midPoint users later on, if the users were created from a reasonably reliable HR data. Also, it will usually take some tweaking to get the HR import right. The possibility to easily clean up midPoint and get to a clean slate is extremely useful. However, this "wipeout" approach is possible only if the HR feed is the first resource that is connected to midPoint.

A clever reader would notice, that we assumed that the source feed is taken from a CSV file. This is indeed the case in many deployments. The CSV file is usually produced as an automatic scheduled export of HR system data, running every night. If a new employee or contractor is about to join the company, there is usually no hurry. This information is entered into the HR system at least a few days in advance, therefore daily CSV export is perfectly acceptable. However, there may be cases when we want a faster response. Maybe we do not even want additional burden of dealing with CSV exports. Of course, there is a solution. In theory, any connector can be used for source resource. There are specialized connectors that are taking data directly from the HR system. For example, there is a connector for Oracle HCM system. Unfortunately, there is no connector that can take data from SAP HR system yet.

Synchronization and provisioning are intimately related. Everything that we have explained about provisioning in the previous chapter also applies to synchronization. In fact, provisioning and synchronization are just applications of the same basic mechanisms. Provisioning starts with modification of a user. Synchronization starts a bit earlier: inbound mappings are used to map values from source system to the user. The result of inbound mapping evaluation is modification of the user object. According to midPoint principles, it does not matter how the user was modified. The reaction is the same: accounts are provisioned, modified or deleted.

The synchronization (inbound processing) and provisioning (outbound processing) usually happen in one seamless operation. For example, the HR connector detects update in the last name of an employee. That modification is applied to midPoint user, therefore the family name of midPoint user is updated. The operation continues by evaluating all templates, roles and outbound mappings. The outbound mappings usually map the family name change to the resource attributes. Therefore, the resource accounts linked to the user are immediately updated. All of that happens in a single operation. That is how midPoint works. MidPoint is not a human. It never procrastinates (unless explicitly instructed do to so). MidPoint does not postpone the operation for later if the operation can be executed immediately. MidPoint tries to get the data right on the first try. Therefore, there are no specialized propagation or provisioning tasks that you might know from older identity management systems. MidPoint does not need them.

There are other advantages in doing everything in one operation. It is all one operation, therefore midPoint knows all the details: what was the cause, what is the effect, what exactly has been changed. Such context is extremely important for troubleshooting. Some identity management systems decouple the cause from the effect. Such a divided approach may have its advantages, but it is an absolute nightmare when an engineer needs to figure out why a certain effect happened. For that reason, midPoint has both the cause and the effects bundled together in a single operation. Therefore, it is much easier to figure out what is going on. Having the cause and effect connected in one operation makes it possible to neatly record entire operation in the audit trail. Then there is another huge advantage: midPoint knows exactly what has been changed. This means that midPoint does not only know the new value of a property. MidPoint also knows the old value and values that were added or removed. This is a complete description of the change that we call a delta. This is recorded at the beginning of the operation, and propagated all the way until the operation is done. Therefore the mappings can be smart. This approach enables a lot of interesting behavioral patterns. For example, it is quite easy for midPoint to implement the "last change wins" policy. In this case midPoint will simply overwrite only those attributes that are really changed in operation. MidPoint can leave other values untouched. In fact, this is the default behavior of midPoint. It is a very useful behavior during deployment of a new identity management system.

Careful processing of the operations allows configurations that are not feasible with older identity management systems, e.g. a resource that is both a source and a target. In fact a lot of identity management systems can have resource that is both a source and a target - as long as it is a source for one attribute and a target for another attribute. However, midPoint can live with a resource where the same attribute is both a source and a target. In fact there may be many sources and many targets for the same property at the same time. This is a very useful configuration, indeed. Just think about telephone number property. It is usually something that the user sets up himself. This may be set up by some kind of specialized self-service, it may be updated by a call center call, the user may update that in his Active Directory profile …​ there are many ways how this information is changed. Yet, we want this property to be consistent. We want telephone number to be the same everywhere. We do not care where it was changed. We just want to propagate the last change from anywhere to all the other systems. MidPoint can easily do this. Just specify both inbound and outbound mappings for the same attribute:

In this case, the change in user property telephoneNumber is propagated to the account attribute mobile (outbound change). However, also a change in the account attribute mobile is propagated back to user property telephoneNumber (inbound change). Last change wins.

A clever reader certainly grumbles something about infinite loops now. No need to not worry here. MidPoint can see complete operation context, both inbound and outbound sides. Therefore, midPoint knows when to stop processing the operation. There are even mechanism how to avoid loops caused by connectors detecting changes caused by the connector itself. MidPoint will break those loops automatically.

Mappings and expressions provide a very powerful mechanism. In fact, most of midPoint initial configuration is about setting up correct mappings. However, with great power comes great responsibility, and mappings may look a bit intimidating at a first sight. Fortunately, there are some tips and tricks that make the life with mappings and expressions a bit easier.

Most mappings are aware of the context in which they are used. Therefore, paths of mapping sources and targets can be shortened - or even left out entirely. Activation and credential mappings used in the HR feed example are the obvious cases. Yet, even paths in ordinary mappings may be shortened. For example take the outbound mapping source:

As the mapping knows that its source is a focus (user) the definition may be shortened:

Typical midPoint deployment has tens or hundreds of mappings. Deployments with thousands of mappings are definitely feasible. There are two things that can make maintaining the mappings easier. Optionally, you can specify the mapping name. Mapping name appears in the log files and some error messages. It may be easier to identify which mapping is causing problems, or it may help locate the trace of mapping execution in the log file. It is strongly recommended to provide name for your mappings. Mapping can also have a description. The description can be used as a general-purpose comment or a documentation explaining what the mapping does.

Mappings can become quite complex. There may be multi-line scripting expression in the mapping, it may not entirely obvious what is the input and output. Therefore, each mapping and each expression have an ability to enable tracing using the trace element:

If tracing is enabled, then the mapping or expression execution is recorded in the log files. Tracing can be enabled at both mapping level and expression level. Mapping tracing is shorter. It provides overview of the mapping inputs and outputs. Expression-level tracing is much more detailed.

However, even this level of tracing may not be enough to debug expression code. Therefore, there is a special expression function for logging. Arbitrary messages may be logged by script expression code:

Generally speaking, troubleshooting of mappings may be quite difficult as it is often intertwined with midPoint internal algorithms. Still, there are ways how to do it. The Troubleshooting chapter provides much more details on this.

The systems that midPoint connects to are not created equal. In fact, those systems significantly differ in their capabilities. Most systems can create accounts. However, not all of them can delete accounts. There are systems that keep the accounts forever, the accounts can be only permanently disabled. Yet another systems cannot enable or disable accounts. While most systems support password authentication, other system do not. There is a lot of natural diversity in the provisioning wilderness. The connector may introduce additional limitations as well. Even if target system supports a particular feature, connector may not have appropriate code to use it. MidPoint needs to take all these differences into consideration when executing synchronization and provisioning operations.

MidPoint refers to these features of the systems and connectors as resource capabilities. Although capabilities may look quite complex, they are essentially just a list of things that a connector and resource can do. MidPoint is aware of the resource capabilities, and their limitations. Therefore, midPoint can work with resource data correctly. E.g. midPoint will not try to modify account on a read-only resource.

Capabilities are usually automatically discovered by midPoint, and everything just works out of the box. There is usually no extra work to maintain the capabilities. Yet, sometimes there is a need to tweak the capabilities a bit. Maybe the connector cannot detect resource capabilities well enough. Maybe there is a read-only resource, but the connector has no way of knowing this. In that case, the write capabilities have to be manually disabled in midPoint. For that reason there are two sets of capabilities:

There are many ways to tweak the capabilities by the administrator. Yet, there is one case that is particularly interesting for synchronization and provisioning: simulated activation capability.

MidPoint connectors can be tailored specifically for a particular system. E.g. there are often connectors that are developed specifically for one custom enterprise application. At the other side of the spectrum are generic connectors, that can fit a wide variety of systems and applications. LDAP, CSV and database table connectors are examples of such generic connectors. Such connectors are so useful that they are used in almost every midPoint deployment. However, there is no standardized way to disable an account in database table or a CSV file. Various columns and various values are used to represent account activation status. Quite surprisingly, there is no standardized way to disable an account in LDAP directory either. That is bad news for midPoint. MidPoint takes a significant advantage from knowing whether account is disabled or enabled. We had to do something about this "disable ambiguity". And we did.

There is way to tell midPoint which attribute and what values are used to represent account activation status. Configured activation capability is used for that purpose:

Configured capability above specifies resource attribute active as the attribute that controls account activation status. If this attribute is set to value true then the account is enabled. If the attribute is set to value false then the account is disabled. That is it. Once this configured capability is part of resource definition, then midPoint will pretend that the resource can enable and disable accounts. Attempt to disable account will be transparently translated to modification of active attribute. Moreover, it also works the other way around. If an account has attribute active set to false value, midPoint will display that account as disabled. No extra logic or mapping is needed to achieve that. The capability does it all.

The situation is slightly more complex in our LDAP server in ExAmPLE company. ExAmPLE is using OpenLDAP server, which does not have any reasonable method to disable users. Therefore, we have to be very creative. Perhaps the least bad method is to extend OpenLDAP schema with custom auxiliary object class MidPointPerson which defines new attribute midPointAccountStatus. The midPointAccountStatus takes values enabled and disabled to represent account status. No value of midPointAccountStatus attribute means enabled account as well. As there are two options for representing enabled status, we have to specify that in the definition of simulated capability:

This is not an ideal solution as this MidPointPerson object class is quite cumbersome. However, when combined with some access control list (ACL) magic, it can work quite well.

Previous example demonstrated the use of synchronization for HR data feed. That is the most obvious use of synchronization mechanisms. However, midPoint synchronization can do more tricks than just feeding data to midPoint. Synchronization is frequently used even for target resources. In that case the synchronization is usually used for several purposes:

Older identity management systems used synchronization mostly to get data from the source resources to identity management system. Synchronization in midPoint is much more powerful than that. It can be applied both to source systems and target systems, it can pull data, push data, detect inconsistencies and fix them. Synchronization is a general purpose mechanism, it is a real work-horse of identity management with midPoint. This is the principle of reuse again. Synchronization mechanism can be reused for variety of purposes.

In this example we will be using synchronization to connect existing LDAP server to midPoint. We assume that our midPoint is already connected to the HR system and we have imported the HR data. Now we have midPoint users created for all our employees. Then there is this LDAP server. It is really important LDAP server. This server is used by company enterprise portal and also by a variety of smaller web applications. Those applications are using the LDAP server for user authentication and access authorization. The LDAP server was deployed many years ago. Initially, it was populated by the HR data. However, the LDAP server was managed manually by a system administrator during all these years. Therefore, it is expected that there are some accounts that belong to former employees. Also, it might have happened that some accounts are missing. It is quite likely that a lot of the accounts have wrong data.

First task is to set up the connector for this resource. As LDAP servers are used for identity management purpose all the time, MidPoint comes with a really good LDAP connector. All we need is to set up the resource to use that connector:

What we can see here is a slightly more sophisticated method to reference the connector. So far we have seen only a direct connector reference by OID. This works well for almost all the references in midPoint, because OID never changes. However, connectors are somehow elusive objects. Objects that represent connectors are dynamically created by midPoint when a connector is discovered. Therefore, the OID is generated at random when midPoint discovers new connector. There is no practical way for a system administrator to predict that OID. Yet, we still want our resource definitions to refer to a particular connector when we import the definition. Therefore, there is an alternative way to specify object references. This method is using a search filter instead of direct OID reference. When this resource definition is imported to midPoint, midPoint uses that filter and looks for LDAP connector. If that connector is found, then the OID of that connector is placed in the reference (connectorRef). Therefore, the next time midPoint is using this resource, it can follow the OID directly. This is a very convenient method. However, there are few limitations. Firstly, the filter is resolved only during import of the resource definition object. Which means that it is resolved only once. If the connector is not present at import time, then the reference needs to be corrected manually. Secondly, this approach works if there is only one LDAP connector deployed to midPoint. This is usually the case. However, the connector framework can contain several connectors of the same type in different versions. This is a very useful feature for gradual connector upgrades, testing of new connector versions and so on. Yet, in case that the filter matches more than one object, the import will fail. In that case the connector reference has to be set up manually. However, this method of connector references is very useful in practice, and it is used all the time.

Once we have proper reference to LDAP connector we need to configure the connection:

This is all very similar to the configuration of the other resource that were already presented in this book. It should be quite self-explanatory.

The basic resource configuration above is sufficient to connect to the resource. Therefore, the test connection operation on resource details page should be successful. However, LDAP servers support many object classes and midPoint does not yet know which object class represents account. Therefore, we need to add schema handling section to our resource:

This is pretty much the standard definition as we have already seen. However, there is one new aspect: midPointPerson auxiliary object class. This is an additional LDAP object class which is going to be applied to all object that midPoint eventually creates. It is not important for synchronization at this stage, yet we are going to set it up now, so we have a complete working configuration.

We continue with setting up mappings for the attributes:

There should be outbound mapping for each mandatory LDAP attribute for the inetOrgPerson object class. Such mappings are very typical for a target resource definition.

Once we set up the schema handling, we should be able to conveniently list LDAP accounts in midPoint. However, we need to click on the Reload button. The accounts are stored in the LDAP server and midPoint can access them. However, midPoint have not processed the accounts yet. Therefore, there are no account shadows in midPoint repository yet. Click on the Reload button initiate reading of accounts from the LDAP server.

We are going to import (or reconcile) the resource accounts. However, if we try to do this now, nothing would really happen. The accounts are not linked to users, therefore midPoint would not synchronize the attributes. MidPoint was not told to do anything with the accounts, therefore midPoint does nothing. That is one of midPoint principles: midPoint does not change the accounts in any way unless it is explicitly told to do so. We would rather do nothing than to destroy the data.

Before we can import the accounts, we need to set up the synchronization configuration for this resource. There are accounts in the LDAP server that should belong to users that already exist in midPoint. We want to link them. However, we do not want to do the linking manually. We would rather set up a correlation mechanism that does this automatically. We would like to use LDAP attribute employeeNumber to correlate accounts.

The employeeNumber attribute is configured as correlator by using the correlator configuration element. The employeeNumber attribute is mapped to personalNumber user property, therefore midPoint knows that values of employeeNumber attribute are to be compared with values of personalNumber. When midPoint encounters an LDAP account, it takes value of employeeNumber attribute, transforms it using the inbound mapping defined above, and looks for matching value of personalNumber among all the users.

If correlation values match, then we assume that the account should be linked to the user. In that case midPoint decides that synchronization situation is unlinked (they should be linked, but they are not yet linked). We want midPoint to link the account in this case, therefore we define appropriate reaction:

Unliked accounts get linked. This takes care of accounts for whose we can automatically find an owner by using correlation mechanism. What we should do with other accounts? We will do nothing about them just yet. Therefore, we do not need to define any other reactions. This may be somehow surprising. We do not want illegal accounts in our LDAP server, do we? Then perhaps we would like to see a reaction to delete unmatched accounts, right? That would be a good approach, but it is just too early for this. We do not want to delete unmatched account just now. There may be accounts that are perfectly legal, just the employeeNumber attribute is missing or mistyped. Data errors like those happen all the time, especially when the data were managed manually. We do not want to over-react and start deleting accounts too early. Therefore, we go just with this one synchronization reaction for now.

Now it is the right time to start import or reconciliation task for LDAP resource. After the task is finished the situation may look like this:

It looks like we have quite a good data in the LDAP server. Most of the accounts were successfully correlated and linked to their owners. Yet, there are few accounts that were not correlated. Those accounts ended up in unmatched situation. You can resolve this situation by manually linking the unmatched accounts to their users. Simply click on the small triangle button next to the unmatched entry and select Change owner from the context menu. Then select the right user (Isabella Irvine) in the dialog that appears. After that the account is linked to the user. Repeat this process to link all unmatched accounts.

There is one interesting thing in the screenshot above. Have a look at the LDAP account identified by uid=carol. While most other accounts have their uid value taken from the surname of the user, this account is an exception. Even though the uid value is obviously wrong, midPoint have linked the account to the correct user (Carol Cooper). The reason is that we have set up midPoint to use employeeNumber for correlation. Even accounts whose usernames violate the convention can be automatically linked to their owners - as long as there is any reliable piece of information that can be used for correlation.

Search panel on top of account list can be used to make manual linking faster. The Situation search control can be used to select accounts that are in Unmatched situation.

We can quickly handle the obvious unmatched accounts, and link them manually. When all the accounts are linked to their owners, we end up with two accounts that do not have owners: john and oscar. In identity management parlance, these are called orphaned accounts. While having a closer look at the accounts, these two accounts obviously belong to former employee. John and Oscar do not work for ExAmPLE company for several years. While John’s account is disabled at the very least, Oscar’s account is still active. What a deprovisioning disaster this is! We have illegal account here, still active, still accessible by an employee that might not be entirely happy to be laid off. Quite obviously, we have to do something about it.

This is the right time to complete the synchronization policy. Once the correlation is complete and all accounts have an owner, detecting an unmatched account means that an illegal account is created in LDAP server. Now we can tell midPoint to delete any unmatched accounts.

When reconciliation task is completed, all orphaned accounts are gone. We have reduced our security risks, which is one of the primary reasons for having identity management system in the first place.

We have our accounts partially cleaned up now. All accounts are linked to owners and orphaned accounts are gone. However, there may be some accounts in the LDAP server that have wrong attribute values. By "wrong" we mean that the attributes have different values than the values that are computed by the outbound mappings. However, midPoint is not correcting those values just yet. Remember the midPoint principle that it does not change the account unless we have explicitly told to do so? Those accounts are in the linked situation, and we have not configured any reaction for this situation. Therefore, midPoint did nothing. Now we need to tell midPoint to synchronize the values:

A clever readers is now surely wondering whether we have forgotten something. We have, indeed. Attribute values are synchronized by running reconciliation process. However, our outbound mappings will not work in reconciliation. They do not have any explicit definition of strength, therefore midPoint assumes normal strength. Those mappings are supposed to implement the last change wins strategy. Therefore, reconciliation cannot overwrite the account data, as midPoint does not know whether it was account attribute or user property that was the last to change. If midPoint is not sure about something, then it assumes a conservative position and does nothing. We do not want to destroy the data. Therefore, what we need to do now is to let midPoint know that we really mean it, that the mappings are really strong:

Clever reader is uneasy once again. What is this limitations thing here? Simply speaking, the limitations specify that the attribute is optional (minOccurs=0) and that it is single-valued (maxOccurs=1). However, isn’t midPoint supposed to be completely schema-aware and figure that all by itself? Yes, it is, and midPoint does it all right. In fact, that is the reason why we need to override the information from the schema using this limitations element here. The cn attribute is specified in LDAP schema as a mandatory attribute. However, we have just specified outbound mapping for that attribute. Even if no value for attribute cn is entered in midPoint user interface, we can still determine that value by using the expression. Therefore, even though LDAP schema specifies attribute cn as mandatory, we want to present that attribute as optional in midPoint user interface. Hence the minOccurs limitation. The maxOccurs limitation is immediately obvious to anyone who is intimately familiar with LDAP peculiarities. In the LDAP world, almost everything is multi-valued by default. Even commonly used attributes for account identifiers and names are multi-valued. Nobody is really using them as multi-valued attributes, because vast majority of applications will probably explode if they ever encounter two values in the cn attribute. Yet, those attributes are formally defined as multi-valued in LDAP schema, and that is what midPoint gets from LDAP connector. The maxOccurs limitation is overriding the schema, and forcing midPoint to handle this attribute as if it was single-value attribute.

The last thing that we need at this point is to handle activation. We need a very simple mapping for that:

The mapping is entirely empty (<outbound/>), as it completely relies on default settings. Source is default ($focus/activation/effectiveStatus), target is default (activation/administrativeStatus of account), also the expression is default (asIs). All midPoint needs to know is that we want such mapping at all.

However, there is one more detail to set up. As LDAP does not have any standard reasonable account disable mechanism, we have to tell midPoint which method are we using for this particular case. We do that using configured capability, as we have already seen before:

From this point on we can keep reconciliation task running periodically by setting task schedule. Our synchronization policies are in good shape, ready to be continually enforced. Periodic reconciliation runs keeps account attributes updated. Even more importantly, it keeps us protected from serious security risk posed by orphaned accounts.

That is all. Now you can schedule reconciliation tasks to keep an eye on the LDAP server. The task corrects any attribute values that step out of line and deletes any illegal accounts. This is how synchronization tasks can be useful, even in case of pure target resources.

However, there is one last word of warning. Those accounts were synchronized and linked to existing midPoint users. The accounts were not created by midPoint. Therefore, there is nothing in midPoint that would say that those accounts should exist. In midPoint parlance, there are no assignments for those accounts. MidPoint makes clear distinction between policy and reality. Therefore, midPoint is aware that those accounts exist, but there is no policy statement that would justify their existence. By default, midPoint does nothing, and it lets the accounts live. The accounts are created or deleted only if there is an explicit change in the assignments. There is no such change now, therefore the accounts are not deleted. However, this is quite a fragile situation. Accounts that are linked but not assigned can easily get deleted if midPoint administrator is not careful. Of course, there are methods to handle such situations. One way would be to create the assignments together with the links. Those that are interested in this method should look up keyword "legalize" in midPoint docs. However, there are much better methods to handle such situations. Perhaps the best approach would be to utilize the roles (RBAC). This is the topic of the Role-Based Access Control chapter later. Yet, there are still more things to learn about synchronization until we get there.

Reconciliation is a process of comparing current state of an account (reality) to a desired state of the account (policy). Reconciliation does not only compare the accounts, it is actively correcting the inconsistencies. Reconciliation can correct wrong data on resources (outbound direction). Yet, it also works the other way. It can correct the data in midPoint (inbound direction). Therefore, reconciliation is one of the most useful tools in the identity management toolbox.

Reconciliation can be used in a variety of ways. Reconciliation can be initiated for one specific user by using midPoint user interface. In that case, midPoint compares the values of all user’s accounts to the values that were computed using the mappings. If there are differences, midPoint corrects account values. This approach is perfect for testing reconciliation setting using just a single user. This feature is also useful for fixing values of one specific user.

Reconciliation of a specific user may be useful, but it is an ad-hoc approach. We usually favor systemic approaches in identity management. Therefore, reconciliation is usually used in a form of a reconciliation task. Reconciliation task lists all the accounts on the resource, and then it reconciles each account, one by one. This is a way to keep all resource accounts continuously synchronized.

There is a couple of things regarding reconciliation that can be somehow surprising. Firstly, reconciliation of an account may cause modification of a user. This happens if there are inbound mappings for that account. This is perhaps quite expected. However, the operation does not stop there. If a user is changed, then such change may propagate to other accounts on other resources, usually by the means of outbound mappings. MidPoint does not like procrastination, therefore it will try to apply all the changes immediately. It means that reconciliation of one account may cause changes to other accounts. Which makes a lot of sense, yet it may be quite surprising. Secondly, reconciliation skips all normal-strength mappings. We have already explained the reasons for that, but this is something that can surprise even an experienced midPoint engineer from time to time. If we are sure that we want the mapped value to be present in the account all the time, then strong mappings are the way to go.

A curious reader that has already explored midPoint user interface has surely noticed recompute function. To the untrained eye, recompute looks almost exactly the same as reconciliation. However, there are subtle differences. Firstly, recompute does not force fetching of account data. When recomputing, fresh account attributes are not retrieved from the resource, unless midPoint inevitably needs them for the computation. This usually happens if weak or strong mappings are used, in that case the attribute values are retrieved. However, if there are normal mappings only, then recompute may skip retrieval of fresh account data. MidPoint compares and corrects account attribute values only for those accounts that are retrieved from resource during this process. That is how recompute works. Correcting account data is more or less just a side effect of recompute. The purpose of recompute is to correct data of midPoint users, which means evaluation of object templates and other internal policies.

On the other hand, reconciliation always tries to read all the accounts regardless whether they are needed for computation or not. Therefore, all the attributes on all the accounts are corrected. That is the purpose of reconciliation: correct the account data.

There is yet another difference between recompute and reconcile tasks. The purpose of recompute task is to correct user data. Therefore, recompute task iterates over midPoint users. Recompute task does not detect new accounts on the resource, and it may even overlook if an account is deleted. However, reconciliation task is different. In fact, reconciliation task has several stages. Main reconciliation stage lists all resource accounts. It determines owner of each account, compares the attributes and corrects them. As this process iterates over real accounts on a resource, it can easily detect new accounts. When the main stage is completed, the next phase looks at account shadows stored in midPoint. The task looks for shadows that have not been processed in the main phase. Those are accounts that used to be on the resource some time ago, but have disappeared since. That is how reconciliation detects deleted accounts.

Rule of the thumb for the reconciliation versus recompute dilemma is this:

Usernames take many forms. Some people like usernames based on first names (john), others prefer last names (smith), many practical solutions end up using a combination of both (jsmith). Some organizations prefer immutable identifiers, such as employee numbers (1458363), which is not entirely favored by users. We will discuss that later, in Focus Processing chapter. Whatever username convention we are going to choose, we have a problem in our current ExAmPLE midPoint deployment: midPoint has usernames based on employee numbers (001), while LDAP has usernames based on names (anderson). We should use one or the other, not both of them at the same time.

Clean, simple and systematic solution would be to force usernames based on employee numbers from midPoint to LDAP. This is very simple to do, all we need is to change strength of dn mapping from weak to strong. Next reconciliation run would change all the usernames in LDAP to match values of user name property in midPoint. However, this may not be entirely desirable. This process changes all usernames in the LDAP directory, changing their usernames (uid), but also distinguished names (dn) in the directory. There are usually other applications connected to the directory, as that is the reasons for directories to exist. Application accounts are almost always bound to either uid or dn in LDAP directory. Strictly speaking, there is immutable entryUUID attribute in OpenLDAP directory, which is almost ideal for account binding. Of course, almost no application is using entryUUID, they all rely on uid or dn. If uid and dn are changed, account bindings in applications may be lost. Users may lose their settings, preferences, user profiles, access to data and home directories - and they are not going to like that. Not to mention the simple fact, that they are not going to like the new numeric usernames in the first place.

It may be a much better idea to keep existing LDAP usernames, and change usernames in midPoint to match them. Fortunately, this is also very easy to do. All we need is one inbound mapping:

Once reconciliation run is completed, LDAP usernames are applied to midPoint users. Users have user-friendly (although slightly non-consistent) usernames now.

It is a good idea to remove the inbound mapping after midPoint usernames are corrected. We do not need it any more, and it can be quite confusing or even harmful in the long run. We probably want to generate usernames in a systematic and consistent way from now on, which will be described in Focus Processing chapter.

It is also a good idea to disable inbound mapping from empno attribute to user’s name in the HR resource. If we would leave this mapping active, it can ruin our usernames by replacing them with employee numbers from HR system. Even worse, as we have configured outbound mappings to LDAP directory, change of user’s name is reflected to LDAP uid attribute, effectively renaming all LDAP accounts. We definitely do not want that! Disabling the inbound HR mapping leaves us in a situation in which there is no mapping for user’s name at all. This means creation of new midPoint user from new HR record is going to fail. We have to live with that now, as we are not going to risk destruction of all our usernames. We will fix that later, when we learn how to generate proper usernames.

Reconciliation is really useful mechanism. It is reliable and thorough. However, it is also quite slow, and it consumes o lot of computational and network resources. There are good reasons why reconciliation is such a heavyweight beast. Reconciliation works with absolute state of accounts. It means that reconciliation is reading all the accounts with all the values of all relevant attributes. Then it recomputes everything. Even the attributes and values that were not changed are recomputed. This is a very reliable way of computation, and it is also the method used by the majority of traditional identity management systems.

Yet, there is also a better way. If we know that just one attribute was changed, we can recompute that single attribute only. We do not need to care about other attributes. Moreover, if we know that attribute foo has changed in such a way, that there is a new value bar, then it gets even better. We just need to recompute the value bar, and we need not care about any other values. This is what we like to call a relative change. We care just about the values that were changed. That is how midPoint works internally. We could say that MidPoint is relativistic system.

This is where delta comes in. Delta is a data structure that describes the change of a single midPoint object. Add delta describes a new midPoint object that is about to be created. Modify delta describes existing midPoint object where some properties have changed. Delete delta describes an object that is going to be deleted.

This is a very powerful mechanism. Just remember that everything in midPoint can be represented as an object: user, account, resource, role, security policy …​ everything. Therefore, delta can represent any change in midPoint. It may be a change of user password, deletion of an account, change of connector configuration or introduction of a new policy rule. If all the changes can be represented in a uniform way, then they can also be handled in a uniform way. Therefore, it is easy for midPoint to record all the changes in an audit trail – including configuration changes. It is easy to route any change through an approval process. And so on. MidPoint can create a relatively simple mechanisms to handle changes, and then those mechanisms can be applied to any change of (almost) any object.

Let’s have a closer look at an anatomy of a delta. There are three types of delta: add, modify and delete. Add delta is quite simple. It contains a new object to be created.

Delete delta is even simpler. It contains just object identifier (OID) of an object to be deleted.

Last one is modify delta. This delta contains a description of modified properties of an existing object. As the object can change in a variety of ways, modify delta is the most complex of the three. Modify delta contains a list of item deltas. Each item delta describes how a particular part of an object changes. For example, following delta describes that a new value pirate is added to a user property employeeType.

The item delta may have three modification types: add, delete and replace. Add modification means that new value or values are added to an item. Delete modification means that value or values are removed from an item.

In both add and delete cases, the values that are not mentioned in the delta are not affected. They remain unchanged as they are. However, replace modification is different. This means that all existing values of the item are going to be discarded, and they are replaced with the value or values from the delta.

The deltas are designed to work with both single-valued and multi-valued items. In fact, add modification and delete modification deltas are specifically designed with multi-value items in mind. Those deltas can work efficiently even in cases that there is a multi-valued attribute that has a very large number of values. There is a good reason for this. Multi-valued properties are quite common in the identity management field. Just think about how roles, groups, privileges and access control lists are usually implemented. Everyone who ever managed a large group in LDAP server certainly remembers that experience in vivid colors. Fortunately, midPoint is designed to handle situations like these.

Everything in midPoint is designed to work with deltas: user interface, mappings, authorizations, auditing …​ all the way down to the low-level data storage components. Mappings work in a relativistic ways. That is one of the reasons why we need to explicitly specify sources of the mapping. Mapping source definitions are matched with items in the delta to control execution of the mapping. Deltas permeate entire midPoint computation. Deltas are input to the mappings, and mapping produce other deltas as output. Therefore, we can have a complete chain: deltas that are result of inbound mappings is applied to the user object, but those deltas are also input to outbound mappings. Everything is relativistic in midPoint.

This might seem to be a bit over-complicated at the beginning. But do not worry. You will get used to it. Clearly, this approach has major advantages.

However, a clever reader does not seem to be impressed. How can this relativistic approach conserve any significant portion of computational resources? We usually fetch the entire account from the resource anyway. Therefore, there is no harm to recompute all the attributes. The computation itself is fast, it is the fetch operation that is slow. Isn’t it? The clever reader is, of course, right - or rather partially right at the very least. Most resources indeed fetch all the account attributes in a single efficient operation. For those cases, there is no big increase in efficiency if we go with the relativistic methods. However, there are exceptions. For example, some resources does not return all the values of big attributes, e.g. all the members of a large group. Additional requests are needed to fetch all the values – and there may be a lot of requests if the group is really large. Relativistic approach has significant benefit in those cases. The benefits will be even more obvious when we get to the live synchronization in the next section. Yet, performance is not the primary motivation for the relativistic approach. There is one extremely strong reason to be relativistic: data consistency. Consistency is something that brings ugly nightmares to many engineers who try to design distributed system. Identity management solution is a distributed system, managing data in many independent applications and databases. It is also a very loosely-coupled distributed system. There is no support for locking or transactions in the connectors. Even if there was some support, the vast majority of resource cannot provide those consistency mechanisms on their identity management APIs. This means that midPoint cannot rely on traditional data consistency mechanisms. MidPoint cannot make sure that the data are not changed during computation or between computations. That is why relativistic approach is so useful. Relativistic computation has a very high probability of achieving correct result even without locking or transactions. This is more than acceptable for typical identity management deployments. For those rare cases where relativistic computation can fluctuate, there is always reconciliation as a last resort. Yet, thanks to the relativistic nature of midPoint, the need for reconciliation is significantly reduced.

That was a lot of long words, but clever reader seems to be satisfied now. At least for a while. For the readers that are still scratching their heads, there is quite a simple summary: relativistic approach of midPoint can do miracles. For example, midPoint resource can be both sources and targets, even a single attribute can be both source and target of information. It is the relativistic approach that enables configurations like this. The principle of relativity is relatively simple. Yet, its effect in midPoint is nothing short of being revolutionary.

MidPoint has a range of synchronization mechanisms. Slow, brutal but reliable reconciliation is at one end. Live synchronization is on the other. Live synchronization is a lightweight mechanism that can provide almost-real-time synchronization capabilities. Live synchronization is looking for recent changes on a resource. When such changes are detected, live synchronization mechanisms process those changes immediately. The synchronization delay is usually in order of seconds or minutes, provided live synchronization is used properly. Unlike reconciliation, live synchronization is not triggered manually. That would make very little sense. Live synchronization works in a long-running task, repeatedly looking for fresh changes in short time intervals.

If a resource is already configured for synchronization, then all that is needed to run live synchronization is to set up a live synchronization task. MidPoint user interface can be used to do that easily. An example of live synchronization task was provided in the HR Feed section above. Live synchronization task wakes up at regular intervals. Each time the task waves up, it invokes the connector. Connectors capable of live synchronization have special operation that is used to get fresh changes from the resource.

The connector can support any reasonable change detection mechanism – in theory. Yet, two mechanisms are commonly used in practice:

Those mechanisms are called live synchronization strategies, and they are further explained in following section.

All live synchronization methods need to keep the track of what changes are "recent", i.e. which changes were already processed by midPoint and which were not processed yet. There is usually some value that needs to be remembered by midPoint: timestamp of last scan, last sequence number in the change log, serial number of last processed change and so on. Each connector has a different value with a connector-specific meaning. MidPoint refers to those values as "tokens". The most recent token is stored in the live synchronization task. That is how midPoint keeps track of processed changes. There are (hopefully quite rare) cases when resource and midPoint token get out of alignment. This may happen in cases such as the resource database is restored from a backup, if network time gets out of synchronization and so on. If that happen, then deleting the token from the live synchronization task is usually all it takes to get the synchronization running again.

Live synchronization is fast and very efficient. However, it is not entirely reliable. MidPoint may miss some changes. This is quite a rare situation, but it may happen. Reconciliation can remedy the situation in such a case. Just remember, all the synchronization mechanism share the same configuration. It is perfectly acceptable to run live synchronization and reconciliation on the same resource at the same time. Of course, it would be a good idea to run reconciliation less frequently than live synchronization.

Live synchronization is fast and efficient - in theory. However, as usual, the devil is in the details. There is no one single synchronization protocol or standard that would work for all the resources. Every system has its own way to synchronize data. Some systems (such as LDAP servers) even have several mechanisms to choose from. Then there are source systems that have no practical way to implement efficient synchronization. We would refer to such methods as live synchronization strategies.

Synchronization strategy is configured on connector level, and the details should be, theoretically, hidden inside the connector. MidPoint would not know and would not need to know what synchronization strategy is used. That might work in an ideal world. Yet, we live in a practical world, and there are many details that leak through the connector interface.

Let us use LDAP as an example. LDAP is, theoretically, a standard. However, the standard does not specify any synchronization mechanism. There is experimental RFC 4533, which is not widely adopted. Yet, synchronization capabilities are necessary, and every major LDAP server provides some synchronization mechanism. Some mechanisms are quite good, some are not. There is an ancient "Retro change log" mechanism, going back to Netscape/iPlanet LDAP servers originating in the 1990s. That mechanism is still used today, in several variants. Active Directory, in a very typical way, has its own "DirSync" synchronization mechanism. OpenLDAP has yet another mechanism based on the access log. There is this RFC 4533 standard, which is used so rarely, that there was no request to implement it in midPoint LDAP connector. Then there is a catch-all synchronization mechanism that looks for recent changes based on modifyTimestamp attribute.

In theory, all the synchronization strategies above should be equivalent - but they are not. For example, some variants of "Retro change log" synchronization cannot reliably detect rename operations. There may be problems with delete operations as well, especially if coupled with rename operations. Almost every mechanism has its quirks. Then there is the modifyTimestamp, which is the most problematic of all.

Unfortunately, it is quite common practice to use a synchronization strategy based on last modification timestamp. Not just for LDAP, but also for database tables and other types of source systems. This is perhaps understandable, as this is a very simple mechanism. However, it has a lot of problems. The obvious problems can be caused by de-synchronized time on network, although in the age of Network Time Protocol (NTP) this should not be a problem at all. The other problem is a timestamp granularity. If the timestamp is granular to one second, that can be a big problem. One second is a very long time for a computer. A lot can happen in one second. Therefore, the connector has to include the "boundary" second to both consecutive synchronization runs, which means that the records may be processed twice. Going for millisecond granularity makes the problem less severe, but the problem is still there.

However, the worst problem is that this strategy cannot detect deleted objects. Deleted objects are not there anymore, they do not have last modification timestamp, therefore they will not be included the search. This means that there must be a reconciliation process running together with live synchronization. Wait a minute, it is usually recommended running reconciliation anyway, as a form of "safety net", isn’t it? It is, but the difference is in the timing. It is one thing to run reconciliation once a week to make sure that no records were missed. Yet, it is a completely different thing to run reconciliation every hour to make sure deleted objects are properly handled. This makes a huge difference, especially for deployments with millions of entries. Strategies based on last modification timestamp may look like a good idea at the beginning. However, they usually turn into a major liability in the long run. Avoid them if you can.

The bottom line is, that synchronization strategies are not created equal. In fact, the individual strategies tend to have vastly different characteristics. Our advice is to learn how each synchronization strategy works, what are the limitations and when it fails. Also, avoid the use of strategies based on last modification timestamp if there is any other viable alternative.

Synchronization is one of the most important mechanisms in the entire identity management field. Primary purpose of synchronization is to get the data into midPoint. That is good approach when an identity management deployment begins: feed your midPoint with data first. Get the data from the HR system. Correlate the data with Active Directory. Connect all the major resources to midPoint and correlate the data again. MidPoint does not need to make any changes at this stage. In fact, it is perfectly good approach to make all the resource read-only at this stage. The point is to let midPoint see the data. Why do we need that?

This is a good start. Even if this is all that you do in the first step of the deployment, it is still a major benefit. You can get better visibility, and with that comes better security. You have the data to analyze your environment, and plan next step of the identity management deployment. You won’t be blind any longer. That is extremely important. It is indeed a capital mistake to theorize before one has data.

MidPoint is designed as a schema-aware system, from the bottom to the top. MidPoint has a definition for every bit of data that passes through it. We know whether a particular piece of data is string, integer or timestamp. We know whether it is single-valued or multi-valued. We know whether it is optional or mandatory. We know whether this is a sensitive piece of data that requires extra protection. We know whether it is part of technical meta-data that we usually do not want to show by default. We usually also know what label we should use when we are presenting the data, and how that label translates to other languages. We know quite a lot about the data that we work with. All the objects that midPoint works with are completely defined by the schema. There is a schema for user, role, org, resource, system configuration and everything else.

Such awareness of the schema brings significant advantages to midPoint. The most obvious advantage is in data presentation. We know that we need to render a calendar selector because that particular data property is timestamp. We know that we need to render a text field with a plus button to add values because that particular property is a multi-valued string. We know that some fields should be disabled because those properties are read-only. This behavior is not hard-coded in the user interface code. Vast majority of midPoint user interface is rendered by interpreting midPoint schema.

This approach is absolutely crucial for any serious data management system to operate efficiently, doubly so for identity management. One of the reasons is that the identity management system works with data that are retrieved from other systems (resources). It is not realistically possible to hard-code midPoint user interface for all the various attributes that all the possible resources could have. A different strategy is needed here, a strategy that is much more dynamic.

When midPoint connects to a new resource for the first time it attempts to retrieve resource schema. The resource schema specifies what object classes the resource supports, which attributes the object classes have, what types are those and other details about the data model of the resource. MidPoint transforms this resource schema to its own native format, and stores that in the resource definition. This means that midPoint has the schema available anytime it is needed for dynamic interpretation. That schema is used to display resource data in the most natural and user-friendly way. It is also used by automatic data type conversions, which makes configuration of mappings easier.

MidPoint schema is not just a nice way to describe user, role or organizational structure. It has a much deeper meaning. The primary purpose of a schema is integration, data translation and unification. A clever reader would certainly remember that we have already talked about star topology or hub-and-spoke integration pattern. MidPoint is like a hub of the wheel and all the resources connect to midPoint as spokes. MidPoint is actively discouraging direct resource-to-resource communication. Everything in midPoint is built for resource-to-midPoint and midPoint-to-resource communication. MidPoint is always the center – for a very good reason. All resource data need to be translated to and from midPoint "data language". Thus, midPoint creates a common language that everybody can understand. This is the very purpose of midPoint schema. The schema of user, role, org and service is designed to contain properties that are often used in identity-related integration scenarios. Therefore, an engineer who is designing a mapping is quite likely to find a suitable property in midPoint schema that is prepared to be used.

MidPoint schema forms a lingua franca, a common language that can be translated to various data dialects used by the resources. Even more than that, it also provides a basic framework that can be reused for many midPoint deployments. Therefore, an engineer starting a new deployment does not need to start on a completely green field. The basic schema will always be there to provide a starting point.

When it comes to identity management field, there is one concept that is at the center of everything: concept of user. User is undoubtedly the most important object in the entire midPoint schema. Therefore, it is worth to have a closer look at how this object looks like. This is going to be a really educative lesson, as it will explain several fundamental principles of midPoint.

User is represented by schema datatype identified as UserType. Adding the Type suffix to data types is a common convention in midPoint, there are UserType, RoleType, OrgType, ResourceType and so on. This convention is partially historic, partially given by XML Schema conventions, partially a convenience to developers. Regardless of the origins, this convention is used for all the data types in midPoint schemas. You will get used to it eventually.

UserType is what we call an object definition in midPoint parlance. This means that UserType data structure specifies a complete midPoint object with all the things that any self-respecting object needs. There is object identifier (OID), name that can be presented in different forms and languages, free-form description and so on. All midPoint objects have those things.

The UserType data structure has many additional properties, containers and references. Property is a primitive data item such as string, integer or a timestamp. Container is a complex data structure that contains a bunch of properties or other containers. Reference is a pointer to another midPoint object.

Definition of UserType is summarized in the following table:

This is a basic outline of the schema for UserType. This description is slightly simplified. Not all the items that are defined for UserType are shown in the table above. Deprecated items are not shown at all. Only some operational properties are shown. Some items are simplified or entirely omitted for clarity.

Following example illustrates the use of midPoint UserType schema:

Most of the items in midPoint schema are quite ordinary and they behave as expected. Such as the fullName property. The property can be set and changed by using midPoint user interface. However, then there are some extraordinary items. Those are automatically determined and controlled by midPoint core engine. Those items are essential for correct operation of midPoint. Therefore, they are called operational items. Operational items are usually not directly displayed in the user interface. They are either completely hidden, displayed indirectly or displayed only when user chooses to display them.

MidPoint schema has grown and evolved over time, and it is still evolving. Therefore, it is quite expected that the schema will slightly change over time. However, we do not want to affect midPoint deployments by incompatible schema changes in every midPoint release. Therefore, items are usually not removed from midPoint schema without a warning. An item that we plan to remove is marked as deprecated first. At that point, such item is still working as it was working before. However, it is not displayed in the user interface, to discourage use of that item. Deprecated items are removed in one of the subsequent midPoint releases. This gives enough time for midPoint users to adapt to schema changes.

There is also another kind of schema evolution. Development of most midPoint features is quick and straightforward. Then there are features that are quite complex or features that involve some degree of exploration. Those features cannot be implemented in a single midPoint release. There are also features that are provided to the midPoint community as a "preview", to gather feedback for further development. All such features are marked as experimental. Those features are not officially supported, but you are free to use them at your own risk. Most new features require extensions of midPoint schema. This is also true for those experimental features. However, when going experimental, there is a fair chance that something will change in the future. Therefore, we are explicitly marking parts of the schema as experimental. This is a warning that those parts are likely to change. We are not promising any kind of compatibility for experimental parts of midPoint schema. They may change any time, they may even completely disappear. There will be no deprecation or any other warning. Simply speaking: if you are dealing with experimental features, you are completely on your own. Do not come crying when those things stop working. You have been warned.

Time is cruel, everything that we do is in some way temporary. Except perhaps for stupidity, which seems to be utterly endless. Sadly, all other things have a beginning and an end. Employees have hiring date, contracts have end dates, users can be deactivated, roles may get replaced and so on. We use the terms lifecycle and activation to encompass all those things that deal with the questions of digital life and death of the objects.