Approach

Last modified 09 Aug 2024 14:01 +02:00
Table of Contents

MidPoint has a lot of unique features. But what is perhaps even more interesting than the features is the approach to midPoint product management, planning, design and development. We tend to follow a couple of simple but very powerful principles:

Deployment and Maintenance Efficiency

The very most important principle of midPoint is: it has to be efficient to deploy and maintain. Our team has been working with identity management products for more than a decade. We have suffered. Therefore we know how an IDM system should work. And we know very well that it is the deployed and working system that matters. Not the ideas, not diagrams, not the latest and greatest Java fashion - it is only the result that matters. It has to be affordable to deploy. Not only by cost of licences (which is zero) but especially by the cost of effort to deploy it. This effort is huge for deployments of traditional IDM systems. But midPoint’s goal is to make this effort reasonable.

We achieve efficiency by reuse and standardization. If some feature is frequently used in the IDM deployments then it needs to be a feature of the product. The deployer needs to flick couple of switches instead of writing thousands of lines of code. The difficult part is to realize which features are often used in IDM deployments. But we have been deploying IDM solutions for a decade and we know this quite well. We also listen to our customers and partners. In fact the feature set of last few releases was heavily inspired by the feedback from our partners.

What exactly the efficiency means in midPoint? It means this:

  • IDM means integration and often everything relates to everything. MidPoint simplifies configuration and maintenance by reusing the schema through the entire system. E.g. a connector can fetch account definition from a resource (e.g. LDAP inetOrgPerson object class). MidPoint will process it, parse it and automatically generate schema definition for itself. This definition is then used to check for attribute validity. It is used to automatically figure out correct data types in mappings. The schema is also used for automatic conversion of data types if they do not match (e.g. automatic conversion of date and time values). The schema is also used to correctly present data in midPoint user interface. Therefore midPoint can correctly show how an LDAP account really looks like - automatically, out of the box. No form or GUI customization needed.

  • IDM systems often need to pre-create an account before user actually starts the work. But more frequently the IDM solution contains a delayed actions, such as "delete this account 30 days after user is fired". Traditional IDM system require complex interplay of triggers. processes and deferred actions to implement it. MidPoint needs just correctly defined mapping time constraints and then it automatically takes care of all the messy management of time and tasks.

  • Frequent requirement of IDM solution is to disable an account instead of deleting it. Traditional IDM systems require complex customization to do this. MidPoint needs just correctly defined activation mappings to do this.

  • …​ and much more.

Our mantra is the Pareto principle: 20% effort, 80% effect. With midPoint you should have 80% of the solution very quickly. This usually means days or weeks, not months or years. The remaining 20% can also be achieved with midPoint. It may require a slightly more of sophistication, maybe some scripting or even a bit of programming, but it is definitely achievable and feasible. And actually even this extra 20% is much more efficient that was the case in traditional IDM systems. It has to be. This is the reason why we have created midPoint in the first place. We have created it for ourselves. For IDM engineers.

Separation of Concerns

Identity management is a complex field full of hard problems. It is perhaps even beyond the capacity of any sane person to understand it all in its entirety. But even though it cannot be efficiently tackled as a one huge lump it still can be efficiently handled. The separation of concerns principle can be used to divide big hard problems into smaller ones. This principle is used very frequently in midPoint. We try to design simple and efficient mechanisms that are good for one facet of the problem. Then we combine them together to solve complex problems. The mechanisms are relatively simple and therefore they are easy to master. Yet, when used together they can create very sophisticated configurations.

Few examples of the use of the separation of concerns principle are:

  • The complex problem of synchronization is divided to several steps (inbound, template, outbound) that are more or less independent.

  • Each of the synchronization steps is also using simpler mechanisms, such as mappings which handle the transformation of attribute values.

  • The processing of synchronization actions is independent from both steps and also mappings inside them.

  • Mappings are using expressions to transform individual values. The mapping does not care (and does not know) which specific type of expression or scripting language is used.

  • and so on …​ it’s almost everywhere

The individual partial mechanisms can then be combined into a very complex and powerful configurations. The somehow extreme examples are perhaps meta-roles and higher-order inducements. Please have a look at OrgSync Story Test description to get an idea how complex configuration can be created by using just a handful of mechanisms.

Reuse

MidPoint has to be efficient to deploy and maintain. But it also has to be efficient to understand and develop. Therefore we are re-using a huge amount of concepts and mechanisms all over the midPoint. Reuse has many advantages. The mechanisms needs to be developed once, understood once, tested once - and the used at many places. This approach gives us huge effect while keeping the cost down at the same time.

Perhaps the best examples of reuse include:

  • Mappings and Expressions are used in many places in midpoint: Inbound Mapping, Outbound Mapping, Object Template, Correlation and Confirmation Expressions and we even use expressions inside expressions.

  • Synchronization policy is configured just once. Then the same policy is used by all the synchronization flavors such as live synchronization, reconciliation or discovery. It does not matter how a change was discovered. The reaction will always be consistent.

  • Roles and object templates can be applied to numerous types of objects, not just users. E.g. we can have roles that are applied to organizational units or even to roles itself (creating meta-roles).

  • The principle of outbound mappings is used both in resource definitions and in roles. The principles are the same.

  • Organizational structure can be reused for many purposes: formal functional organizational structure of a company (divisions, sections), project-based structure, ad-hoc teams, multi-tenancy, even for a simple groupings of objects for the purpose of delegated administration. Or all of that at once. The same principles are used.

  • Deltas are used mostly for normal computation of changes (e.g. in mappings). But they are also stored in the role requests that wait for an approval. Deltas are stored in shadows while they wait for a target resource to come back online. Deltas are used to re-start an operation after resolving an incosistency.

  • Roles are used mostly for provisioning. But the same roles are also used to hold internal midPoint authorizations. Therefore any kind of role approval process can also be applied to midPoint internal security. This also means that provisioning policies and internal midPoint policies can be defined in the same roles which simplifies the configuration. And there is one more reuse: the organizational units are considered to be also roles. So it is easy to assign an account, group or midPoint authorization to all members of an organizational unit.

Everything Changes

IDM solution is not a piece of software frozen in time and space. Quite the contrary. Everything changes in IDM deployments: data, requirements, processes, environment - even the software itself. The ability to handle this dynamic whirlwind is a critical feature of midPoint. Not just the software part of midPoint but the whole midPoint solution which also includes support services, partners, planning, and lot of cooperation. But more about that later.

When it comes to the software itself it has to handle changes in data. You might think that this is trivial topic - the computers do this all the time for decades. But it is lightyears from being a trivial problem. IDM solutions live in dynamic, volatile and heterogeneous environment. And during more than a decade of commercial identity provisioning system almost nobody got this right. What’s the problem? It is both simple and complex: consistency.

The consistency is an achilles' heel of Identity Management. Identity management solutions integrate data from many sources and deliver them to even more targets. It is no exception for an IDM solution to have tens of data sources and thousands of targets. The hallmark of provisioning system is that they do not disable the native system administration tools. Given hundreds of entities it is almost sure that some data will get out of sync: the data will get inconsistent. IDM solution thinks that it is A, the remote system thinks that it is B. And this is where traditional IDM system fail miserably. Of course, this can be handled by brute-force methods such as reconciliation. But there are still major issues: It is painfully slow and inefficient. And even more importantly: somebody needs to decide whether it should be A or B in the end.

We have studied this problem in depth. We have fought with it while deploying traditional IDM solutions. We have explored it from the software architecture point of view. We have conducted academic research. And we think that we have a solution. Or rather a set of solutions:

  • MidPoint is using relative change model. Every change is represented as delta. Deltas can be easily merged, stored for a long time and re-applied and they can be processed and re-computed efficiently. For example, delta can be stored for days or weeks while a role approval request waits for an approved to get back from vacation. A delta can be used to recompute just the values that have been changed in the source and leave the other values untouched. The delta-based design is not just an afterthought. It is an integral part of midPoint design and architecture from the day 0.

  • MidPoint is essentially a lock-free system. Remember the times when you must wait until one role approval finished before starting a new one? How users were not able to work for days because some approval of a petty role blocked out a request for a really important role? That won’t happen in midPoint. The deltas can be processed and approved in parallel. Independently. And as the results of an approval and recomputation are also deltas these can be easily merged at the end. No need for locking.

  • MidPoint prefers real-time synchronization. This is usually the best solution. The data are fresh. The load is more spread through time avoiding many hot spots. And it is easier to look up a single change for diagnostic purposes. MidPoint is designed to be what is usually called almost-realtime system: efficient, quickly responding to change and fun to work with.

  • Of course the is also reconciliation. But we use it mostly only as safety net. Reconciliation is serving legacy remote system that cannot do any better than that. It also caches the very unlikely events when delta-based algorithms converge to a slightly imprecise results. However, running reconciliation in midPoint is easy. Because of reuse the reconciliation has exactly the same basic configuration as real-time sync. Therefore adding reconciliation to the solution essentially means only to enable it and set up a schedule.

  • MidPoint is a self-healing system. When midPoint finds out that there is a problem it tries to fix it automatically. E.g. if midPoint tries to modify an account which is gone then midPoint will try to re-create it. If midPoint tries to create an account which conflicts with another account midPoint automatically reconciles the newly found account to check for ownership and to see what to do with it. And then midPoint re-starts the original operation (remember, we have deltas, we can do it easily and safely). This "opportunistic" synchronization really makes sure that the data are as consistent as practically possible.

  • One of the curial points in midPoint design is that midPoint clearly understand the difference between policy (what should be) and reality (what is). It has doubled mechanisms to efficiently handle the distinction. This was a major problem in traditional IDM systems. Pretending that the reality always matches policy is futile. MidPoint knows better. The assignments can be used to set the policy, the links and shadows track the reality and synchronization is used to align them. Therefore midPoint knows whether the result should be A or B.

Continuous change of data is really a major concern in IDM solutions. However the data and the environment are not the only changing things in the solution. Requirements and processes change quite often. Which means changes in data structures. MidPoint is based on the idea of the unified data model which in turn is based on Single Source of Truth principle. The data model of midPoint is defined as an abstract schema. Java class files are generated from this schema during the build. The same schema is also used by midPoint user interface to display the data. Therefore a simple change of the schema definition echoes through the entire system. Which means it is easier to integrate: parts of the user interface will adapt automatically without any change to the code. And the parts of the code which are not compatible with the changed schema will immediately be identified because the compilation breaks. It is very easy to fix bugs in this way. And that’s what we do. Parts of the schema can also be customized after the build when midPoint is deployed. And again, user interface and other dynamic parts of the system will adapt automatically. Without any need for programming.

There is still one more source of change. A very important one: the software itself. MidPoint has a rapid release cycle with two releases per year. This approach used by many high-profile open source projects and it works incredibly well for them as it works for us. This means that midPoint software changes quite often. This is an excellent news on one hand. Partners and customers can get their features and improvements very quickly. MidPoint deployments are very efficient. But there is a risk on the other had. Keeping midPoint solution operational may require frequent upgrades. We are more than aware of this risk and therefore we spend considerable effort to make the upgrade process easy. The goal is to make it almost effortless with risks kept as low as practically possible. Upgradeability is an integral part of midPoint design and architecture from the very beginning. E.g. we pay a lot of attention to data format compatibility, compatibility of extensions and so on. That’s one of the reason we are so meticulous about proper namespaces and URIs which are used a lot in midPoint. However, upgradeability is a hard problem. We have gone a long way and we believe that our ability to upgrade is several levels above average we are continually improving it.

Cooperation

The midPoint software is only a part of midPoint solution. It is undoubtedly a very important part, but it is only a part nevertheless. It would be nothing without the ability to deploy and operate it efficiently. And it is the people that deploy and operate midPoint, not the software. Therefore we pay a great attention to work with our partners and customers. Cooperation is a crucial part of our day-to-day work.

What does it mean, exactly?

  • The business model of Evolveum is strictly partner-based. We strongly believe that everybody should do what he does best. We are best at development of identity management software. So that’s what we do. Our partners are best at designing IDM solutions, deploying them, counselling, identifying new customers, communicating with them and keeping them. And that’s what our partners do. We support our partners and we do not compete with them.

  • MidPoint source code is open and the licence is one of the most liberal open source licenses available. Anyone can see source code, anyone can modify it and use it. MidPoint development process is also open. Our plans, roadmaps and designs are documented and publicly available. Therefore it is not only the anyone can see the source code but also anyone can understand it.

  • We are encouraging midPoint users to participate. Major features of several recent midPoint versions were planned and designed largely based on user feedback. We also encourage participation of developers outside of midPoint core team. We have chosen the tools that are appropriate for this kind of cooperation.

  • Complete documentation of midPoint is publicly available. It is maintained in the wiki form and we try to follow the encyclopedic style used by Wikipedia. We gladly grant write access to the documentation to any partner who is willing to help with documentation updates.

  • We have a great team. Really great. But it is a small team and it will remain relatively small for foreseeable future. We prefer efficiency, technological excellence and merit over world domination. And therefore we are more than aware that we cannot do everything. Therefore our plan is to cooperate with other software vendors that have similar mindset that we have. Our plan is to stick to identity provisioning. That’s currently the missing piece as other existing products lacks either functionality, cost efficiency or both. When it comes to access management, directory services, compliance, analytics and other identity management subfields we prefer cooperation over competition. Our plan is to integrate with existing products on the market.

See Also

Was this page helpful?
YES NO
Thanks for your feedback