MidScale: MidPoint Scalability

MidPoint is an open source identity management and governance platform. MidPoint is an established solution for mid-size organizations providing transparency and accountability for personal data processing. Our midPrivacy initiative aims at implementation of unique data protection capabilities to midPoint, thus creating privacy-enhancing identity management platform. Overall, midPoint is a leading open source solution in several identity and access management areas.

However, midPoint has one significant limitation. MidPoint was originally built to address the needs of mid-size enterprises, agencies and universities. Initial design of midPoint data store components favored flexibility and time to market. As midPoint was targeting mid-size organizations the scalability was not high on a list of implementation priorities. But now, midPoint is being deployed to handle scenarios with large number of identities. Deployments that manage students, subscribers and consumers are becoming more and more common. Which makes sense, as these types of users can especially benefit from the data protection capabilities of midPoint. However, such deployments are hitting scalability limitations of current data storage components of midPoint.

Project Goals

Future scalability issues were foreseen in original midPoint design. MidPoint is not bound to any particular data store or database. Thanks to such foresight, midPoint has a flexible and replaceable data storage components. We would like to take advantage of this design feature and re-implement data storage components in a scalable way. Our plan is to redesign the database schema with scalability in mind. Also, we plan to take advantage of innovation in open source databases which was not readily available when midPoint development started almost a decade ago. Therefore we plan to fit our implementation specifically to PostgreSQL database to take full advantage of its capabilities.

We also plan to improve midPoint clustering mechanisms. Our goal is to support autoscaling capabilities used in cloud platforms, thus enhancing the on demand character of midPoint deployments. Higher scale also implies harder requirements on stability and robustness of the product. Therefore we plan to invest part of the effort to improve our quality assurance environment, especially focusing on scalability, performance and stability testing. MidPoint would greatly benefit from user experience improvements that can make administration of millions of identities easier.

Documents

Blog, Articles And Other Media

Timeline

Milestone Goal Planned date Status

START

Project start

15-October-2020

DONE

M1

Performance environment

26 Nov 2020

DONE
See M1 Outcomes

M2

Performance evaluation and repository analysis/design

15 Jan 2021

In progress

M3

Multithreading, Schrodinger and Query language

26 Feb 2021

M4

Performing Repo prototype, UI basic tests, Multinode Tasks

12 Apr 2021

M5

PostgreSQL, Performance environment (2), UX Analysis & Design

31 May 2021

M6

Repository optimization, performance evaluation

9 Jul 2021

M7

Migration Procedure, GUI Improvements, Auto-scaling

31 Aug 2021

FINISH

Project finish

14-October-2021

Funding

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the NGI_TRUST grant agreement no 825618.