Outcomes of MidScale Milestone M3
Milestone goal: Multithreading, Schrodinger and Query language
Status: DONE (26 Feb 2021, on time)
Activities and Outcomes
There were many parallel activities during this milestone:
Multithreading and robustness improvements. This milestone was focused on improving resiliency of midPoint in a large-scale deployments. Error handling and recovery of long-running synchronization tasks was identified as the most painful problem, therefore substantial part of the development effort was focused on improving this functionality. There were major code quality improvements, especially in midPoint provisioning components and related components. The result is major improvement in error detection and visibility, and automatic error recovery, especially improved resiliency of massive multi-threaded and multi-node synchronization tasks.
In addition to synchronization resiliency, there were also improvements to thread safety of Prism code, which should also contribute to overall robustness of large scale deployments.
Schrödinger testing framework. The work on Schroedinger continued in M3. Most of the work still focused on fluent API improvements, to clean up the framework and support ease of writing of future tests. There were also major improvements in visibility, adding new reports, mostly focused on performance. Also, support for new pages and pannels was addeded to the framework.
Query language. New, user-friendly query language was designed during M2, ahead of plan. Integration of the language into midPoint core was finished in M3. The query language is used in some parts of midPoint user interface.
Testing environment. Work on large-scale testing environment contninued in M3. The entire testing environment is based on Kubernetes, all the essential infrastructure (e.g. DNS) is in place as is monitoring and support infrastructure. The environment is ready.
There are few leftovers that could be addressed in future milestones, such as full migration of Jenkins and Schrödinger to Kubernetes. However, these currently do not pose an obstacle for using the environoment.
PostgreSQL repository prototype. Work on the repository prototype continues at full speed. At the end of M3, the work has brought promising results. Basic midPoint instace can start with a new prototype implementation, although all the operations are not implemented yet. Even though midPoint 4.3 development now enters a feature freeze period, the work on the prototype will continue, as its development is mostly independent from other midpoint code.
There were also several minor improvements, such as the ability to store repository data in JSON, which was based on several improvements in the Prism core.
Survey of the midScale plans was started during M3. The survey is still active.
The activities of this milestone were mostly carried out according to original plan. Yet, there were some differences:
In previous milestones, several activities have started ahead of plan, to utilize free resources and reduce future risks. This includes query language, Schroedinger and multithreaded tasks. The project plan was stabilized in this milestone, proceeding without any major deviations from original plan.
More people were involved in the project, outside of the NGI funding. It was decided to strengthen the architectural part of the project (Radovan Semančík) and support for testing infrastructure (Kamil Jireš).
We have utilized the results of midPrivacy initiative in this project, especially the Axiom modeling language. The ideas of Axiom were applied especially to the design of query language, making the midPoint design more consistent. This was not included in the original design of midScale, as Axiom did not exist at that time yet.