Infrastructure Design Meeting
Guests:
-
midPoint monitoring (performance, actuators, …) - who?
-
Technical testing - advanced coordination and design decisions make it part of the infra - who?
Overview
The goal of design meeting is to figure out design details about particular functionality of a software, particular solution, or even a methodology or process. It is usualy applied to midPoint design. But it may be also useful for non-technical and organizational aspects.
Design meeting is a creative, free-form discussion. It may be following an agenda, but it is usually closer to a free-form brainstorming.
Schedule
Design meetings are schedules as needed, the schedule is not fixed. For midPoint, the design meeting usually happens at the beginning of a development cycle or milestone cycle.
What To Discuss
Motivation and Requirements
What is the motivation to discuss this topic at all? Do customers need it? What are the use cases? Will it help make deployments better/faster? How exactly?
Use cases:
-
Our infrastructure has to support for heavy load/performance test: setup, init, execute, collect/monitor, evaluate and tear down the tests
-
Open question: the evaluation of data (eg logs) what are the requirements from people who will need and evaluate it? Like how to collect data from nodes, what loggers to enable, when, …
What will happen if we just do nothing? Do we really need this? It is worth the effort?
Should we do this now? Maybe we can postpone it, there may be better opportunity in the future.
What are the assumptions? What we expect that customers will do? Maybe we are not certain about some requirements and we just assume something?
Assumptions:
-
Focus is to make it work in our private cloud. No effort shall be spend on the abstractions and preparation on the other clouds.
-
system components: mP, LDAP, PSQL, other resource prefer DBTables (PSQL) not files (scalability).
-
We will focus on docker and dockerization, not hybrids for now (VM/Windows).
Requirements:
-
Monitoring - we would need to sample environment
-
midPoint metrics - midPoint is doing itself, but how to leverage those data is open question
-
Spring actuators - we will sample also these. open question - what data?
-
Profiling - open question for now
-
PSQL - performance monitoring and tuning.
Do we have some performance or scalability targets? Do we know how big a system do we want to support? How many users, how many requests per second, special usage patterns (bursts), anything else?
Performance and scalability goals:
-
Start with 1mio of records, target 10+ mio, in order of magnitude tens of millions
-
The records are like cartesian product: 10 mio of users, each 10 accounts is like 100 million shadows
-
Open question: number of other objects? Like roles, services, orgs? And also many assignments slow down problem
-
Problem of many attributes for an object (100+).
Ideas and Concepts
When thinking about the use cases, do not limit your thoughts to just that one specific use case. Think about generic mechanisms, broader principles. Focus on concepts and ideas, rather than algorithms. Design mechanism that can handle your use case, and thousands of similar use cases as well. Design generic re-usable mechanisms.
Make sure the new mechanisms work well with existing mechanisms. We are looking for synergies. We want to combine mechanisms together into more flexible and more powerful solutions.
Implementation
Kubernetes
-
Goal: running on at least 2 different machines
-
AI: Patrik: Decide if we would like to go to physical or virtual machines
-
Logging & Monitoring: AI: Patrik: Is Prometheus the right for us?
-
Goal: monitoring with context (not only basic stats, but also collect and display logs)
-
Logging: collect logging data to one place. what component shall do that? Maybe the goal above will decide for us
-
-
Deployment: AI: Kamil: Decide if Lens IDE is the right way to go for us? Or anything else? Hybrid web UI + Lens?
-
(Docker) image repository
-
AI: Accesible to outside? Security?
-
-
Testing: TODO: wait few more weeks to complete other design sessions
-
Connect to Jenkins, collect test outputs
-
API access for testing (volumes, k8s (Env. setup/Containers start/stop), …)
-
Performance and Scalability Considerations
TODO
Testability Considerations
Security Considerations
TODO
Rolling Wave Design
TODO
Write It Down
Notes from design meeting at the appropriate place. For midPoint, the appropriate place is usually Design Notes at docs for public notes, or MidPoint Design Notes at guide for private notes.
Do not forget to document:
-
Requirements and assumptions. Interesting use cases.
-
Outline of the approach, important aspects of algorithms, schemas and so on.
-
Decisions that were made, also the explanation or motivation why the decision was made.
-
Outline of a plan. What do we implement now? What parts will remain to be implemented later?
-
Risks and challenges. What parts are likely to be problematic? Where can the design fail?
-
Open questions. What we cannot answer now? What problems remain to be solved later?