Cluster deployment

Last modified 30 Jun 2022 04:40 +02:00

What is it about

This page will cover use cases when more midpoint nodes in the cluster configuration is use. This page is extending the information valid for single node deployment.

In case you need working sample you can go directly to the part related to NFS (it is recommended deployment option to fulfill several requirements).

The full configuration is available in statefulset-pg-native_cm-sec-nfs.yaml on github.

Cluster of midpoint nodes

There are few things which has to be handled to be able to operate midpoint in cluster - more cooperating nodes.

  • taskManager

  • nodeID

  • keystore

Once all the "cluster requirements" is met you can increase amount of replicas in statefulset definition for midpoint.

In case of statefulset the suffix of the pod is increasing order. First created pod has suffix -0. In case you increase the amount of replicas the pod are added "to the end" of the list -1, -2, -3, etc. In case you are decreasing the amount of replicas the latest one is removed. It is not possible to remove the pod "in the middle" of the list. This may be important in case of utilizing persistent volumes for the pods.

Midpoint pods can be operated even without persistent volumes as the important objects are stored in the repository and shared between the nodes. The areas which may need specific handling:

  • logs
    to not lost the records after removing / re-creating the pod

  • connectors
    It can be distributed using shared object (configMap, R/O shared volume between the pods, etc.)

  • exports / reports
    In some situation the output can be stored in the filesystem. In that case we probably prefer to keep the files even after re-creating the pod.

The list is an example and it does not have to be complete. The design of the deployment may contain other specific objects to handle.

Task Manager

Midpoint’s task manager has to be run clustered. This setting has to be added to all nodes.

...
          env:
            - name: MP_SET_midpoint_taskManager_clustered
              value: true
...

Node ID

Node as cluster member has to have the unique ID. To be able to run the node there have to be set the way how to generate / set the ID. As the pods name are unique (generated) we can use hostname for this purpose. To do that we need to set one additional environment variable.

...
          env:
            - name: MP_SET_midpoint_nodeIdSource
              value: hostname
...

Keystore

Keystore is generated with the start in case it is not available. The result is that each node would generate "own" key in the keystore and the object will be readable only by the node which has created it. To address this "issue" we have to prepare the keystore to be available to each node once it is starting.

  • Share the file

    • Network Share
      Kubernetes natively offer mount NFS store as volume to pod. We can share the space and first node will generate it. All other node or even this node after restart / recreate will use the file so the key for description will be available. Using persistent volume for NFS server is good idea.

      The issue may happen once two pods would start in parallel and both would want to generate it.

    • Volume Share
      Not all the drivers offer concurrent write access to the volume. This option not necessary have to be available in general.

  • Pre-generate the file into secret object and share it with all the pods as mounted volume
    For the testing purpose this approach offer sharing the keystore even between the whole env deployment.

NFS Mount

NFS has dedicated part of this document.

To mount volume you can use syntax like this example:

Example for NFS volume
...
    spec:
      volumes:
        - name: nfs
          nfs:
            server: mp-demo-nfs.mp-demo.svc.cluster.local
            path: /exports
...

Once the shared NFS volume is utilized we don’t need to handle manual generating of the keystore as the first starting midpoint node will generate it. All other midpoint nodes will just use it as far they will see the file in the location.

You can generate the keystore even with NFS utilizing in case you prefer e.g. other than default key size.

Secret object with keystore object

To create the secret object we will need to create the keystore on the filesystem.

Generate the keystore
keytool -genseckey -alias default -keystore keystore.jceks -storetype jceks -keyalg AES -keysize 128 -storepass changeit -keypass midpoint

Once the file exists we can use it to create the secret object in the kubernetes environment.

Create the secret object from the file
kubectl create secret generic -n mp-demo mp-demo-keystore --from-file=keystore.jceks --from-literal=keystore=changeit

Once the secret is created it cannot be changed. In case we will need to update it the command to delete the object may be useful.

Delete the secret object
kubectl delete secret -n mp-demo mp-demo-keystore

Once the secret is created we have to modify the stateful set for the midpoint.

Environment variable to check for presence
...
      volumes:
        - name: keystore
          secret:
            secretName: mp-demo-keystore
            defaultMode: 420
...
          env:
            - name: MP_SET_midpoint_keystore_keyStorePath
              value: /opt/midpoint/mount-keystore/keystore.jceks
            - name: MP_SET_midpoint_keystore_keyStorePassword_FILE
              value: /opt/midpoint/mount-keystore/keystore
...
          volumeMounts:
            - name: keystore
              mountPath: /opt/midpoint/mount-keystore
...

Additional certificate to keystore

In some situation we need to add additional certificate to keystore. The common use case may be to set trust to the certificate for SSL / TLS sessions. As the keystore is loaded during the start of midPoint it has to be prepared in advance. The ideal location for it is initContainer.

In case you have persistent volume with already generated keystore the relevant steps are related just to import the certificate to keystore.

The option for additional certificate is available in midpoint.sh file. It is described also in docker related part of documentation (kubernetes is utilizing docker images).

To add the certificate we need to generate the keystore. For this purpose we need to use following environment variables:

  • midpoint keystore password (one of the options)

    • MP_PW_DEF
      Where to store the default password for keystore - changeit

    • MP_PW
      Where to store generated password for the keystore

  • MP_KEYSTORE
    location of the jceks file

  • MP_CERT
    The certificate(s) to add into the keystore.+ The structure of the certificate(s) should be:

    -----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----

    or

    -----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
example of the simple standalone pod with custom trusted certificate (H2 DB) | link: Github
apiVersion: v1
kind: Pod
metadata:
  name: mp-demo-cert
  namespace: mp-demo
spec:
  volumes:
    - name: vol
      emptyDir: {}
  initContainers:
    - name: mp-config-init
      image: evolveum/midpoint:4.4.1-alpine
      command:
        - /bin/bash
        - /opt/midpoint/bin/midpoint.sh
        - init-native
      env:
        - name: MP_PW_DEF
          value: /opt/mp-home/keystorepw
        - name: MP_KEYSTORE
          value: /opt/mp-home/keystore.jceks
        - name: MP_CERT
          value: |
            -----BEGIN CERTIFICATE-----
            MIIDxDCCAqygAwIBAgIUfImu6HQ145sOn1ra6pYn9KgIQzkwDQYJKoZIhvcNAQEL
            BQAwFDESMBAGA1UEAwwJZGVtbyBjZXJ0MB4XDTIyMDIyMjA5MDQ1M1oXDTMyMDIy
            MDA5MDQ1M1owFDESMBAGA1UEAwwJZGVtbyBjZXJ0MIIBIjANBgkqhkiG9w0BAQEF
            AAOCAQ8AMIIBCgKCAQEAu4k5OzoZdBuBlc6ZVp3uwhiQi2sfRRPU8dzTATb6zN/W
            7Fn3mBngjHRfKKNBcmKgnlIA75qziiKew75/Zis50kT5lxTcS3fVylBfkNFO1WQO
            qg0FJBMoGPdrKRxww8XbAB5fThSNHe5ZlY0RhFX7xWSADzA7X4FzGsuou6l1xhrT
            MNu8brfUnf6JWG019mdevclhZzKycAW70UdhAOYj7a3LPMxGev7tLQCA25LCL0Gd
            jfZnZzkkyhYQbqkUmt6wAKmTesF3Az8uW7FKWIZ9kLkrXVP7xviGI7ga+XYrWM+Q
            1515ecUXL3YmODn/WdFbwGSnAXl9mQIyZmNRAsHuSwIDAQABo4IBDDCCAQgwCQYD
            VR0TBAIwADAdBgNVHQ4EFgQUTtuRHta8KDnjuGzpV8F0c/nApmwwTwYDVR0jBEgw
            RoAUTtuRHta8KDnjuGzpV8F0c/nApmyhGKQWMBQxEjAQBgNVBAMMCWRlbW8gY2Vy
            dIIUfImu6HQ145sOn1ra6pYn9KgIQzkwCwYDVR0PBAQDAgXgMGkGA1UdEQRiMGCC
            GG1wLXBnLWRlbW8ubXAtZGVtby5sb2NhbIIcd3d3Lm1wLXBnLWRlbW8ubXAtZGVt
            by5sb2NhbIIJbG9jYWxob3N0ghVsb2NhbGhvc3QubG9jYWxkb21haW6HBH8AAAEw
            EwYDVR0lBAwwCgYIKwYBBQUHAwEwDQYJKoZIhvcNAQELBQADggEBACXhqQlHdnNF
            XP5rHGBM98PSH7MZeavcQnBF3T4LDgCXeI++ab6sUpFaJAGdx156gd6sLr9OyV5h
            M2CloNT0omXkJOI1uDAJyqUu9RD47NiLDf6yr9z91t4O1pAu4rcWaaKT/wqVcGda
            WQ0mBgNgJLW+TwsT7rn93mdkhEIVXqmjhNNd6nfDdgkpozZCSPQkdt8IllKdPvrC
            Aq4wBlDHzKSIIhnDFa0+1DUnAATY9dREARGQ5SMjFf2j5+k/mrH+vpkSx7QD0rQt
            Rk5bDe6gVf6E1xl6s8e6Un65xncF7keoDUDHVHs6d16nhdLA2/yOMKSdic2EbfYB
            8WjXHbt7O9M=
            -----END CERTIFICATE-----
      volumeMounts:
        - name: vol
          mountPath: /opt/mp-home
          subPath: mp-home
      imagePullPolicy: IfNotPresent
  containers:
    - name: mp-demo-cert
      image: evolveum/midpoint:4.4.1-alpine
      ports:
        - name: gui
          containerPort: 8080
          protocol: TCP
      env:
        - name: MP_NO_ENV_COMPAT
          value: '1'
      volumeMounts:
        - name: vol
          mountPath: /opt/midpoint/var
          subPath: mp-home
      imagePullPolicy: IfNotPresent
  restartPolicy: Always

In the log of the initContainer we will see log similar (the hash is related to the specific certificate used to add) log:

sample output visible on the console (log)
- - - - -
CN=test CA
1D:8D:E5:7A:29:13:09:A5:A3:9E:D5:CF:13:1A:41:65:EC:51:79:D5 .:. Not Found
36:C3:F0:82:BC:76:59:5F:B9:D9:22:93:F5:23:48:84:27:29:C6:88:04:E2:17:24:E7:AC:5C:2A:FB:BC:2E:68 .:. Not Found
Adding cert to certstore...
- - - - -

There can be used reference to the secret object instead of directly including the cert. This approach make it easier to update the certificate

standalone pod with H2 DB (difference to previous sample - full object is on the Github) | link: Github
spec:
  initContainers:
      env:
        - name: MP_CERT
          valueFrom:
            secretKeyRef:
              name: cert-mp-pg-demo
              key: tls.crt
The Secret with certificate with cert-mp-pg-demo certificate is used in the example.

NFS

NFS volume is natively supported with the kubernetes (it is described e.g. in Kubernetes documentation related to the Volumes).

To have it working there are few thing which should be checked on kubernetes node:

  • NFS tools available on the operating system
    Kubernetes call system tool to mount the NFS volume. The required package name may differ based on the used distribution - on the debian based distribution (including ubuntu) the name of the package is nfs-common.

  • DNS resolving
    In case we want to use "internal" cluster FQDN it has to be resolvable for the kubernetes' node OS. by default the names are resolvable in the cluster but node’s resolver may use "external" DNS server where the cluster FQDNs are not known. The solution is point OS’s resolver to the cluster "internal" IP as the node can communicate with any cluster "internal" IPs.

Example of the change on debian based distribution (e.g. IP of DNS is 10.96.0.10)
cat << EOF >>/etc/systemd/resolved.conf
#[Resolve]
DNS=10.96.0.10
Domains=~cluster.local
EOF
systemctl restart systemd-resolved

Server

Statefulset definition for server | Github
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mp-demo-nfs
  namespace: mp-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mp-demo-nfs
  template:
    metadata:
      labels:
        app: mp-demo-nfs
    spec:
      containers:
        - name: mp-demo-nfs
          image: 'k8s.gcr.io/volume-nfs'
          command: ["/bin/bash", "/usr/local/bin/run_nfs.sh", "/exports"]
          ports:
            - name: nfs
              containerPort: 2048
              protocol: TCP
            - name: mountd
              containerPort: 20048
              protocol: TCP
            - name: rpvbind
              containerPort: 111
              protocol: TCP
          securityContext:
            privileged: true
          volumeMounts:
          - mountPath: /exports
            name: mp-demo-nfs-store
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
      terminationGracePeriodSeconds: 10
  serviceName: mp-demo-nfs
  volumeClaimTemplates:
    - kind: PersistentVolumeClaim
      apiVersion: v1
      metadata:
        name: mp-demo-nfs-store
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 256Mi
        storageClassName: csi-rbd-hdd
        volumeMode: Filesystem
The size 256 MB is used in the example. This size have to be set based on the usage of the NFS share. In case you will be using shared storage also for Export directory the requirements for the space may be higher.

There has been used the same image as in kubernetes documentation. Feel free to use any other image containing nfs server tool you are familiar with.
Service definition for the server | Github
apiVersion: v1
kind: Service
metadata:
  name: mp-demo-nfs
  namespace: mp-demo
spec:
  ports:
    - name: nfs
      port: 2049
    - name: mountd
      port: 20048
    - name: rpcbind
      port: 111
  selector:
    app: mp-demo-nfs

midPoint with NFS volume

Statefulset definition | Github
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mp-pg-demo
  namespace: mp-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mp-pg-demo
  template:
    metadata:
      labels:
        app: mp-pg-demo
    spec:
      volumes:
        - name: mp-home
          emptyDir: {}
        - name: db-pass
          secret:
            secretName: mp-demo
            defaultMode: 420
        - name: mp-poi
          configMap:
            name: mp-demo-poi
            defaultMode: 420
        - name: nfs
          nfs:
            server: mp-demo-nfs.mp-demo.svc.cluster.local
            path: /exports
      initContainers:
        - name: mp-config-init
          image: 'evolveum/midpoint:4.4-alpine'
          command: ["/bin/bash","/opt/midpoint/bin/midpoint.sh","init-native"]
          env:
            - name: MP_INIT_CFG
              value: /opt/mp-home
          volumeMounts:
            - name: mp-home
              mountPath: /opt/mp-home
          imagePullPolicy: IfNotPresent
      containers:
        - name: mp-pg-demo
          image: 'evolveum/midpoint:4.4-alpine'
          ports:
            - name: gui
              containerPort: 8080
              protocol: TCP
          env:
            - name: MP_ENTRY_POINT
              value: /opt/midpoint-dirs-docker-entrypoint
            - name: MP_SET_midpoint_repository_database
              value: postgresql
            - name: MP_SET_midpoint_repository_jdbcUsername
              value: midpoint
            - name: MP_SET_midpoint_repository_jdbcPassword_FILE
              value: /opt/midpoint/config-secrets/password
            - name: MP_SET_midpoint_repository_jdbcUrl
              value: jdbc:postgresql://mp-demo-db.mp-demo.svc.cluster.local:5432/midpoint
            - name: MP_UNSET_midpoint_repository_hibernateHbm2ddl
              value: "1"
            - name: MP_NO_ENV_COMPAT
              value: "1"
          volumeMounts:
            - name: mp-home
              mountPath: /opt/midpoint/var
            - name: db-pass
              mountPath: /opt/midpoint/config-secrets
            - name: mp-poi
              mountPath: /opt/midpoint-dirs-docker-entrypoint/post-initial-objects
            - name: nfs
              mountPath: /opt/midpoint/var/post-initial-objects
              subPath: poi
          imagePullPolicy: IfNotPresent
  serviceName: mp-pg-demo
      volumes:
        - name: nfs
          nfs:
            server: mp-demo-nfs.mp-demo.svc.cluster.local
            path: /exports
...
      containers:
        - name: mp-pg-demo
          volumeMounts:
            - name: nfs
              mountPath: /opt/midpoint/var/post-initial-objects
              subPath: poi

subPath label offer the option to structure NFS store. In case we prefer use just post-initial-object (poi) mount point we can use subdirectory on NFS structure. Using this option we don’t need to prepare the structure in advance as in case of defining /export/poi in volumes section.
The pod will not start (it will wait in state PodInitializing) until the NFS will be available. It can be unavailable as NFS server is not up yet or the FQDN can’t be resolved. The reason can be find out in the pod’s information.

Scaling

Once everything is ready we can scale up ( even starting is scaling : 0 ⇒ 1 ) the number of replicas.

To run 2 nodes midpoint cluster
kubectl scale -n mp-demo --replicas=2 statefulset/mp-pg-demo
To reduce to just 1 node in midpoint cluster
kubectl scale -n mp-demo --replicas=1 statefulset/mp-pg-demo

New pods are added with increasing suffix ( -0, -1, -2, etc.). With scale down the pods are removed in reverse order - the first is terminated the highest suffix. You cannot terminate pod with the suffix "in the middle of the list". In that case it is identified as failed pod, it is re-created by the statefulset definition.

In case you don’t use persistent volume all the data related to the jus terminated pods are lost. If you use persistent volumes the data is kept. With scale up the persistent volume will be attached if it exists (previously it has been created) otherwise it will be newly created.

There are some data which may be important like logs from the node, exported (generated) content (e.g. reports). If you don’t need it is ok but if so, please, think about utilizing NFS storage at least partially for the needed content if not for the whole midpoint home directory.

Session affinity

With the clustered instance of midpoint the session handling have to be covered. Even the midpoint nodes are operating in cluster the session is created against specific node. Kubernetes is using round-robin distribution by default.

It means than you would create session against one node but next query will go (the most probably) to the different one. This node has no idea about session on the "other" node, so it would forward you to the login form. This is not definitely the behavior we would expect.

With the following setting we are able to make sessions by the client IP.

Service

On the service object we can set directly the affinity:

...
spec:
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800
...

Ingress

Ingress object cooperate with the service but mainly for the list of backend(s). The session handling would be set also on this level.

Example relevant for nginx ingress
...
metadata:
  annotations:
    nginx.ingress.kubernetes.io/session-cookie-expires: '172800'
    nginx.ingress.kubernetes.io/session-cookie-max-age: '172800'
    nginx.ingress.kubernetes.io/session-cookie-name: mp-demo
...