Used kubernetes objects

Last modified 11 Mar 2022 09:41 +01:00

What is it about

This page is describing used kubernetes objects and its small description focused on midPoint environment.

Type of used k8s objects

Namespace

To have objects logically grouped the namespace can be used. The namespace has an impact on generated FQDNs or even permissions in the cloud environment in case they are utilized.

Sample object is available in Single Node deployment.

Pods

Pods are running instances of the images. We will not manage them directly as they will be a result of the settings in other objects (like statefulSets).

Sample object is available in Single Node deployment.

ConfigMaps

The common configuration can be stored in ConfigMaps which can be used in the environment to the definition of the environment variables in the pods or it can be mounted to filesystem. From the point of view of the pods they are read only objects, what is good in many scenarios. In case we would need R/W objects, the configmaps are not the right objects to be used (in that case volume may be the option).

Sample object is available in Single Node deployment.

The configMap can be created in several ways.

  • kubectl apply -f file.yaml
    This way the content goes also to annotation field in the creating object. This may sometime cause the issue as the size of the value for the field is limited. For the common small objects is doesn’t cause the issue. The namespace can be defined as parameter -n mp-demo or in the content of the yaml file.

  • kubectl create configmap -n mp-demo mp-demo-poi --from-file=.
    This way the annotation field is not created. If you provide directory as a value of from-file the all the files in the directory became the keys of configMap and the contents of the files become the value of the keys. In case you provide just a single file only one key is created in the configMap.
    The content of the kubernetes object is generated so the namespace have to be defined as parameter of the kubectl command.

Secrets

For some purposes we need to store the sensitive information like passwords for the environment. In this case ConfigMaps is not the best object to choose. The secret object type is used for this purpose. The content is base64 encoded value. The short values can be directly shown (decoded) in the tools like Lens. In principle, I can provide examples just with placeholder instead of the real values.

Example of the secret object containing x509 certificate | Github
kind: Secret
apiVersion: v1
metadata:
  name: cert-mp-pg-demo
  namespace: mp-demo
stringData:
  Expiration: Feb 20 09:04:53 2032 GMT
  SAN: DNS:mp-pg-demo.mp-demo.local, DNS:www.mp-pg-demo.mp-demo.local, DNS:localhost, DNS:localhost.localdomain, IP Address:127.0.0.1
data:
  tls.key: |
    LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2Z0lCQURBTkJna3Foa2lHOXcwQkFRRUZB
    QVNDQktnd2dnU2tBZ0VBQW9JQkFRQzdpVGs3T2hsMEc0R1YKenBsV25lN0NHSkNMYXg5RkU5VHgz
    Tk1CTnZyTTM5YnNXZmVZR2VDTWRGOG9vMEZ5WXFDZVVnRHZtck9LSXA3RAp2bjltS3puU1JQbVhG
    TnhMZDlYS1VGK1EwVTdWWkE2cURRVWtFeWdZOTJzcEhIRER4ZHNBSGw5T0ZJMGQ3bG1WCmpSR0VW
    ZnZGWklBUE1EdGZnWE1heTZpN3FYWEdHdE13Mjd4dXQ5U2Qvb2xZYlRYMloxNjl5V0ZuTXJKd0Ji
    dlIKUjJFQTVpUHRyY3M4ekVaNi91MHRBSURia3NJdlFaMk45bWRuT1NUS0ZoQnVxUlNhM3JBQXFa
    TjZ3WGNEUHk1YgpzVXBZaG4yUXVTdGRVL3ZHK0lZanVCcjVkaXRZejVEWG5YbDV4UmN2ZGlZNE9m
    OVowVnZBWktjQmVYMlpBakptClkxRUN3ZTVMQWdNQkFBRUNnZ0VBQ0VkRmRIemJGTW9HRFdQaVZ2
    Vi96YWVoOGRXWHVzZm1WWXJtOFozSXNuU0wKRzBIWmNTaHJSaWY0NzJWTmhVd2ZSUDNmdHJRQWl1
    OStUS3ZER2ZKOGlmQ3JoK1RPMkxJQWlQN0ZTVEpFSloxVwpIR0dZd1gvcS9EY256dGZIam96VVh3
    djRMdzB2TVl3TS9sSU0zT2VpeHBMcUtFRjd6WHA4WjBsb09HaUJuMjRLClZsL0wwb3dDVzc4bFNy
    ME9BcU5KN1krWDQ5aGQ4T05KdlYwSUtZQm5MbEVIUGQwSysxL3ZsdmJQNkMwSS9TNkkKYXZ0MXlu
    bHQ2MVV1eEVCZzVUQTZHWEJnQ2Y2N01sWWMyRDJoSHJjaDNMdlY1bjVocVdTaXNJblpiR0xNSTdD
    cgoycW1uSDB1QlpqR0FLaG5xNHdWUDZSVmJDRSsrejYxTHMrMDd5dS9Eb1FLQmdRRG41NDlYaDdE
    dlV4WnZIS3hECjlURi9yNmlHeXIyY3VsN0tpOThYTUFZdXFsQmwxVDNNMTB4UnQzNGdYbTBOSjZs
    a0xjRUhxTEhsdWZPQ3U4MlMKMzZ3NmYwd1NMYUo3ckJlZlNOK3EyNmFObVlkcW9iclZQUWc1R0ti
    Q1E3UWYvOTViMTc5S3V5UFpEdDQvQTJUZwowQ294S1FVWUVnd1hXRWowTUJmQ3R3RlEyd0tCZ1FE
    UEJZRHdnaW55SlNwZ3JtZkZLdk4wUXYrNEQ3eUR4SllwCmxNMTVTWnpwUU41ZFJlZkdVcVlMZnRx
    R0dSSWk2aFRhU3VTeEVtV1Y1R21EWlQxR1RmMkVVQWpkNnIyM0JVWS8KcXMxQlRmUDVMaXhNaEZz
    YkRoMlpxT0tnb1pBVy9DSXVtVU9GcFZyVkIwdFQvc0lZV1ZIMUJTcEhCZFczdHRqNgpKTjVMMDBU
    YlVRS0JnUURRNWFNV3l0RlVWRGtpTCtieGRINTVNYWVTZkRDZmJMYVVwN2gzTDdoeDh0Tm5WOVpL
    Cm5pcE9kZ2IwYTNxNlhkN1ZzZTF1MDhRa1Z3UUc0TnVHa1Z1WWVqRHhNeEJwYlJUK1UrdS9BZ3Vm
    QkF5YXZrTUoKY09mbVh0RmdyeElqdlIzMUIzeDJlZXNKekwzZ2IxTkF1K00xMDJpV1RUTjlGL2Ju
    bUNiZkxIRFAvd0tCZ1FDaQp6REF1Y2gyc1J4L0JNY1YyM3FUZXE0Vk82RWtWMGZWSU03VTFpUGIw
    MHNkSzBCdEk4VnVVTktpQnhadG1pMi9rCmpmcXphTVVzRDVnTjlRSXZsRXNsem81NmZRdTlyazhr
    NkJ0TEhKTWNRL3dnSEFGTkVGZWtxNU8wUC9rQXFqdnoKS3lGWEtzWjRPYWs2SDEvaEovUjBXeUI1
    QTVTaVNRN3QveW8wdEtvMFVRS0JnRXlKMDAzYVZjK2MxSHhHaTNEWApKZDUyL1d5YXVvemZhQVFx
    djd4UE9EdXNOSGdmRStIS3pyNHZpckcxbkVNaWE3ZTZldEtLeW9YRldmT3RBTmZNCnBnODZYUEhO
    QmJnR3lYTjY1YVhvdHdqbHVXSXhCVkU5QkVRRGRKZjBRd3JvaGpnTkZIUm1sUzlYL05UZWQ4YWUK
    QmdlazlZdlBOOHd5WTFkSzNSOVB2TTZPCi0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0K
  tls.crt: |
    LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR4RENDQXF5Z0F3SUJBZ0lVZkltdTZIUTE0
    NXNPbjFyYTZwWW45S2dJUXprd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0ZERVNNQkFHQTFVRUF3d0pa
    R1Z0YnlCalpYSjBNQjRYRFRJeU1ESXlNakE1TURRMU0xb1hEVE15TURJeQpNREE1TURRMU0xb3dG
    REVTTUJBR0ExVUVBd3dKWkdWdGJ5QmpaWEowTUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGCkFBT0NB
    UThBTUlJQkNnS0NBUUVBdTRrNU96b1pkQnVCbGM2WlZwM3V3aGlRaTJzZlJSUFU4ZHpUQVRiNnpO
    L1cKN0ZuM21CbmdqSFJmS0tOQmNtS2dubElBNzVxemlpS2V3NzUvWmlzNTBrVDVseFRjUzNmVnls
    QmZrTkZPMVdRTwpxZzBGSkJNb0dQZHJLUnh3dzhYYkFCNWZUaFNOSGU1WmxZMFJoRlg3eFdTQUR6
    QTdYNEZ6R3N1b3U2bDF4aHJUCk1OdThicmZVbmY2SldHMDE5bWRldmNsaFp6S3ljQVc3MFVkaEFP
    WWo3YTNMUE14R2V2N3RMUUNBMjVMQ0wwR2QKamZablp6a2t5aFlRYnFrVW10NndBS21UZXNGM0F6
    OHVXN0ZLV0laOWtMa3JYVlA3eHZpR0k3Z2ErWFlyV00rUQoxNTE1ZWNVWEwzWW1PRG4vV2RGYndH
    U25BWGw5bVFJeVptTlJBc0h1U3dJREFRQUJvNElCRERDQ0FRZ3dDUVlEClZSMFRCQUl3QURBZEJn
    TlZIUTRFRmdRVVR0dVJIdGE4S0RuanVHenBWOEYwYy9uQXBtd3dUd1lEVlIwakJFZ3cKUm9BVVR0
    dVJIdGE4S0RuanVHenBWOEYwYy9uQXBteWhHS1FXTUJReEVqQVFCZ05WQkFNTUNXUmxiVzhnWTJW
    eQpkSUlVZkltdTZIUTE0NXNPbjFyYTZwWW45S2dJUXprd0N3WURWUjBQQkFRREFnWGdNR2tHQTFV
    ZEVRUmlNR0NDCkdHMXdMWEJuTFdSbGJXOHViWEF0WkdWdGJ5NXNiMk5oYklJY2QzZDNMbTF3TFhC
    bkxXUmxiVzh1YlhBdFpHVnQKYnk1c2IyTmhiSUlKYkc5allXeG9iM04wZ2hWc2IyTmhiR2h2YzNR
    dWJHOWpZV3hrYjIxaGFXNkhCSDhBQUFFdwpFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0V3RFFZ
    SktvWklodmNOQVFFTEJRQURnZ0VCQUNYaHFRbEhkbk5GClhQNXJIR0JNOThQU0g3TVplYXZjUW5C
    RjNUNExEZ0NYZUkrK2FiNnNVcEZhSkFHZHgxNTZnZDZzTHI5T3lWNWgKTTJDbG9OVDBvbVhrSk9J
    MXVEQUp5cVV1OVJENDdOaUxEZjZ5cjl6OTF0NE8xcEF1NHJjV2FhS1Qvd3FWY0dkYQpXUTBtQmdO
    Z0pMVytUd3NUN3JuOTNtZGtoRUlWWHFtamhOTmQ2bmZEZGdrcG96WkNTUFFrZHQ4SWxsS2RQdnJD
    CkFxNHdCbERIektTSUlobkRGYTArMURVbkFBVFk5ZFJFQVJHUTVTTWpGZjJqNStrL21ySCt2cGtT
    eDdRRDByUXQKUms1YkRlNmdWZjZFMXhsNnM4ZTZVbjY1eG5jRjdrZW9EVURIVkhzNmQxNm5oZExB
    Mi95T01LU2RpYzJFYmZZQgo4V2pYSGJ0N085TT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
type: kubernetes.io/tls
...
stringData:
  Expiration: Feb 20 09:04:53 2032 GMT
  SAN: DNS:mp-pg-demo.mp-demo.local, DNS:www.mp-pg-demo.mp-demo.local, DNS:localhost, DNS:localhost.localdomain, IP Address:127.0.0.1
...

These lines are optional just for better overview - information about the containing certificate.
Commands to generate self signed certificate
# generate the key and the certificate
openssl req -new -sha256 -newkey rsa:2048 -keyout tls.key -nodes -subj "/CN=demo cert" | openssl x509 -req \
-signkey tls.key -out tls.crt -days 3650 -sha256 -extfile <(cat <<EOF
basicConstraints = CA:FALSE
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = digitalSignature, nonRepudiation, keyEncipherment
subjectAltName = DNS:mp-pg-demo.mp-demo.local, DNS:www.mp-pg-demo.mp-demo.local, DNS:localhost, DNS:localhost.localdomain, IP:127.0.0.1
extendedKeyUsage = serverAuth
EOF
)

#create secret object for kubernetes
kubectl create secret tls -n mp-demo cert-mp-pg-demo --cert=tls.crt --key=tls.key

# show the content of the certificate
openssl x509 -in tls.crt -text -noout

Services

As the pods are in principle dynamic objects, the IPs are changing each time the pods are recreated. To have "stable" point for interaction between the pods, the services are defined. The service looks for the pods based on the label. The service itself has assigned IP. The traffic is "forwarded" to the pods relevant to the service based on the label selector.

One example has been already provided related to the deployment with PostgreSQL DB. Other example may be for the midpoint itself. Here is example ready for the cluster environment. The difference is in Session Affinity setting.

Example for the midPoint service (cluster ready) | Github
apiVersion: v1
kind: Service
metadata:
  name: mp-pg-demo
  namespace: mp-demo
spec:
  ports:
    - name: gui
      protocol: TCP
      port: 8080
      targetPort: 8080
  selector:
    app: mp-pg-demo
  type: ClusterIP
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800
SessionAffinity is necessary for the midpoint as the session is not shared between the nodes of the cluster. In case there is just one node the missing of the affinity setting is not critical. Once there are more than 1 node the missing of the affinity setting cause loop of the login process.
The reason is that after sending login information the session is created with one node but the next communication is handled by other node - the default is round-robin distribution of the communication. This other node doesn’t know anything about just created session on previous node so the redirect to login page occur.

TODO: We will improve the configuration samples later to use session cluster replication to remove need of session affinity

Ingress

To be able to reach the services from outside on shared ports (80,443) there is ingress in place. It utilizes SNI, which is nowadays automatically used so there is no additional requirement. We are defining the rules for the conditional traffic forwards to the specific service and port.

Example of the ingress object definition (assumption: mp-demo.local domain) | Github
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: mp-pg-demo
  namespace: mp-demo
  annotations:
    ingress.kubernetes.io/affinity: cookie
    nginx.ingress.kubernetes.io/session-cookie-expires: '172800'
    nginx.ingress.kubernetes.io/session-cookie-max-age: '172800'
    nginx.ingress.kubernetes.io/session-cookie-name: mp-demo
spec:
  tls:
    - hosts:
        - mp-pg-demo.mp-demo.local
      secretName: cert-mp-pg-demo
  rules:
    - host: mp-pg-demo.mp-demo.local
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: mp-pg-demo
                port:
                  number: 8080

StatefulSets

This is the glue for all. This object defines the setting for the future pods and handle the amount of replicas. In case some pod will fail, the StatefulSet definition will handle the situation and recreate the new one.