Native repository demo

Last modified 25 Nov 2021 14:10 +01:00

Since 4.4

This functionality is available since version 4.4.

Table of Contents

Configuration prerequisites
- DB structure
- config.xml
Customization in midpoint.sh
- init-native
- container
Docker-compose yaml file

Native repository implementation can only be operated on postgres DB system so it is not possible to "simply run" the docker image to test it. Until the H2 DB backend will be available, you can run docker image directly but generic DB backend will be utilized.

Configuration prerequisites

To this topic a dedicated docs page is available.

DB structure

For the generic repository implementation the structure (the basic content) of DB could be automatically created if does not exist yet. This "feature" is not available with native repository implementation. As native repository implementation can only be operated with postgresql (an optimization focused on specific DB system is available). There is a dependency, which can’t be handled automatically anyway.

In the distribution archive with midpoint (and also docker image) there is a doc folder available, which contain required SQL structure. To keep docker part independent from application, the structure is taken from these files so its maintaining is realized through the distribution of midpoint.

The files are located at /opt/midpoint/doc/config/sql/native-new folder.

There are more files located and the order is important - some files contain structure, which extends other existing structure.

config.xml

To connect with the database there is a need to set a type for repository and for audit too. Especially because of audit it is not possible to set just using -D parameter.

The sample (and "core" part) of configuration is also available under doc folder. With this file the rest can be handled with -D parameter.

The sample config file is located at /opt/midpoint/doc/config/config-native.xml.

Customization in midpoint.sh

To make things easier there is a prepared "specialized" functionality focused on container environment. Relevant options are:

init-native
container

init-native

This section is focused on handling available files. What exactly will happen is controlled by the environment variables. All these options are available since v4.4devel-1921-g81d58591ac (github).

MP_CHECK
Touch file which can be checked during midpoint start to prevent starting application until the init structure is ready.
example of the value
```
MP_CHECK=/opt/mp-home/init_in_progress
```
MP_INIT_DB
The native "init" sql files will be copied from doc structure (distributed directly in docker image) to the destination provided as value for the variable. This option will work without change even if the list of init file will change in the future.
example of the value - destination directory is /opt/db-init
```
MP_INIT_DB=/opt/db-init
```
MP_INIT_DB_CONCAT
The native "init" sql files will be concatenated to one file in "known" working order. The location of the file is set by the value of the variable. The files are listed so in case of future change in the structure, the list has to be manually updated.
example of the value - target file is /opt/db-init/init.sql
```
MP_INIT_DB_CONCAT=/opt/db-init/init.sql
```
MP_INIT_CFG
Sample config.xml file required for native repository will be copied (and properly renamed) to the directory set by the value of the variable. With this file all the rest required variable can be set using MP_SET_ prefixed environment variables
example of the value - target directory is /opt/mp-home (file will be /opt/mp-home/config.xml)
```
MP_INIT_CFG=/opt/mp-home
```
MP_INIT_LOOP
This variable will cause the infinitive loop at the end of the processing. The benefit may be in some container environments, when the end of processing may be understood as failure (e.g. the container would be restarted). Only "acceptable" value is 1. Any other value will have no impact.
example of the value - the container will never end (have to be forcedly terminated to end)
```
MP_INIT_LOOP=1
```
MP_DB_PW
The password for the database access has to be the same on the client and server side. As far as the roles are split, the password has to be set in advance to be the same on both side of communication. The value is the path to the file, where the generated password should be saved.
example of the value - the generated password will be saved to /opt/db-pw/dbpassword
```
MP_DB_PW=/opt/db-pw/dbpassword
```
MP_PW_DEF
A default password for the keystore is "changeit". The value for this variable says, where the default password should be saved.
example of the value - the default password will be saved to /opt/mp-home/keystorepw
```
MP_PW_DEF=/opt/mp-home/keystorepw
```
MP_PW
In case you prefer to have your own generated password also for keystore, this option will interest you. As a value the location for the file is provided.
example of the value - the generated password will be saved to /opt/mp-home/keystorepw
```
MP_PW=/opt/mp-home/keystorepw
```
MP_CERT
In case a certificate has to be added to certstore the content of the variable will be processed. The list of certs is processed iteratively so one or more certificates can be provided.
- MP_KEYSTORE
  This variable has to be set in order to know where the keystore is stored - where the certificate should be added.
  example of the value to add the cert
  MP_CERT="-----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----" MP_KEYSTORE=/opt/mp-home/keystore.jceks
  exaple of the value to add more certs
  MP_CERT="-----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----" MP_KEYSTORE=/opt/mp-home/keystore.jceks

Prerequisites for the MP_CERT are MP_KEYSTORE & MP_PW or MP_KEYSTORE & MP_PW_DEF - it is necessary in order to locate and access / generation of the keystore.

container

In comparison to start, the midpoint application is run in foreground. The "normal" start is utilizing nohup and "send to background" (&) options. This behaviour is not acceptable as far as the container without extra setting would be handled as terminated (main PID would ends almost immediately). With keeping application in foreground the container is properly handled as running.

Starting the application

bash syntax
```
/opt/midpoint/bin/midpoint.sh container
```

docker syntax

command: [ "/opt/midpoint/bin/midpoint.sh", "container" ]

Docker-compose yaml file

The following docker-compose configuration file can be used to run a working environment with postgres DB backend including the native repository implementation. There is example with post-initial-object.

MP_ENTRY_POINT option is pointing to the folder in the container’s filesystem, which is handled as a read only source for (not only) post-initial-objects. The content is copied to proper midpoint’s structure (/opt/midpoint/var) before starting the midpoint instance with keeping the same subfolder structure. It can be also used for any other files like csv or jar (e.g. connectors).

During the processing of the MP_ENTRY_POINT it checks the existence of the file or file with extension .done (processed post-initial-object is renamed with suffix .done). Once the file in any form ("exact" name or with the .done suffix) exists, the file is skipped so any future changes on the copied version are kept without overwriting - only new files are copied. This way the post-initial-objects can be re-used several times with the same behaviour all the time.

example of the behaviour : MP_ENTRY_POINT=/opt/entry-point

/opt/entry-point
+ icf-connectors
| + connector-gitlab-rest-1.0.jar
| - connector-ssh-1.0.jar
|
+ post-initial-objects
| + user.xml
| - role.xml
|
- sources
  - hr.csv

/opt/midpoint/var
+ export
+ icf-connectors
| - connector-ssh-1.0.jar
+ idm-legacy
+ import
+ lib
+ log
| + midpoint.out
| - midpoint.log
+ post-initial-objects
| - user.xml.done
+ schema
- tmp

icf-connectors exists so no change based on the folder
connector-gitlab-rest-1.0.jar does not exist so it will by copied to /opt/midpoint/var/icf-connectors/connector gitlab-rest-1.0.jar - GitLab Connector
connector-ssh-1.0.jar exists in the destination so no action will happen - SSH Connector
post-initial-objects exists so no change
user.xml in the destination there exists user.xml.done so no action will happen
role.xml does not exist so it will be copied to /opt/midpoint/var/post-initial-objects/role.xml
sources folder does not exist so it will be created - /opt/midpoint/var/sources
hr.csv does not exist in the destination so it will be copied to /opt/midpoint/var/sources/hr.csv

In theory you can mount it directly to the midpoint’s structure but the resulting behaviour will be, the most probably, a little bit different than expected. With the first run there can be two possible situations:

the mount point will be in "writable" mode
In that case the file will be renamed with adding suffix ._done and respective next run (with new container) will be ignoring the files.
the mount point will be read-only mode
The midpoint start will fail and it will not be possible to rename the file, which is handled as critical error.

If MP_ENTRY_POINT feature is not needed the following lines can be removed:

     - MP_ENTRY_POINT=/opt/midpoint-dirs-docker-entrypoint

     - ./midpoint_server/container_files/mp-home:/opt/midpoint-dirs-docker-entrypoint/:ro

In case the lines are kept in the example, the folder ./midpoint_server/container_files/mp-home has to exist (or the path have to be changed to the existing one) otherwise docker-compose will fail to start.

The MP_VER variable is used in the config file. In case the variable is not set the "latest" is used as default. For the specific version you can set this variable before (or during) the docker-compose command start.

example of run with default (latest) version

docker-compose up

example of run with specific version

MP_VER=4.4 docker-compose up

example of "simple" environment with post-initial-objects & root CA cert used by Let’s encrypt

version: "3.3"

services:
  data_init:
    image: evolveum/midpoint:${MP_VER:-latest}
    command: >
        bash -c "
        chmod 777 /opt/db-pw/ ;
        touch /opt/db-pw/db_init_in_progress ;
	echo -e '#!/bin/sh\ntouch /opt/db-pw/db_init' >/opt/db-init/000-start.sh ;
        echo -e '#!/bin/sh\necho DB structure init process has finished...\nrm -f /opt/db-pw/db_init_in_progress /opt/db-pw/db_init' > /opt/db-init/999-finish.sh ;
        /opt/midpoint/bin/midpoint.sh init-native
        "
    environment:
     - MP_CHECK=/opt/db-pw/init_in_progress
     - MP_INIT_DB_CONCAT=/opt/db-init/010-init.sql
     - MP_INIT_CFG=/opt/mp-home
     - MP_DB_PW=/opt/db-pw/dbpassword
     - MP_PW_DEF=/opt/mp-home/keystorepw
     - MP_KEYSTORE=/opt/mp-home/keystore.jceks
     - |
       MP_CERT=-----BEGIN CERTIFICATE-----
       MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAw
       TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
       cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4
       WhcNMzUwNjA0MTEwNDM4WjBPMQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJu
       ZXQgU2VjdXJpdHkgUmVzZWFyY2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBY
       MTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK3oJHP0FDfzm54rVygc
       h77ct984kIxuPOZXoHj3dcKi/vVqbvYATyjb3miGbESTtrFj/RQSa78f0uoxmyF+
       0TM8ukj13Xnfs7j/EvEhmkvBioZxaUpmZmyPfjxwv60pIgbz5MDmgK7iS4+3mX6U
       A5/TR5d8mUgjU+g4rk8Kb4Mu0UlXjIB0ttov0DiNewNwIRt18jA8+o+u3dpjq+sW
       T8KOEUt+zwvo/7V3LvSye0rgTBIlDHCNAymg4VMk7BPZ7hm/ELNKjD+Jo2FR3qyH
       B5T0Y3HsLuJvW5iB4YlcNHlsdu87kGJ55tukmi8mxdAQ4Q7e2RCOFvu396j3x+UC
       B5iPNgiV5+I3lg02dZ77DnKxHZu8A/lJBdiB3QW0KtZB6awBdpUKD9jf1b0SHzUv
       KBds0pjBqAlkd25HN7rOrFleaJ1/ctaJxQZBKT5ZPt0m9STJEadao0xAH0ahmbWn
       OlFuhjuefXKnEgV4We0+UXgVCwOPjdAvBbI+e0ocS3MFEvzG6uBQE3xDk3SzynTn
       jh8BCNAw1FtxNrQHusEwMFxIt4I7mKZ9YIqioymCzLq9gwQbooMDQaHWBfEbwrbw
       qHyGO0aoSCqI3Haadr8faqU9GY/rOPNk3sgrDQoo//fb4hVC1CLQJ13hef4Y53CI
       rU7m2Ys6xt0nUW7/vGT1M0NPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNV
       HRMBAf8EBTADAQH/MB0GA1UdDgQWBBR5tFnme7bl5AFzgAiIyBpY9umbbjANBgkq
       hkiG9w0BAQsFAAOCAgEAVR9YqbyyqFDQDLHYGmkgJykIrGF1XIpu+ILlaS/V9lZL
       ubhzEFnTIZd+50xx+7LSYK05qAvqFyFWhfFQDlnrzuBZ6brJFe+GnY+EgPbk6ZGQ
       3BebYhtF8GaV0nxvwuo77x/Py9auJ/GpsMiu/X1+mvoiBOv/2X/qkSsisRcOj/KK
       NFtY2PwByVS5uCbMiogziUwthDyC3+6WVwW6LLv3xLfHTjuCvjHIInNzktHCgKQ5
       ORAzI4JMPJ+GslWYHb4phowim57iaztXOoJwTdwJx4nLCgdNbOhdjsnvzqvHu7Ur
       TkXWStAmzOVyyghqpZXjFaH3pO3JLF+l+/+sKAIuvtd7u+Nxe5AW0wdeRlN8NwdC
       jNPElpzVmbUq4JUagEiuTDkHzsxHpFKVK7q4+63SM1N95R1NbdWhscdCb+ZAJzVc
       oyi3B43njTOQ5yOf+1CceWxG1bQVs5ZufpsMljq4Ui0/1lvh+wjChP4kqKOJ2qxq
       4RgqsahDYVvTH9w7jXbyLeiNdd8XM2w9U/t7y0Ff/9yi0GE44Za4rF2LN9d11TPA
       mRGunUHBcnWEvgJBQl9nJEiU0Zsnvgc/ubhPgXRR4Xq37Z0j4r7g1SgEEzwxA57d
       emyPxgcYxn/eR44/KJ4EBs+lVDR3veyJm+kXQ99b21/+jh5Xos1AnX5iItreGCc=
       -----END CERTIFICATE-----
    volumes:
     - db_init:/opt/db-init
     - db_pw:/opt/db-pw
     - midpoint_home:/opt/mp-home

  midpoint_data:
    image: postgres:13-alpine
    command: >
        bash -c "
        while [ ! -s /opt/db-pw/dbpassword -o -e /opt/db-pw/init_in_progress ] ; do
         echo 'Waiting to the end of the init process...';
         sleep 1;
        done ;
        {
         sleep 2 ;
         if [ ! -e /opt/db-pw/db_init -a -e /opt/db-pw/db_init_in_progress ] ;
          then echo 'DB init did not start...' ;
          rm -f /opt/db-pw/db_ini*;
          echo 'The lock files has been removed...';
         fi ;
        } &
        docker-entrypoint.sh postgres
        "
    user: "70:70"
    depends_on:
     - data_init
    environment:
     - POSTGRES_PASSWORD_FILE=/opt/db-pw/dbpassword
     - POSTGRES_USER=midpoint
     - POSTGRES_INITDB_ARGS=--lc-collate=en_US.utf8 --lc-ctype=en_US.utf8
    ports:
     - 5432:5432
    networks:
     - net
    volumes:
     - midpoint_data:/var/lib/postgresql/data
     - db_init:/docker-entrypoint-initdb.d/
     - db_pw:/opt/db-pw

  midpoint_server:
    image: evolveum/midpoint:${MP_VER:-latest}
    depends_on:
     - data_init
     - midpoint_data
    command: [ "/opt/midpoint/bin/midpoint.sh", "container" ]
    ports:
      - 8080:8080
    environment:
     - MP_CHECK=/opt/db-pw/db_init_in_progress
     - MP_SET_midpoint_repository_jdbcUsername=midpoint
     - MP_SET_midpoint_repository_jdbcPassword_FILE=/opt/db-pw/dbpassword
     - MP_SET_midpoint_repository_jdbcUrl=jdbc:postgresql://midpoint_data:5432/midpoint
     - MP_SET_midpoint_keystore_keyStorePassword_FILE=/opt/midpoint/var/keystorepw
     - MP_UNSET_midpoint_repository_hibernateHbm2ddl=1
     - MP_NO_ENV_COMPAT=1
     - MP_ENTRY_POINT=/opt/midpoint-dirs-docker-entrypoint
    networks:
     - net
    volumes:
     - midpoint_home:/opt/midpoint/var
     - db_pw:/opt/db-pw
     - ./midpoint_server/container_files/mp-home:/opt/midpoint-dirs-docker-entrypoint/:ro

networks:
  net:
    driver: bridge

volumes:
  db_init:
  db_pw:
  midpoint_data:
  midpoint_home: