Starting with new repository

Last modified 22 Jul 2021 11:21 +02:00
This is work-in-progress document for feature still in development!

What is similar and what is different when configuring midPoint for the new repository?

  • New repository supports only PostgreSQL database, preferably version 13, 12 is possible too. The database cannot be embedded and must be prepared as documented here.

  • Different schema SQL files are used, so stop before the Run database script step.

  • Basic repository configuration documented here is working as before.

  • New repo doesn’t use Hibernate, so any Hibernate related configuration is ignored.

  • Main difference in config.xml is to use <type>sqale</type> instead of <repositoryServiceFactoryClass> element. Alternatively, just set the value of <repositoryServiceFactoryClass> element to sqale.

  • At this moment, SQL audit is not supported and must be commented out.

Database setup

Installation

This guide does not cover the installation process as there are many possible combinations. To install PG 13 on Ubuntu 20.04 one can use these steps for an inspiration. Adjust the setup in pg_hba.conf to the real IP address of the server. Setup can be different if PG is used only on localhost, but we assume host-to-host communication which is typical for production setup.

The short checklist:

  • Install PostgreSQL 13 on your OS or server or VM.

  • Setup listen_addresses = '*' in postgresql.conf.

  • While in postgresql.conf it’s also good to add statements statistics extension and query logging for better visibility (the latter may not be a good option for production and small disks). See the snippet lower.

  • Setup host…​md5 line in pg_hba.conf, otherwise you (and JDBC driver) will not be able to connect to the database remotely.

  • Restart PostgreSQL.

In short, this can all be added to the end postgresql.conf (then restart the server):

listen_addresses = '*'

# this is necessary for pg_stat_statements extension
shared_preload_libraries = 'pg_stat_statements'

# this is to log all the queries, just be aware of the free disk space
log_directory = 'pg_log'
log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'
log_statement = 'all'
logging_collector = on

Database preparation

This roughly copies the docs for old repository.

First command is to be executed in bash on the server with Postgres, the rest is executed inside psql:

sudo -i -u postgres psql

CREATE USER midpoint WITH PASSWORD 'password' LOGIN SUPERUSER;
CREATE DATABASE midpoint WITH OWNER = midpoint ENCODING = 'UTF8' TABLESPACE = pg_default LC_COLLATE = 'en_US.UTF-8' LC_CTYPE = 'en_US.UTF-8' CONNECTION LIMIT = -1;

To initialize the database connect to midpoint database as midpoint user and execute the content of repo/repo-sqale/sql/pgnew-repo.sql file from midPoint sources. The location of the file will likely change, and it will be packed in the distribution as well. You can use any client to do this, or we can download the file on the VM and use it like this:

wget -q https://raw.githubusercontent.com/Evolveum/midpoint/master/repo/repo-sqale/sql/pgnew-repo.sql
# without this export psql will prompt for the password
export PGPASSWORD=password
psql -h localhost midpoint midpoint -f pgnew-repo.sql

If you plan to use statement statistics extension (not discussed here), initialize it like this:

psql -h localhost midpoint midpoint -c "create extension pg_stat_statements"

Quartz tables

Quartz scheduler in midPoint can be configured to use a database with taskManager/jdbcJobStore option in config.xml set to true. This is also the default if clustered is set to true. See Task Manager Configuration for further details.

In that case Quartz requires that its tables are ready in the database. By default, the same database is used for repository and for Quartz, which can be changed by using jdbcUrl and other options inside taskManager section (see the link above).

Let’s simplify things and assume that we need the tables, and the same database is used. Connect to the midpoint database with midpoint user and execute the commands stored in repo/task-quartz-impl/src/main/resources/com/evolveum/midpoint/task/quartzimpl/execution/tables_postgres.sql from the midPoint sources.

The following commands can be used in the bash on the database VM:

wget -q https://raw.githubusercontent.com/Evolveum/midpoint/master/repo/task-quartz-impl/src/main/resources/com/evolveum/midpoint/task/quartzimpl/execution/tables_postgres.sql
psql -h localhost midpoint midpoint -f tables_postgres.sql

Example config.xml

Example config.xml is here. The main difference is type element (shorthand for repositoryServiceFactoryClass) set to sqale and removal of SQL audit - which is not implemented for new repo yet.

With this config.xml you can start midPoint as usual.

Running tests with new repository

New repository has its own tests in repo-sqale module. Tests from repo-sql-impl-test are not usable for new repository at all, but the module contains support for running later integration tests with the new repository.

To run integration tests (e.g. from model-intest or story modules) with the new repo, you must provide this JVM argument to Maven of IDEA run configuration: -Dtest.config.file=test-config-new-repo.xml

Further -Dmidpoint.repository.jdbc*=…​ arguments can be added to point to the non-default database. Check the test-config-new-repo.xml in midPoint project for the default values.

Dedicated Postgres database must be running just like for normal operation.

Vagrantbox with new DB

If you’re a friend with Vagrant, you can use this Vagrant box prepared for new repository with Postgres 13. Just check the IP of the VirtualBox adapter on the host and adjust the IPs in Vagrantfile and provided config.xml.