CSVFile Connector Migration

Last modified 23 Mar 2021 17:18 +01:00
OBSOLETE
This functionality is obsolete. It is no longer supported or maintained.

This information is applicable for midPoint 3.6. It describes migration from midPoint 3.5.x CSVFile connector to midPoint 3.6 CSV connector.

Synopsis

Up until version 3.6 midPoint included a legacy CSVFile connector. The legacy CSVFile Connector was replaced by* *new CSV Connector in midPoint 3.6. The new CSV connector is a rewrite from scratch. The old CSVFile connector was written even before midPoint project started and it was not designed for real deployment use. We have maintained and improved the connector during the years. But it was not maintainable any more. Also the ConnId framework evolved over the time and we needed a connector that will use these features. Therefore we have decided to rewrite the connector completely. MidPoint 3.6 no longer bundles the old connector. New CSV connector is bundled instead. Old CSV connector can still be used and it is still supported for deployments that purchased midPoint subscription before midPoint 3.6 was released. As the old connector is not bundled with midPoint any more you have to download the connector JAR and deploy it explicitly.

Migration: New CSV Connector

The new CSV connector is very similar in its operation and configuration as the old CSVFile connector. However it is not entirely backwards compatible with the old connector. There are several major differences:

  • connector name and bundle

  • some connector configuration properties are different and some are missing

  • attribute names: native column names are used instead of icfs:name and icfs:uid

  • the new connector does not support script execution yet

Connector Name and Bundle

The new CSV connector is using bundle and connector name from com.evolveum.polygon namespace. The configuration XML namespace also reflects that:

Bundle name (connectorBundle) com.evolveum.polygon.connector-csv

Connector name (connectorType)

com.evolveum.polygon.connector.csv.CsvConnector

Configuration namespace

http://midpoint.evolveum.com/xml/ns/public/connector/icf-1/bundle/com.evolveum.polygon.connector-csv/com.evolveum.polygon.connector.csv.CsvConnector

Connector Configuration

CSV Connector has configuration schema similar to old connector, but there are differences. The new configuration properties are summarized below:

Property name Label Description

multivalueDelimiter

Multivalue delimiter

Multivalue delimiter character user for splitting multivalue attributes.

commentMarker

Comment marker

Comment marker character.

filePath

File path

Path to CSV file with records.

quoteMode

Quote Mode

What field should be quoted. Allowed values are ALL, MINIMAL, NON\_NUMERIC, NONE.

ignoreEmptyLines

Ignore empty lines

Whether connector should ignore empty lines in CSV file.

trailingDelimiter

Trailing delimiter

Whether connector should use trailing delimiter.

encoding

Encoding

CSV file encoding.

fieldDelimiter

Field delimiter

Delimiter character between fields in one CSV record (escaped as regexp, if needed).

ignoreSurroundingSpaces

Ignore surrounding spaces

Whether connector should ignore surrounding spaces.

quote

Quote

Quote character.

trim

Trim

Whether fields should be trimmed.

headerExists

Header exists

Whether header exists in csv file.

passwordAttribute

User password attribute name

User password attribute name is not required. Its used only in authenticate operation.

tmpFolder

Tmp folder

Folder where csv connector can write temporary files. By default the same folder as where csv file resides.

readOnly

Read only

Whether file is for read only acces only. Default is false.

preserveOldSyncFiles

Preserve old sync files

How many old sync filed do we want to preserve.

escape

Escape

Escape character.

nameAttribute

objectClassDefinition

Object class definition

File which contains definitions for other object classes.

recordSeparator

Record separator

Record separator string.

uniqueAttribute

Unique attribute

Unique attribute name.

Configuration samples for CSV resources were updated to the new CSV connector. The Configuration Samples are located at the usual places.

Shadows

The shadows in the old connector looked like this:

Shadow: Legacy CSVFile connector
<shadow>
  <objectClass>AccountObjectClass</objectClass>
  ...
  <attributes>
    <icfs:name>007</icfs:name>
    <icfs:uid>007</icfs:uid>
    <ri:username>foo</ri:username>
    <ri:fullname>Foo Bar</ri:fullname>
    ...
  </attributes>
</shadow>

Shadow for new connector looks like this:

Shadow: New CSV connector
<shadow>
  <objectClass>AccountObjectClass</objectClass>
  ...
  <attributes>
    <ri:guid>007</ri:guid>
    <ri:username>foo</ri:username>
    <ri:fullname>Foo Bar</ri:fullname>
    ...
  </attributes>
</shadow>

The new CSV connector is using native names of attributes and object classes and therefore it has obviously cleaner and better data structure. But this data structure is different and currently there is no way how to automatically transform the shadows.

There is a known issues in midPoint that prohibits the use of native attribute with the name of 'id'. (MID-3872). If the attribute name in the CSV file cannot be changed then the workaround is to force the use of legacy schema. In that case midPoint will use the legacy ConnId attribute names (icfs:name and icfs:uid) even with the new CSV connector.

Migration steps

We recommend the following migration procedure:

  1. Add resource that will use the new CSV connector. Configure it as a completely new resource using the configuration equivalent to the old one.

  2. Change assignment enforcement level to a permissive value (none or positive).

  3. Set up a correlation expression to correlate users with the CSV accounts. Reconcile the new CSV resource. The result should be that the CSV accounts on the new resource are linked.

  4. Modify definitions of role and/or direct assignments to point to the new CSV resource instead of the old one. Resource reference (resourceRef) needs to be changed, but also any mappings for identifier attributes (icfs:name and icfs:uid in the old connector).

  5. Delete old CSVFile resource and all shadows that belong to that resource (there is now an option to do this efficiently in the Repository Objects GUI page).

  6. Recompute the users. This should remove the orphaned linkRefs in user objects.

  7. Double-ckeck that every thing is switched to the new resource (roles, assignments, shadows exist and are linked to users).

  8. Change assignment enforcement level to the original.

Migration: Legacy CSVFile Connector

The legacy CSVFile connector is still available and it still can be used. This avoids the need for data (shadow) migration.

To use the legacy CSVFile connector in midPoint 3.6 or later please follow these steps:

  1. Download JAR of the legacy CSVFile connector and deploy it into midPoint

  2. In all the resource definitions change connectorRef to point to the newly discovered legacy CSVFile connector.

  3. No change in configuration, shadows or roles is needed.

The legacy CSVFile connector will be maintained for a reasonable migration period which mostly depends on the requirements of midPoint subscribers. After that period the legacy connector will no longer be supported. Therefore please plan the migration to the new connector accordingly.