Shadow Caching

Last modified 08 Nov 2024 11:49 +01:00

Attribute caching feature

This page is an introduction to Attribute caching midPoint feature. Please see the feature page for more details.

MidPoint usually works with fresh data. When midPoint needs to get data about an account then the data will be retrieved on-demand from the resource. This is usually the best method. But there are cases when this approach is problematic. There may be resources that are often down. There may be resources that are very slow. Yet others may be costly or limited regarding accessing them, typically when in cloud. Therefore, midPoint has an ability to cache the values of resource objects and use them instead of retrieving them from the resource.

Moreover, there is another benefit of having the data cached: instant reporting. With the relevant parts of data being cached right in the repository, one can write reports against them, usually allowing for more complex queries and providing faster execution times.

How It Works

The only supported caching method is passive caching: Caches are maintained with minimal impact on normal operations. Generally the data are cached only if they are retrieved for other reasons. There is no read-ahead. The writes are always going to the resource (synchronously): read-through, write-through. There is no cache eviction (but old information is overwritten if newer information is available).

If caching is turned on, which is by default for new midPoint installations, then midPoint will cache the data in Shadow Objects. The caches are build gradually as midPoint reads the objects. If you need to populate the caches with information you have to make an operation that is searching all the objects. E.g. reconciliation or similar operation should do the trick. See Refreshing the Cache below.

Configuring Caching

The caching can be enabled or disabled at the level of the whole system, a resource, an object class, an object type, or even an individual item (attribute, association, activation, and so on).

An Example

For a quick overview, let us consider the following example. (The complete description is below.)

Listing 1: An example of custom caching configuration

<resource>
    ...
    <schemaHandling>
        <objectType>
            <kind>account</kind>
            <intent>default</intent>
            ...
            <attribute>
                <ref>ri:jpegPhoto</ref>
                <cached>false</cached> (4)
            </attribute>
            ...
            <caching>
                <cachingStrategy>passive</cachingStrategy> (1)
                <scope>
                    <attributes>all</attributes> (2)
                    <associations>none</associations> (3)
                </scope>
                <timeToLive>P1D</timeToLive>
                <defaultCacheUse>useCachedOrFresh</defaultCacheUse>
            </caching>
        </objectType>
    </schemaHandling>
    ...
    <caching>
        <cachingStrategy>none</cachingStrategy> (5)
    </caching>
    ...
</resource>

1	Enables the caching for `account/default` object type.
2	Enables the caching of all attributes.
3	Disables the caching of associations.
4	Overrides the caching for a specific attribute.
5	Disables the caching for all other objects.

Shortly speaking, the caching is disabled for the resource as a whole, except for the account/default type, for which it is enabled: Cached are all attributes except for jpegPhoto, activation, but no associations.

Configuration Details

At the level of resource, object class, or object type definition, the caching configuration item can be provided. It has the following sub-items:

Table 1. The `caching` configuration item content
Item	Description	Default	Default for caching-only read capability
`cachingStrategy`	The overall switch that turns the caching on and off. It can have a value of `none` (no caching) and `passive` (passive caching as described above).	`passive`
`scope`	Scope of the caching (see below).	see below
`timeToLive`	For how long we consider the cached data as fresh.	P1D	unlimited
`defaultCacheUse`	How is the cache used for midPoint operations.	`fresh`

Table 2. The `scope` configuration item content
Item	Description	Default	Default for caching-only read capability
`attributes`	Scope of the caching for simple attributes (i.e., not reference ones)	`defined`	`all`
`associations`	Scope of the caching for associations (i.e., reference attributes).	`all`
`activation`	Scope of the caching for the activation information.	`all`
`auxiliaryObjectClasses`	Scope of the caching for the auxiliary object classes.	`all`
`credentials/password`	Scope of the caching for the password.	`all`	`none`

Table 3. The values of the `scope` configuration items for simple attributes
Value	Description
`none`	No items of given kind will be cached.
`defined`	Only attributes defined (refined) in the `schemaHandling` section will be cached.
`mapped`	Only mapped items of given kind, i.e., those that have any mapping defined right in the object type, either inbound or outbound, will be cached.
`all`	All items of given kind will be cached.

Table 4. The values of the `scope` configuration items for other items (associations, activation, and so on)
Value	Description
`none`	The data are not cached.
`all`	The data are cached.

Exceptions (both positive and negative) to the scope can be defined by using cached boolean property present for individual attributes.

System-wide defaults for the caching can be set in the system configuration. This is the default setting for all new installation, taken from the initial object holding the system configuration:

Listing 2: Setting the default values for the shadow caching

<systemConfiguration>
    ...
    <internals>
        <shadowCaching>
            <defaultPolicy>
                <cachingStrategy>passive</cachingStrategy> (1)
            </defaultPolicy>
        </shadowCaching>
        ...

1	Enables the shadow caching for all resources as a default. The defaults for individual values, e.g., time to live, can be overridden here as well.

When changing the defaults in the system configuration, you need to apply changed values to all resource. The easiest way how to do that is to restart midPoint (all nodes in the cluster). An alternative is to make (any) change to each resource. For example, you can let midPoint to reload the schema, or put the resource into maintenance mode and back, and so on. We plan to resolve this issue in the near future.

Configuring Cache Use

Even with caching turned on, the data may or may not be used for regular midPoint operation. For example, import from a resource may run against the actual resource data, or against the cached shadows in the repository. In a similar way, user recomputation may also use the actual or cached data. When someone opens a user projection in GUI, the cached or actual data can be displayed. And so on.

Use of Cached Data for Data Processing

This section describes the use of cached data for data processing in midPoint.

This primarily means providing source data for inbound mappings. But it also means providing the idea of what data are on resources when outbound mappings are concerned, e.g., for the correct application of weak and strong mappings.

What it does not cover, though, is the primary input of synchronization tasks, like import and reconciliation. It is taken always from the resource. (Unless turned off by an experimental setting described in Use of Cached Data for Import and Reconciliation Tasks section below.)

So, the data processing is driven by the defaultCacheUse configuration property. It can have the following values:

Value Description

Value	Description
`useFresh`	When we need the data from the shadow, we will fetch it from the resource. This is the same behavior as it was before midPoint 4.9.
`useCachedOrFresh`	When we need the data from the shadow, we will use the cached data, if they are available. If they are not, we will fetch them from the resource.
`useCachedOrIgnore`	When we need the data from the shadow, we will use the cached data, if they are available. If they are not, we will skip the respective part of the processing (e.g., a mapping). Experimental.
`useCachedOrFail`	When we need the data from the shadow, we will use the cached data, if they are available. If they are not, we will fail. Experimental.

useFresh

When we need the data from the shadow, we will fetch it from the resource. This is the same behavior as it was before midPoint 4.9.

useCachedOrFresh

When we need the data from the shadow, we will use the cached data, if they are available. If they are not, we will fetch them from the resource.

useCachedOrIgnore

When we need the data from the shadow, we will use the cached data, if they are available. If they are not, we will skip the respective part of the processing (e.g., a mapping).

Experimental.

useCachedOrFail

When we need the data from the shadow, we will use the cached data, if they are available. If they are not, we will fail.

Experimental.

For given task, the default behavior can be overridden using cachedShadowsUse model execution option (if that task does support specifying these options). However, this setting is currently experimental.

Use of Cached Data for Import and Reconciliation Tasks

The import and reconciliation tasks fetch their primary input, i.e., resource objects, right from the resource they are running against. It has a benefit of consistently update the cache for these objects.

There may be situations, though, when you’d need to avoid contacting the resource, and take data from the cache. It can be attained by specifying noFetch option, like this:

Listing 3: Sample reconciliation task running from the shadow cache

<task xmlns="http://midpoint.evolveum.com/xml/ns/public/common/common-3"
    oid="007c5ef2-3d1f-4688-a799-b735bbb9d934">
    <name>reconcile-hr-persons</name>
    <executionState>runnable</executionState>
    <activity>
        <work>
            <reconciliation>
                <resourceObjects>
                    <resourceRef oid="c37ff87e-42f1-46d2-8c6f-36c780cd1193"/>
                    <kind>account</kind>
                    <intent>person</intent>
                    <searchOptions>
                        <option>
                            <options>
                                <noFetch>true</noFetch> (1)
                            </options>
                        </option>
                    </searchOptions>
                </resourceObjects>
            </reconciliation>
        </work>
    </activity>
</task>

1	Specifies that the resource should not be contacted.

For the "remaining shadows" activity of reconciliation, where shadows that are presumably dead, are reconciled, we always go to the resource. If you need this behavior to be avoided, just turn off that activity.

This feature is experimental.

Refreshing the Cache

The best way of refreshing the cache is to run an import or reconciliation task. However, if you want to avoid the cost of data processing inherent in these tasks, you can use the following alternative:

Listing 4: A task that just refreshes the shadow cache

<task xmlns="http://midpoint.evolveum.com/xml/ns/public/common/common-3"
      oid="45012c3e-3ce5-46ed-8d27-8648d2cbbca0">
    <name>Reload objects on HR resource</name>
    <ownerRef oid="00000000-0000-0000-0000-000000000002" type="UserType"/>
    <executionState>runnable</executionState>
    <activity>
        <work>
            <import>
                <resourceObjects>
                    <resourceRef oid="42a11d38-afbb-4f0e-8aea-c848db8ba0ab"/> <!-- HR -->
                    <kind>account</kind>
                    <intent>default</intent>
                </resourceObjects>
            </import>
        </work>
        <execution>
            <mode>none</mode> (1)
        </execution>
    </activity>
</task>

1	Avoids data processing.

The same effect can be attained by clicking on Reload button on the resource page for accounts, entitlements, or generics.

Impact on API Operations

The cached data are accessible by using the usual IDM Model Interface. There are two operation options that provide access to the cached data:

noFetch option: This option returns the data from midPoint repository. Therefore, if there are data cached in the repository then the noFetch option returns them.
staleness option: Requirement how stale or fresh the retrieved data should be. It specifies maximum age of the value in milliseconds. The default value is zero, which means that a fresh value must always be returned. This means that caches that do not guarantee fresh value cannot be used. If non-zero value is specified then such caches may be used. In case that Long.MAX_VALUE is specified then the caches are always used and fresh value is never retrieved.

Both options can be used to get cached data. The primary difference is that the noFetch option never goes to the resource, and it returns whatever data are in the repository. On the other hand, the staleness option is smarter, and it determines whether it has to go to the resource or not. In case that the "maximum" staleness option is used it will result in an error if cached data is not available.

Those options can be used both with getObject operations and search operations. For getObject the staleness option work as expected. But there is one special consideration for the search operations. The search operations cannot easily determine how fresh the data in the repository are. E.g. there may be new objects on the resource that are not in the repository. Therefore, to be on the safe side the search operations will always make search on the resource even if staleness option is specified. There is just one exception: the maximum staleness option will force repository search. However, if the search discovers any object that does not have cached data then it will result in an error (specified in the fetchResult object property).

Caching Metadata in Returned Shadows

Shadow Objects contain cachingMetadata property. This property can be used to determine whether the returned shadow represents fresh or cached data:

If no cachingMetadata property is present in the shadow then the data are fresh. They have been just retrieved from the resource.
If cachingMetadata property is present then the data are taken from the cache. The cachingMetadata property specified how fresh the data are (when they were originally retrieved).

Relation to the "Caching-Only" Read Capability

When the "caching only" read capability is present (e.g., for manual resources), the full shadow caching is enabled, with the following differences in default values (comparing to the standard caching):

Table 5. Default values differences for "caching-only" read capability
Item	Standard default value	Default when turned on by "caching-only" capability
`timeToLive`	P1D	unlimited
attributes caching scope	`defined`	`all`
password caching scope	`all`	`none`

The caching for "caching only" read capability can be turned off by specifying cachingStrategy to none.

The defaults above can be turned back to standard default values (the second column) by specifying cachingStrategy of passive.

Limitations

For Both Native And Generic (Oracle, SQL Server) Repository

Attributes whose values are provided by the resource should have caching turned off. For example, the uid attribute in LDAP can be derived from the dn attribute (if it’s in the form of uid=xxx,…). In such cases, midPoint could mistakenly assume that the value of this attribute is missing for new accounts being created, when useCachedOrFresh approach to data retrieval is used.

The solution is either to turn their caching off (by setting cached property to false for those attributes), or to make sure they are explicitly provided by midPoint each time their source (e.g., dn for uid) is created or modified. You can do that by providing an explicit mapping for, e.g., uid attribute along with the mapping for dn.
Only hashed or "present vs not present" storage of cached passwords is supported.
Localized attributes (see MID-5210) are not cached yet.

For Generic Repository (Oracle, SQL Server) Only

values cannot be larger than 255 characters;
values cannot be empty (on Oracle);
illegal XML characters (like those with codes under 32) cannot be present in string values.

If you have attributes that do not conform to these restrictions, please either turn off their caching (using cached property with the value of false), or use the experimental storageStrategy of notIndexed.

"Use Cached" setting

Please turn on useCachedOrFresh setting with care. There may be unexpected differences to the pre-4.9 behavior, mainly related to inbound mappings being executed, because of the data being available from the cache. On the other hand, in some cases, inbound mappings for some attributes may be skipped, because the shadow may not be loaded - because of caching. That way or another, please make sure the system behaves as expected with useCachedOrFresh setting.

Migration Note

Before 4.9, this feature was experimental. The default setting was that all attributes and no associations were cached.

Since 4.9, the defaults are more elaborate, as described in this document. Please take that into account when migrating.