Usual Troubleshooting Steps
This page describes the usual steps to try when something does not work as expected.
Error Message
MidPoint has quite a nice system of error messages. In addition to usual text message they contain a lot more information. The error messages are in fact dialogs that can be expanded by clicking on them. They show a hierarchical internal structure of the operation indicating the result of each important step. Each of the step can be further expanded to show more information. This should usually provide all the information that is needed to diagnose the problem.
But there are situations when the error message is not enough. E.g. the operation completes without an error but it does not produce an expected result. Or the operation is not executed through the GUI. In such cases there are other steps.
Log Files
MidPoint log files record activity of midPoint on several levels. The next step in troubleshooting should be to have a look at log files. All the errors and warnings should be logged there by default. This should usually provide enough information to assist in troubleshooting serious misconfigurations or other severe problems. The Useful Loggers page provides some hints how to get the most useful information in the log files.
The logfiles are usually placed in the logs directory of you application server or web container.
The logging configuration can be managed in Configuration > Logging GUI page. The dialog allows to set many logging options. The meaning of individual levels is described in Log Levels page. But there is rule of the thumb for troubleshooting:
-
FATAL,ERROR,WARNINGandINFOlevels are designed to be enabled during normal system operation. They may be safely enabled for any component. This should provide enough information to roughly locate any problem. -
DEBUGlevel is intended to troubleshoot miscofigurations and similar problems. This should provide enough information for midPoint administrator to figure out what is wrong. This provides a lot of information and should not be enabled all the time. Even when hunting down a problem it is advised to enableDEBUGonly for a few components to keep the noise down. -
TRACElevel is intended for developers: either developers of midPoint or developers of midPoint plugins. TheTRACElevel is using software development slang, class names and other ugly things. If a system administrator needs to go as deep asTRACElevel then something is wrong with the logging setup. MidPoint team will appreciate if you let them know so they can improve it.
Clockwork and Projector Troubleshooting
Most midPoint problems occur in mapping configuration, roles definition and similar policy-like statements. All of these are processed in Clockwork and Projector components in midPoint. These components are especially designed to provide detailed diagnostic output to assist in tracing down the configuration bugs.
The most efficient troubleshooting method is to watch changes in Model Context. This can be done by enabling the tracing in clockwork, projector, mapping and expression components (see below). The usual steps go like this:
-
Enable clockwork summary tracing (logger
com.evolveum.midpoint.model.lens.Clockwork, level DEBUG). This will dump Model Context during major state changes in clockwork. Especially watch how primary or synchronization deltas transform to secondary deltas. Primary and synchronization deltas usually act as an input to the Clockwork and Projector. Secondary deltas are the outputs. Look for any errors and warnings between the Model Context dumps. -
If the previous step does not help then enable projector summary tracing (logger
com.evolveum.midpoint.model.lens.projector.Projector, levels DEBUG and TRACE). This will dump the Model Context several times at strategic places during projector computation. You can use the trace to figure out where exactly is the problem. E.g. if you expect that some attribute will be mapped by the inbound mapping, then check the presence of appropriate delta in Model Context dump after the run of InboundProcessor. This step will almost always provide enough information to track down the problem. -
If the previous step does not help and you suspect a problem in one of the mappings then enable mapping trace (logger
com.evolveum.midpoint.model.common.mapping.Mapping, logger TRACE). This will dump inputs and outputs of all the processed mappings. This will mean a lot of data in the log files. A good method is to set a name for the mapping that you suspect and then search for that name in the log file. This will make finding the right part of the logfile easier. -
If you figured out that the mapping output is wrong, but you do not know why then the next step would be do enable expression trace (logger
com.evolveum.midpoint.model.common.expression.Expression, level TRACE). This will provide detailed dump of all the inputs and outputs of an expression. -
If the steps 1 and 2 do not help and you suspect that the problem is in general projector or policy set up, enable detailed projector trace (logger
com.evolveum.midpoint.model.lens.projector, level TRACE) or a complete lens component trace (loggercom.evolveum.midpoint.model.lens, level TRACE). This is a very detailed dump and it goes into the area of developer information. But it can be very helpful when tracing down especially wicked configuration bugs.
Bug Report
MidPoint is well tested, we have reasonably good test suite with thousands of automated tests that are continuously executed. MidPoint is also very flexible, configurable, extensible and customizable. It is easy to make a configuration mistake. Therefore if you experience a problem it is most likely a configuration problem and not a product bug. But no software is ever perfect and bugs happen. If you have gone through all the relevant steps above and you suspect that what you see is not misconfiguration, error caused by external source (e.g. resource being down) or other thing that you can correct then it is a time to follow with a bug report. Please see the Creating a Bug Report page which describes how to create a good bug report.
Aggressive Approach
There are some occasions when all these steps are not enough to diagnose the problem. Such (relative rare) occasions usually hits about a programming bug: wrong exception handling, wrong error reporting a wild variety (usually stupid) programming mistakes. This is a time for a brutal solution. Please see Aggressive Log Report for more instructions.