Log Viewer

Log Viewer can provide a tree view of a (large) log file, giving quick overview of its content and allowing to drill down to details we are interested in. The viewer provides a default configuration showing some standard levels. However, it also allows to change their arrangement, as well as to add new outlining rules based on substring and regular expressions matching.

Standard levels are:

An example - showing the result of the workflow module tests:

We can see there are four system runs. For example, the first one starts at 14:10:35 and is covered by lines 117-196689. After clicking on a line in the outline the cursor in main window jumps to appropriate log line.

Now we can explore the system run by expanding e.g. the first system startup node. We see there is a couple of test methods executed.

So we select e.g. the test080 and expand its node. There is one operation execution; after expanding it we see the following:

There is a lot of context dumps, as the operation traverses through specific execution stages. Columns have the following means:

Threads can be especially problematic when analyzing log files. The outline is currently not adapted to reasonably display output from more than one thread at once (i.e. the tree structure and related information will most probably look like nonsense). That’s why we show Thread column in the outline. The user can then simply select which threads he wants to display in the outline by using %show-in-outline instruction (in the future, also GUI option will be provided).

In this particular case mixing threads is not a problem, because during workflow tests first part of processing (INITIAL stage) occurs in the context of calling thread, but other parts (starting with PRIMARY stage) occur in the background; the serialized nature of processing is preserved.

Other levels are similar to this:

Sometimes it is useful to see whether something went wrong during midPoint execution. Traditionally we search through the logfile for strings like "ERROR", "Exception" etc., browsing through false alarms to find the cause of a problem we fight with.

Log viewer can display errors, and even warnings and selected information messages in Eclipse "Problems" view as errors, warnings and infos. Like this:

In order to eliminate false alarms, this feature is configurable, with the default of:

Rules 1-3 inhibit often-occurring errors. Rule 4 tells the viewer to display log errors as Eclipse errors. Warnings and information messages are not shown by default.

Another example could be:

The result would be that all lines with the given exception are show as warnings. Second rule states that lines with given OID which are not a part of profiling logging entries (those that have "PROFILING" in the first line of the log entry) are shown as infos.

Another example: diagnosing failed TestSecurity.test204 test: wanting to quickly browse through security-related subtests. So added this

and got this:

A special case is marking all lines that took longed than N milliseconds to appear either as errors, warnings or infos. This is quite rudimentary way of diagnosing potential or real performance problems.

An example:

Structures in log files - like context dumps - can be quite extensive and complex. It is useful to be able to collapse and later expand back parts of these structures.

There are three kinds of folding available:

An example of indented structure folding (hiding user old & current, keeping user new):

An example of collapsed entry-exit pair #74 (provisioning getObject call):

Folding can be also done declaratively. E.g. we could tell Log Viewer to automatically collapse all log entries related to ResourceManager:

Or collapse all profiling information related to calls Provisioning→getObject:

Sometimes there are lines that clutter the log and we know we won’t need them any more. We can delete them using %kill instruction, like this:

The algorithm to remove individual lines can be improved - in current state killing many lines in large files can take a quite long time (minutes).

Log viewer scans through the log file and automatically extracts a list of known objects as well as a list of threads. It looks like this:

Each line contains an OID, object type (e.g. account), name (e.g. elaine) and color in which this OID has to be displayed (default). Name and color can be edited. This information is used to highlight these OID in the logfile, as well as to show object name by hovering the cursor over object OID in log file text.

And for threads:

This is fine for quick orientation, and for selection which threads should be included in/excluded from the outline, or even removed from the logfile completely.

As described above, log viewer displays OIDs in bold fond, and optionally with a different color.

For example, if we would like to trace occurrences of jack and barbossa, we could tell Log Viewer to display their OIDs in red and blue, respectively:

Then the log could look like this:

…​giving immediate visual information useful to quickly recognize relevant objects in the log file.

Also, when encountering unknown OID, you can just hover the cursor over it, and you’ll see the type and name of particular object (if known):

Large log files are awkward to process in test editors, especially in complex ones like Eclipse editor. So it may be useful to preprocess them by removing unimportant information.

LogTrimmer tool has the following usage:

Where:

Currently there are the following instructions available:

First one (trim) is used to shorten selected log entries. "Lines to keep" is the number of lines that should be kept from the log entry. Examples:

Second one (select-test) is used to preserve only given test (from a log that contains a set of tests). Example:

TODO - use similar definition language as in Log Viewer (e.g. %kill instruction).

On MS Windows systems it is quite hard to truncate open files. There is a PowerShell command to do that:

…​but on systems without PowerShell there is this Truncater tool to do the same:

Log viewer is a powerful tool to find certain classes of performance problems. To ensure completeness of the data it is advised to have only following class loggers in logging configuration:

Other class loggers of type com.evolveum.midpoint.model.* should be removed from systems configuration as they would interfere with classLogger for com.evolveum.midpoint.model.