Determining Certification Case Outcome

Last modified 01 Feb 2022 19:38 +01:00

Each case is reviewed in one or more stages, by zero, one, or more reviewers. An interesting question is: how should be the overall outcome determined, based on responses from individual reviewers? Moreover, there are borderline situations, like when no reviewers are assigned (e.g. if reviewer is defined as the target role owner, but a particular role has no owner), or when reviewer(s) fail to provide any answer during a given stage. And finally, when stepping up from a stage into another: which cases should be kept in consideration - only those that were explicitly accepted during previous phase? Or those that were not rejected? Or all cases? And then, how to combine results from individual stages into overall result?

There are two parameters that answer the above questions: stage-level-defined outcomeStrategy and campaign-level-defined reviewStrategy. The former determines how decisions of reviewers within a stage are combined into "stage decision", while the latter controls how decisions taken at individual stages combine into the final decision. The second one also determines if a case that was rejected (or not approved) at stage N continues in stage N+1 or not.

Stage-level outcome determination strategies

Currently, there are four strategies defined:

Strategy Description

oneAcceptAccepts

If at least one reviewer accepts, the result is "Accept" regardless of the other votes. This is the default strategy.

allMustAccept

All reviewers must accept. No reviewers mean No response.

oneDenyDenies

If at least one reviewer denies (either via revoke or reduce), the result is not approved. However, for the case to be approved, at least one approval must be present.

acceptedIfNotDenied

Approved if none of the reviewers denies (either via revoke or reduce). So e.g. if nobody says anything, the case is approved.

Moreover, one can define a default response that is used when no reviewers are available. (This doesn’t mean that no reviewers responded! It means that e.g. reviewers were defined as target role owners, but a particular role has no owners.)

In the following decision tables these symbols are used:

Symbol Meaning

present

one or more answers of this kind are present

-

no answers of this kind are present

*

answers of this kind are ignored, i.e. there may be zero, one or more such answers

"No response" can be represented in the following ways:

  • explicit NO_RESPONSE decision is provided,

  • decision with response = null is provided,

  • no decision for given reviewer is provided.

"Delegate" is currently not supported. The plan is to treat delegate as a "forward pointer" to another reviewer. So his/her answer will be used as the answer we are looking for. Currently, Delegate is treated as No response.

OneAcceptAccepts

Revoke Reduce Not decided No response Accept Result

*

*

*

*

present

Accept

present

*

*

*

-

Revoke

-

present

*

*

-

Reduce

-

-

present

*

-

Not decided

-

-

-

present

-

No response

-

-

-

-

-

<default>

AllMustAccept

Revoke Reduce Not decided No response Accept Result

present

*

*

*

*

Revoke

-

present

*

*

*

Reduce

-

-

present

*

*

Not decided

-

-

-

present

*

No response

-

-

-

-

present

Accept

-

-

-

-

-

<default>

OneDenyDenies

Revoke Reduce Not decided No response Accept Result

present

*

*

*

-

Revoke

-

present

*

*

-

Reduce

-

-

*

*

present

Accept

-

-

present

*

-

Not decided

-

-

-

present

-

No response

-

-

-

-

-

*<default>
*

AcceptedIfNotDenied

Revoke Reduce Not decided No response Accept Result

present

*

*

*

*

Revoke

-

present

*

*

*

Reduce

-

-

present

*

*

Accept

-

-

-

present

*

Accept

-

-

-

-

present

Accept

-

-

-

-

-

<default> (it is reasonable to use Accept as the default)

Overall outcome determination strategies

Currently, there are the same strategies as the ones at the level of a stage. The default one is allMustAccept.

However, there is one thing to be configured: when to advance to a next stage. It is defined by two multivalued properties: stopReviewOn and advanceToNextStageOn. These can be defined at the level of stage or the whole campaign. Either one can be defined - the other one is then computed as its complement. If both are specified, stopReviewOn takes precedence. Note that the values are not added to the ones specified on the higher level: they always replace them - i.e. stage-level values replace campaign-level values.

If neither one is specified, the default values are derived from the outcome strategy used, as shown in the following table.

Strategy Description Default stopReviewOn value

oneAcceptAccepts

If at least one reviewer accepts, the result is "Accept" regardless of the other votes.

accept

allMustAccept

All reviewers must accept. No reviewers mean No response.

revoke, reduce

oneDenyDenies

If at least one reviewer denies (either via revoke or reduce), the result is not approved. However, for the case to be approved, at least one approval must be present. This is the default strategy.

revoke, reduce

acceptedIfNotDenied

Approved if none of the reviewers denies (either via revoke or reduce). So e.g. if nobody says anything, the case is approved.

revoke, reduce
Was this page helpful?
YES NO
Thanks for your feedback