On Incident phases

[elfjes]

We (Geant) have a number of phases and states that an incident can go through. These serve a (partly) similar purpose but are separate, due to "historical reasons". Let's call them phases for now. From a UI perspective, we have the following phases:

Pending: The incident has just been recorded and we're still in the process of a root cause analysis, multiple snmp traps may come in that are tied together

• An incident may be overridden (deleted) in case an overarching incident has been identified or merged into existing incidents (which may also involve a delete)
• Description may change
• The UI shows them in a special way (our case blinking)
• The UI shows them read-only: cannot ack or perform other actions

Finalized: After some period (at least 60 seconds, but may be longer if the correlator decides) if an incident is still active, The incident finalizes. The following happens:

• (Coalescing) In order to stop flapping incidents/traps from polluting the UI, the incident is matched against similar incidents that have happened recently (last 24h) based on some rules (same equipment involved, same description, etc). The effect is that only the latest incident is shown in the UI, and a counter shows the number of similar incidents
• (Blacklisting) Based on boolean rulesets that are configurable by the user, the severity of the incident may change. Incidents may be given a higher severity, a lower severity or may even hide the incident completely from the ui.
• Users may now ack and comment on incidents
• Users may force close incidents

Clear: After an incident has become inactive (ie. if the network problem has disappeared) an incident goes to the clear phase. It is still shown on the UI until a user actively closes it.

• Incidents that have been active (since the beginning of Pending) for a minimum of 60 seconds will go to the cleared phase when inactive (also incidents that become inactive during the pending phase if the pending phase lasted longer than 60 seconds)
• A user may ack, commented or close an incident.

Closed: An incident is closed and is no long visible in the default view of the UI. An incident may close from the following triggers:

• A user closes a cleared or finalized incident
• An incident becomes inactive after less than 60 seconds of being active (so called short lived alarms)

I think I covered most/everything wrt phases and states, but this is a relatively complex topic and I may have missed something 😅

Do you think you could use this as a basis on how Argus might want to work with phases?

[hmpf]

Could you mark what is a phase and what is a state?

[hmpf]

In argus currently, we have no phases, and two boolean states: open and acked.

"open=True" means neither the source nor a human has closed the incident and that end_time has not been changed.

"acked=True" means there exists at least one ack that has not expired.

A stateful incident when created has an Event.INCIDENT_START, open set to True and acked set to False.

• It is possible to ack (set acked to True) immediately.
• It is possible to close the incident (source: Event.INCIDENT_END, user: Event.CLOSE) immediately.

Whether something is shown or not in the UI depends on the currently set incident filter.

"pending" (name pending) could be implemented several ways (see #804 and #805). While "pending" IIRC you need to be able to remotely take it out of pending so a flag stored in the database is probably needed. With this flag in one position, deletion is allowed and closing/acking is not. In the other, deletion is not allowed and closing/acking is ok. We can add such a flag without changing anything to our existing process, by not using the flag in its "pending" position.

The problem is "clear". We do not combine incidents into a single incident today (and we do not delete incidents), our plan was to hide, not delete incidents. Our "closed" covers your "clear". The user has full control over which is shown in the UI with the filters. There are multiple groups using argus today and I doubt they use the exact same process which means we have to be backwards compatible, that is: everything needs to work without "clear" being used.

I think, for the time being, it is best if "phase" exists only in Incident.metadata, so that I (we) can think of another solution in the meantime.

[elfjes]

We can live with phase being a metadata only thing (as long as we can delete incidents and have full control over that "delete"-flag)

The problem is "clear". We do not combine incidents into a single incident today (and we do not delete incidents), our plan was to hide, not delete incidents. Our "closed" covers your "clear". The user has full control over which is shown in the UI with the filters. There are multiple groups using argus today and I doubt they use the exact same process which means we have to be backwards compatible, that is: everything needs to work without "clear" being used.

I'm not quite sure I understand why CLEAR is the problem, especially when you say that our CLEAR matches your CLOSED. I think that as long as we can filter out our CLOSED Incidents from the UI by default we should be ok. We might be able to do that just by overriding a tmeplate?

When a user wants to look at CLOSED alarms we should show them as read-only.

[elfjes]

Could you mark what is a phase and what is a state?

The exact details should not matter to the gui/argus. Some of the states, phases (and we also have status, to keep it interesting...) or combinations are only relevant for the correlator. The correlator and the incident UI have evolved separately resulting in a mixed bag of different phases/states/statuses that may or may not be relevant to Argus. I've tried to limit the summary to what is relevant to Argus / an Incident UI, but for completeness sake I can give the following. Yes, this is confusing...:

First I'd like to introduce the concept of an Alarm. An Alarm is what the correlator calls an Incident internally. The correlator has a collection of active Alarms. The correlator has no concept of historical Alarms/Incidents. Whenever I mention an Alarm, this relates to the correlator. Incidents relate to the UI / Argus.

We have 3 phases (for Alarms, of which PENDING and (to a lesser extend) FINALIZED are also relevant for incidents)

• PENDING - as above
• FINALIZED - as above
• KILL (correlator only, clean up a CLOSED (CORRELATOR) Alarm and remove from Correlator). KILL is not really a phase but mere the final transition of an Alarm

We have 2 states that are Correlator/Alarms only

• OPEN (Phase can be PENDING or FINALIZED)
• CLOSED (CORRELATOR) An Alarm may be CLOSED (CORRELATOR), but its Incicent ACTIVE in the UI. If an Alarm reactivates within a certain time interval the correlator may reopen the Alarm without creating a new Incident

We have 3 ui statuses (for Incidents):

• ACTIVE (phase can be either PENDING or FINALIZED, correlator state OPEN or CLOSED (CORRELATOR))
• CLEAR - as above, (there is no associated Alarm in the correlator)
• CLOSED (UI) - as above, (there is no associated Alarm in the correlator)

Which results in the following valid combinations

• PENDING, OPEN, ACTIVE
• FINALIZED, OPEN, ACTIVE
• FINALIZED, CLOSED (CORRELATOR), ACTIVE
• -, -, CLEAR
• -, -, CLOSED (UI)