Display of a result according to the previous state of this same result

[Rajaion]

Hello community,

I apologize in advance, my English being bad, Google Translate is my friend.
My business is starting up on Splunk Enterprise and I am having a problem with a search that is probably simple but which has blocked me for a few days. I will explain the context to you: One of our tools sends supervision alerts to Enterprise with a code concerning its status (0: OK, 1: Warning, 2: Critical and 3: Unknown). The goal for me is to send these alerts to Splunk OnCall to share these alerts with other tools connected to OnCall.
No worries for sending to OnCall but I am blocking the return to OK of my alerts. Here is the query that is sending the alerts currently:

index = events_hp | search state = 2 OR state = 3 | fields hostname service_description output

However, when an alert returns to OK, I cannot send the info to OnCall to close the alert there. I should be able to say in my search to add state OK (state = 0) but only when the previous state was 2 or 3.

Basically, I should be able to send an alert when the state is OK (1) but only if before this OK, it was in 2 or 3.

Do you have any idea how I could do this?

Regards,

Rajaion

[johnhuang]

Depends on your data,  this could be a reasonable solution:

index=events_hp state IN (0, 2, 3)
| transaction hostname endswith=(state=0) startswith=(state=2 OR state=3)
| fields state hostname service_description output

[richgalloway]

Splunk queries do not have a concept of "previous state".  All they know is what they just found.

A workaround is to have a list of alerts with status of 2 or 3 and then consult that list.  Something like this:

index = events_hp (state = 2 OR state = 3) OR (state=0 [|inputlookup alerts.csv | fields foo | format] ) 
| fields hostname service_description output

where alerts.csv is a lookup file containing alerts that previously held a state of 2 or 3.  Field 'foo' is some field that uniquely identifies an alert.

I do not have a good suggestion for populating alerts.csv other than running a separate search.

P.S.  Excellent English!

[PickleRick]

True that. But if a "service" has any form of ID that uniquely identifies it we can build a time-ordered list of states and check the states chronologically 🙂

Question is do have such identifier.

[Rajaion]

Hello,

Thanks for your feedback. I did not know that it was necessary to go through a file but the problem is that the index which contains the data fills up via direct sending of the monitoring tool, there is no csv file between two.

Alerts are uniquely identified by a combination of the hostname and the service in error, there are never two hosts with the same name.

Would it be possible to create, at the same time as filling in the index, the famous csv file to then consult it to return to OK?

Regards,

Rajaion

[richgalloway]

It's not strictly necessary to use a file, but that's the way I can think of.

The point of the CSV is to store alerts until they are cleared.  It can be populated at any time as long as it does not contain alerts with status=0.  The typical method is to run two similar searches.  The first generates alerts or notifications.  The second performs the same query, but saves its results in a CSV for use by the first search.

[Rajaion]

Hello,

Sorry for the late return. I'll dig in this side, see how to create a search that feeds a csv first, then get it to communicate with my send alert request.
I will keep you posted on my progress.

Regards,