Suppress results once a condition is met

[raghul725]

Hello,

index=* "My-Search-String" |rex "My-Regex"| eval Status=if(like (my-rex-extractor-field,"xxx-yyyy%"), "FILE_DELIVERED", "FILE_NOT_DELIVERED")|table Status

I need to run the above between 5-7 AM alert via email. Although the file arrives around 05:15 AM, I want to continue running this as an alert until 07 AM because the alert should continue to state the status to avoid missing and this will be detrimental if the status continues to be FILE_NOT_DELIVERED

But the problem here is the alert continues to output FILE_NOT_DELIVERED albeit containing FILE_DELIVERED in the ouput

Current behaviour - when the alert triggers at 05:45 AM - alert set to run as cron schedule - every 15 mins

FILE_NOT_DELIVERED
FILE_NOT_DELIVERED
FILE_DELIVERED
FILE_NOT_DELIVERED
FILE_NOT_DELIVERED

Expected behaviour

as soon as the SPL finds FILE_DELIVERED, for all subsequent runs the FILE_NOT_DELIVERED result should be suppressed and the SPL should continue to return FILE_DELIVERED

How do I achieve this please?

[ITWhisperer]

| where Status=="FILE_DELIVERED"

and then alert while there are no results.

[raghul725]

Thank you. Sorry for being silly.

If we set the where clause and alert when there are NO results, when the file is delivered the alert will NOT include FILE_DELIVERED message? Have I got this right?

If that is the case, then this MAY NOT meet my requirement, as my alert should include FILE_DELIVERED message once the SPL finds that File was delivered.

Again apologies if I have misunderstood your response

[ITWhisperer]

Thanks for the clarification. If I understand correctly, you want to alert if there is no FILE_DELIVERED event, or if the first FILE_DELIVERED event is the last event?

| streamstats count as event
| eventstats first(eval(if(Status=="FILE_DELIVERED",event,null()))) as first_delivered max(event) as last_event
| where isnull(first_delivered) OR first_delivered == last_event

and set your alert to trigger if there are results

[raghul725]

Hello there,

unfortunately the previous input does not provide the desired result, unless I am lost - which could very well be. But I have explained the requirement again in my previous post. Would it be possible to check my previous post and suggest please?

[ITWhisperer]

What result did it produce and why does it not meet your requirement?

[raghul725]

Hello, I continued to receive FILE_NOT_DELIVERED message, despite the output contains FILE_DELIVERED status

Requirement:

For Example:

So we expect the file to be delivered around 05:15 AM, any alert before that should continue with FILE_NOT_DELIVERED and if the SPL finds the file to be delivered, any alert after that should just output FILE_DELIVERED, removing/suppressing FILE_NOT_DELIVERED in the subsequent runs until 07:00 AM.

If the file was NOT delivered, then the alert should continue stating FILE_NOT_DELIVERED until 07:00 AM

By following this approach I believe we can ensure the status of the file delivery (regardless of the status) cannot be missed. 

I am not sure if the SPL you have posted in your previous post will satisfy this requirement, but I will certainly check.

[ITWhisperer]

There are essentially 3 states, 1) FILE_DELIVERED not present, 2) FILE_DELIVERED last status, and 3) FILE_DELIVERED present but not last.

Taking an example of 5 events with the middle event being FILE_DELIVERED and using head to change the situation, you can see that the search only returns results for the cases head 1 to head 3, with head 4 and head 5 having no results.

[raghul725]

Brilliant, my requirements are, the output should contain FILE_DELIVERED status for head 4 and head 5 as well, as we have received FILE_DELIVERED status for head 3.

In other words, as soon as we see FILE_DELIVERED, the subsequent runs should always include FILE_DELIVERED line ONLY (should NOT include FILE_NOT_DELIVERED from the previous or current run)so the alert won't be missed.
The output should continue stating FILE_NOT_DELIVERED ONLY when no occurrence of FILE_DELIVERED was found.

[ITWhisperer]

| makeresults format=csv data="Status
FILE_NOT_DELIVERED
FILE_NOT_DELIVERED
FILE_DELIVERED
FILE_NOT_DELIVERED
FILE_NOT_DELIVERED"
| head 5
| eval {Status}=Status
| fields - Status
| stats values(*) as *
| eval Status=coalesce(FILE_DELIVERED, FILE_NOT_DELIVERED)
| fields Status

[raghul725]

Thank you - what version of splunk does your suggestion work please, because I ran the query before I can modify mine, no results at all for any value of head.

We are on 8.2.11.2

[ITWhisperer · Accepted solution]

Makeresults changed in version 9 allowing you to specify format and data. If you have a prior version, you need to set up the dummy data in a different way.

| makeresults
| eval _raw="Status
FILE_NOT_DELIVERED
FILE_NOT_DELIVERED
FILE_DELIVERED
FILE_NOT_DELIVERED
FILE_NOT_DELIVERED"
| multikv forceheader=1 
| table Status
| head 5
| eval {Status}=Status
| fields - Status
| stats values(*) as *
| eval Status=coalesce(FILE_DELIVERED, FILE_NOT_DELIVERED)
| fields Status

[raghul725]

Thank you, it works now. I am going to monitor for one more day before I mark your response as accepted solution.

But in the meanwhile, could you kindly explain how the below lines work please?

| eval {Status}=Status
| fields - Status
| stats values(*) as *
| eval Status=coalesce(FILE_DELIVERED, FILE_NOT_DELIVERED)
| fields Status

 I started guessing/ play with it, but certain lines I am unable to understand what it does/ how it fits here to provide me the desired result TBH.

[raghul725]

Thank you for your response. Sorry if I was not clear with my original request.

Alert should continue to say FILE_NOT_DELIVERED until SPL evaluates to FILE_DELIVERED. Once the SPL's output is FILE_DELIVERED, for every 15 mins of the remaining schedule (until 07:00 AM), the alert should just say FILE_DELIVERED and remove FILE_NOT_DELIVERED in the output.

For Example:

So we expect the file to be delivered around 05:15 AM, any alert before that should continue with FILE_NOT_DELIVERED and if the SPL finds the file to be delivered, any alert after that should just output FILE_DELIVERED, removing/suppressing FILE_NOT_DELIVERED in the subsequent runs until 07:00 AM.

If the file was NOT delivered, then the alert should continue stating FILE_NOT_DELIVERED until 07:00 AM

By following this approach I believe we can ensure the status of the file delivery (regardless of the status) cannot be missed. 

I am not sure if the SPL you have posted in your previous post will satisfy this requirement, but I will certainly check.

Thanks again.