Check if value is null for a period of time

[masterpipo]

Hi Guys,

I need help to set-up an email alert for Splunk that will trigger if a value is null for a specific amount of time. The value in question is derived from multiple values and added by eval command and is piped into timechart command with timespan of 1min.

I basically want it to inform me that value is null for x amount of mins.

Thanks!

[rtadams89]

Why would delay_test ever be null?

If it is being null because delaya, delayb, delayc, or delayd are null, then this issue is with the way you are doing concatenation/addition. If any of the fields in an "eval newField=field1+field2...filedX" are null, the newField will be null. If you are attempting to add all those fields (that is, mathematical add them), you'll want to use either the fillnull (http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Fillnull) or "eval ifnull()" function to make sure all the delayX fields have a value (even if it is "0"). Then you may need to use the "convert" function to make sure all the delayX values are treated as numbers. After that, you will be able to add them up to a get a "Delay_test" value for every event.

[masterpipo]

delayb, delay c and delay d will be null if there are no mail logs being generated, hence no mail. This is what I want to monitor. If there are mail logs, then value will always be an integer. I do hope I am making the right assumption 🙂

[rtadams89]

Alternatively, use an eval "case", eval "coalesce", replace function, or fillnull function to set the value of the field to a literal "null". Then use the transaction command along with the "startswith" and "endswith" parameters to merge the series of events that start with a "null" value and end with a non-"null" value into transaction events. The transaction command will output a "duration" field for each transaction. You can then do a conditional alert only when the events have a duration > X.

If you can give some example data, I can write the query described above.

[masterpipo]

Thanks!

The data in question are postfix logs:

Aug 8 07:32:49 localhost Aug 8 07:32:49 postfix/smtp[64151]: 5860A98052B: to=xxxx@domain.com, relay=relay[x.x.x.x]:25, conn_use=13, delay=0.75, delays=0.49/0/0.01/0.25, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as B720D58C6B1)

My query contains the following:

rex field=delays "(?.)\/(?.)\/(?.)\/(?.)" | eval delay_test = delayb + delayc + delayd | timechart span="1m" max(delay_test) by host useother=f usenull=f

So the value I need to be evaluated is delay_test if it is null for any duration of time

[martin_mueller]

You could set a temporary field to 1 if your value is null and to 0 otherwise, sum up that temporary field - that's your number of minutes.