topic How to setup Splunk Alert when status not changed in Alerting

How to setup Splunk Alert when status not changed

babujlinuz — Tue, 29 Sep 2020 12:50:59 GMT

Hi Amigo,

I need to setup a Splunk alert when the status is not changed from "status = pending_app_gw." to "status = ACTIVE. with in 5-10 sec" max. if not changed to active it should alert.

08.02.2017 07:25:26.280 | 344 | INFO | 10098094 | GW: session(90200371) status = INITIALIZING.
08.02.2017 07:25:26.280 | 344 | INFO | 10098094 | GW: session(90200371) status = pending_app_gw.

*****after restart of the application *******
08.02.2017 08:20:36.618 | 1752 | INFO | 10098094 | GW: session(90200371) status = INITIALIZING.
08.02.2017 08:20:36.618 | 1752 | INFO | 10098094 | GW: session(90200371) status = pending_app_gw.
08.02.2017 08:20:36.706 | 5344 | INFO | 10098094 | GW: session(90200371) status = ACTIVE.

I tried with the below query. but it is not working as expected.
pending_app_gw sourcetype=sesionlog |rex "^[^(\n]*((?P\d+)"|join ses_id [ search sourcetype=oslog earliest=-10s latest=now "status = ACTIVE."]|table host ses_id

Appreciate your help to setup the alert working properly.

Thanks in Advance.

Regards
Babujlinuz

Re: How to setup Splunk Alert when status not changed

adonio — Mon, 06 Mar 2017 15:14:35 GMT

Hi babujlinuz,
Take a look here: https://answers.splunk.com/answers/177265/how-to-configure-an-alert-for-when-a-given-event-i-1.html

Re: How to setup Splunk Alert when status not changed

DalJeanis — Mon, 06 Mar 2017 20:07:56 GMT

(your base search) 
| table ses_id status 
| dedup ses_id 
| where status!="ACTIVE" 
| addinfo 
| where _time < info_max_time - 10

Basically, any status that is not ="ACTIVE" as/of 10 seconds before the end of the search needs an alert.

However, it seems like there ought to be identifying information about WHAT is running... the host or service name , so you need to add that into the code as appropriate. Let's assume that it was appname. It could just as well be sourcetype or host or whatever.

(your base search) earliest=-1h latest=-10s 
| table  ses_id appname status 
| dedup ses_id appname 
| where status!="ACTIVE" 
| addinfo 
| where _time < info_max_time - 10

Re: How to setup Splunk Alert when status not changed

niketn — Mon, 06 Mar 2017 20:12:55 GMT

Since you do not have a unique transaction ID, you can make use of streamstats. First you would need to perform field extractions. Since status is key value pain in your data, it should be automatically extracted as search time.
You can use Pipe delimited Interactive Field Extraction in Splunk
transforms.conf

[sesionlog]
DELIMS = "|"
FIELDS = "field1","thread_id","log_level","trans_id","status_details"

Finally, for Session ID you can use regular expression based field extraction.
props.conf

[sesionlog]
EXTRACT-session_id = ^[^\(\n]*\((?P<session_id>\d+)

Finally try streamstats like following which resets counter every time after finding an Active status(there can be multiple options)

index=<YourIndexName> sourcetype=sesionlog status="pending_app_gw." OR status="ACTIVE."
| table _time session_id trans_id thread_id log_level status 
| reverse
| streamstats count current=f reset_after="("status==\"ACTIVE.\"")" 
| stats count as eventcount min(_time) as MinTime max(_time) as MaxTime values(status) as status values(count) as CountFlag by session_id trans_id
| search (eventcount=3 AND CountFlag=2 AND status="pending_app_gw." AND status="ACTIVE.") OR (eventcount=1 AND CountFlag=0 AND status="pending_app_gw." AND status!="ACTIVE.")
| eval OrphanedPendingStatusDuration=now()-MinTime
| rename MinTime as OrphanedPendingStatusTime
| fieldformat OrphanedPendingStatusTime=strftime(OrphanedPendingStatusTime,"%c")
| fieldformat MaxTime=strftime(MaxTime,"%c")

PS: Based on your data there might be other combinations as well. Like (eventcount=2 AND CountFlag=1 AND status="pending_app_gw." AND status="ACTIVE.") represents all expected Pending Status followed by Active Status.