Hello Splunkers ,
I am trying to schedule an alert when there is no data from a particular field which is extracted field from last 30 minutes.
Below is the sample event
Feb 28 12:49:25 hostabc postfix/smtpd[61995]: connect from host1.abc.local[158.xx.xx]
Feb 28 12:49:25 hostxyz postfix/smtpd[61995]: connect from host2.abc.local[158.xx.xx.xx]
Feb 28 12:49:25 host123 postfix/smtpd[61995]: connect from host3.abc.local158.xx.xx.xxx]
I am using below regex to extract sourcehost which gives me host1.abc.local,host2.abc.local,host3.abc.local
| rex field=_raw ".*from (?<sourcehost>.*)"
I want to create alert when I dont see any events for the last 30 minutes from the source hosts. The alert should say "No data received from <sourcehos> in the last 30 minutes"
Thanks in Advance
You need to append a search which returns a list of sourcehosts you are interested in. Often people will store this list in a csv store and use inputlookup to retreive it
| append
[ | inputlookup sourcehostlist.csv
| eval found=2 ]
| stats sum(found) as found by sourcehost
| where found = 2
or they could append the results of a search with a larger timeframe or a search of a summary index for example
| append
[ search index ... earliest=-10d latest=-1s
| dedup sourcehost
| fields sourcehost
| eval found=2 ]
| stats sum(found) as found by sourcehost
| where found = 2
or generate some events by hand each time
| append
[ | makeresults
| fields - _time
| eval sourcehost=split("host1,host1.abc.local,host2,host2.abc.local,host3,host3.abc.local",",")
| mvexpand sourcehost
| eval found=2 ]
| stats sum(found) as found by sourcehost
| where found = 2
Splunk is good at finding data which is in the logs - it is not so good at finding data which isn't there.
In order to get around this, you need to provide Splunk with a list of everything you are looking for and then see if anything exists in your 30 minute period.
index=....
| rex "from (?<sourcehost>.*)"
| dedup sourcehost
| eval found=1
| append
[ <list of sourcehosts you are looking for>
| eval found=2 ]
| stats sum(found) as found by sourcehost
| where found = 2
``` found = 1 if only in index, 2 if only in list, 3 if in both ```
@ITWhisperer Thank you for your reply. I am trying to understand your search what should I do in my case in the search after the append
| append
[ host1.[abc.local],host2.[abc.local],host3.[abc.local]
| eval found=2 ]
| stats sum(found) as found by sourcehost
| where found = 2
I am getting error saying "missing a search command before
You need to append a search which returns a list of sourcehosts you are interested in. Often people will store this list in a csv store and use inputlookup to retreive it
| append
[ | inputlookup sourcehostlist.csv
| eval found=2 ]
| stats sum(found) as found by sourcehost
| where found = 2
or they could append the results of a search with a larger timeframe or a search of a summary index for example
| append
[ search index ... earliest=-10d latest=-1s
| dedup sourcehost
| fields sourcehost
| eval found=2 ]
| stats sum(found) as found by sourcehost
| where found = 2
or generate some events by hand each time
| append
[ | makeresults
| fields - _time
| eval sourcehost=split("host1,host1.abc.local,host2,host2.abc.local,host3,host3.abc.local",",")
| mvexpand sourcehost
| eval found=2 ]
| stats sum(found) as found by sourcehost
| where found = 2
@ITWhisperer Thank you that worked