Splunk Search

How to create alert if there is no data from an extracted field?

power12
Communicator

Hello Splunkers ,

I am trying to schedule an alert when there is no data from a particular field which is extracted field from last 30 minutes.

Below is the sample event

Feb 28 12:49:25 hostabc postfix/smtpd[61995]: connect from host1.abc.local[158.xx.xx]
Feb 28 12:49:25 hostxyz postfix/smtpd[61995]: connect from host2.abc.local[158.xx.xx.xx]
Feb 28 12:49:25 host123 postfix/smtpd[61995]: connect from host3.abc.local158.xx.xx.xxx]

I am using below regex to extract sourcehost which gives me host1.abc.local,host2.abc.local,host3.abc.local
| rex field=_raw ".*from (?<sourcehost>.*)" 

I want to create alert when I dont see any events for the last 30 minutes from the source hosts. The alert should say "No data received from <sourcehos> in the last 30 minutes"

Thanks in Advance

0 Karma
1 Solution

ITWhisperer
SplunkTrust
SplunkTrust

You need to append a search which returns a list of sourcehosts you are interested in. Often people will store this list in a csv store and use inputlookup to retreive it

| append
[ | inputlookup sourcehostlist.csv

| eval found=2 ]
| stats sum(found) as found by sourcehost
| where found = 2

or they could append the results of a search with a larger timeframe or a search of a summary index for example

| append
[ search index ... earliest=-10d latest=-1s
| dedup sourcehost
| fields sourcehost

| eval found=2 ]
| stats sum(found) as found by sourcehost
| where found = 2

or generate some events by hand each time

| append
[ | makeresults
  | fields - _time
  | eval sourcehost=split("host1,host1.abc.local,host2,host2.abc.local,host3,host3.abc.local",",")
  | mvexpand sourcehost

| eval found=2 ]
| stats sum(found) as found by sourcehost
| where found = 2

View solution in original post

ITWhisperer
SplunkTrust
SplunkTrust

Splunk is good at finding data which is in the logs - it is not so good at finding data which isn't there.

In order to get around this, you need to provide Splunk with a list of everything you are looking for and then see if anything exists in your 30 minute period.

index=....
| rex "from (?<sourcehost>.*)"
| dedup sourcehost
| eval found=1
| append
  [ <list of sourcehosts you are looking for>
    | eval found=2 ]
| stats sum(found) as found by sourcehost
| where found = 2
``` found = 1 if only in index, 2 if only in list, 3 if in both ```
0 Karma

power12
Communicator

@ITWhisperer  Thank you for your reply. I am trying to understand your search what should I do in my case in the search after the append


| append
[ host1.[abc.local],host2.[abc.local],host3.[abc.local]

| eval found=2 ]
| stats sum(found) as found by sourcehost
| where found = 2

 

I am getting error saying "missing a search command before

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

You need to append a search which returns a list of sourcehosts you are interested in. Often people will store this list in a csv store and use inputlookup to retreive it

| append
[ | inputlookup sourcehostlist.csv

| eval found=2 ]
| stats sum(found) as found by sourcehost
| where found = 2

or they could append the results of a search with a larger timeframe or a search of a summary index for example

| append
[ search index ... earliest=-10d latest=-1s
| dedup sourcehost
| fields sourcehost

| eval found=2 ]
| stats sum(found) as found by sourcehost
| where found = 2

or generate some events by hand each time

| append
[ | makeresults
  | fields - _time
  | eval sourcehost=split("host1,host1.abc.local,host2,host2.abc.local,host3,host3.abc.local",",")
  | mvexpand sourcehost

| eval found=2 ]
| stats sum(found) as found by sourcehost
| where found = 2

power12
Communicator

@ITWhisperer Thank you that worked

0 Karma
Get Updates on the Splunk Community!

Building Reliable Asset and Identity Frameworks in Splunk ES

 Accurate asset and identity resolution is the backbone of security operations. Without it, alerts are ...

Cloud Monitoring Console - Unlocking Greater Visibility in SVC Usage Reporting

For Splunk Cloud customers, understanding and optimizing Splunk Virtual Compute (SVC) usage and resource ...

Automatic Discovery Part 3: Practical Use Cases

If you’ve enabled Automatic Discovery in your install of the Splunk Distribution of the OpenTelemetry ...