Splunk Search

How to create alert if there is no data from an extracted field?

power12
Communicator

Hello Splunkers ,

I am trying to schedule an alert when there is no data from a particular field which is extracted field from last 30 minutes.

Below is the sample event

Feb 28 12:49:25 hostabc postfix/smtpd[61995]: connect from host1.abc.local[158.xx.xx]
Feb 28 12:49:25 hostxyz postfix/smtpd[61995]: connect from host2.abc.local[158.xx.xx.xx]
Feb 28 12:49:25 host123 postfix/smtpd[61995]: connect from host3.abc.local158.xx.xx.xxx]

I am using below regex to extract sourcehost which gives me host1.abc.local,host2.abc.local,host3.abc.local
| rex field=_raw ".*from (?<sourcehost>.*)" 

I want to create alert when I dont see any events for the last 30 minutes from the source hosts. The alert should say "No data received from <sourcehos> in the last 30 minutes"

Thanks in Advance

0 Karma
1 Solution

ITWhisperer
SplunkTrust
SplunkTrust

You need to append a search which returns a list of sourcehosts you are interested in. Often people will store this list in a csv store and use inputlookup to retreive it

| append
[ | inputlookup sourcehostlist.csv

| eval found=2 ]
| stats sum(found) as found by sourcehost
| where found = 2

or they could append the results of a search with a larger timeframe or a search of a summary index for example

| append
[ search index ... earliest=-10d latest=-1s
| dedup sourcehost
| fields sourcehost

| eval found=2 ]
| stats sum(found) as found by sourcehost
| where found = 2

or generate some events by hand each time

| append
[ | makeresults
  | fields - _time
  | eval sourcehost=split("host1,host1.abc.local,host2,host2.abc.local,host3,host3.abc.local",",")
  | mvexpand sourcehost

| eval found=2 ]
| stats sum(found) as found by sourcehost
| where found = 2

View solution in original post

ITWhisperer
SplunkTrust
SplunkTrust

Splunk is good at finding data which is in the logs - it is not so good at finding data which isn't there.

In order to get around this, you need to provide Splunk with a list of everything you are looking for and then see if anything exists in your 30 minute period.

index=....
| rex "from (?<sourcehost>.*)"
| dedup sourcehost
| eval found=1
| append
  [ <list of sourcehosts you are looking for>
    | eval found=2 ]
| stats sum(found) as found by sourcehost
| where found = 2
``` found = 1 if only in index, 2 if only in list, 3 if in both ```
0 Karma

power12
Communicator

@ITWhisperer  Thank you for your reply. I am trying to understand your search what should I do in my case in the search after the append


| append
[ host1.[abc.local],host2.[abc.local],host3.[abc.local]

| eval found=2 ]
| stats sum(found) as found by sourcehost
| where found = 2

 

I am getting error saying "missing a search command before

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

You need to append a search which returns a list of sourcehosts you are interested in. Often people will store this list in a csv store and use inputlookup to retreive it

| append
[ | inputlookup sourcehostlist.csv

| eval found=2 ]
| stats sum(found) as found by sourcehost
| where found = 2

or they could append the results of a search with a larger timeframe or a search of a summary index for example

| append
[ search index ... earliest=-10d latest=-1s
| dedup sourcehost
| fields sourcehost

| eval found=2 ]
| stats sum(found) as found by sourcehost
| where found = 2

or generate some events by hand each time

| append
[ | makeresults
  | fields - _time
  | eval sourcehost=split("host1,host1.abc.local,host2,host2.abc.local,host3,host3.abc.local",",")
  | mvexpand sourcehost

| eval found=2 ]
| stats sum(found) as found by sourcehost
| where found = 2

power12
Communicator

@ITWhisperer Thank you that worked

0 Karma
Get Updates on the Splunk Community!

Splunk App for Anomaly Detection End of Life Announcment

Q: What is happening to the Splunk App for Anomaly Detection?A: Splunk is officially announcing the ...

Aligning Observability Costs with Business Value: Practical Strategies

 Join us for an engaging Tech Talk on Aligning Observability Costs with Business Value: Practical ...

Mastering Data Pipelines: Unlocking Value with Splunk

 In today's AI-driven world, organizations must balance the challenges of managing the explosion of data with ...