Splunk Search

Search Query to trigger an email of the host is not reporting in Splunk for last 15 minutes

anandhalagaras1
Path Finder

Hi Team

I have set of 5 hosts which are coming from an index=xyz and with sourcetype=iis so for example if any of the host logs from index=xyz and sourcetype=iis is getting stopped then we need to get an email notification from which server the logs has stopped getting ingested into splunk.

Timespan is for last 15 minutes.

5 host information:

abc, def, ijk, lmn, opq.

 

So can you kindly help to build the query.

 

Labels (2)
0 Karma

96nick
Path Finder

I do this with sparkline. It's not based on time but on log count in the _internal index. No logs being sent by host in _internal generally means something is wrong. This isn't the 15m solution you were looking for but wanted to share in case you wanted to implement something similar. You could use bin to create 15m buckets of _time.

index=_internal sourcetype=splunkd destPort!="-" 
| stats sparkline count by hostname, sourceHost, host, destPort, version 
| rename destPort as "Destination Port", host as "Indexer", sourceHost as "Forwarder IP", version as "Splunk Forwarder Version", hostname as "Forwarder Host Name", sparkline as "Traffic Frequency"
| dedup "Forwarder Host Name" 
| sort - count
| where count < 20

 You can play around with the count as well. I try to look for forwarders that aren't just completely dead but aren't communicating as much as normal as well.

0 Karma

manjunathmeti
Champion

hi @anandhalagaras1,

You can schedule the below search which identifies hosts that are not sending any data in the last 15 minutes for every 15 minutes.

| tstats count where index=xyz sourcetype=iis earliest=-15m by host 
| append 
    [| makeresults 
    | eval host=split("abc,def,ijk,lmn,opq", ",") 
    | mvexpand host 
    | eval count=0] 
| stats sum(count) as event_count by host 
| where event_count=0 
| mvcombine delim="," host 
| nomv host

 

Configure email alert action with the message:
Data is not received in  index=xyz and with sourcetype=iis  from hosts: $result.host$.

 

If this reply helps you, a like would be appreciated.

0 Karma

anandhalagaras1
Path Finder

@manjunathmeti ,

 

Thanks for your reply.

But when i checked the query as you mentioned i am getting an error as below:

Error in 'eval' command: The arguments to the 'split' function are invalid.

 

So kindly help on the same.

0 Karma

manjunathmeti
Champion

Post your query.

0 Karma

anandhalagaras1
Path Finder

 Sorry for the delay.

This is the query which I have used to search the logs and which in turn throwed an error.

 

| tstats count where index=xyz sourcetype=iis earliest=-15m by host
| append
[| makeresults
| eval host=split("abc,def,ijk,lmn,opq")
| mvexpand host
| eval count=0]
| stats sum(count) as event_count by host
| where event_count=0
| mvcombine delim="," host
| nomv host

 

Error Information:

Error in 'eval' command: The arguments to the 'split' function are invalid.
The search job has failed due to an error. You may be able view the job in the Job Inspector.
 
So kindly help to correct me where i am wrong.

@manjunathmeti

0 Karma

manjunathmeti
Champion

you need to provide a delimiter in the split function.

| tstats count where index=xyz sourcetype=iis earliest=-15m by host
| append
[| makeresults
| eval host=split("abc,def,ijk,lmn,opq", ",")
| mvexpand host
| eval count=0]
| stats sum(count) as event_count by host
| where event_count=0
| mvcombine delim="," host
| nomv host

 

0 Karma

anandhalagaras1
Path Finder

@manjunathmeti ,

Thank you & now I can able to see some results. Here in my case I have ran the below mentioned query for last 60 minutes and currently I can see latest logs for sourcetype iis in splunk for all the hosts. 

index=xyz sourcetype=iis

But if i use the query which you have provided I can see some result as below:

host            event_count
ijk,lmn             0

But actually there are latest logs in splunk for ijk,lmn host for last 60 minutes but here in this case it shows the event_count as 0.

 

Not sure about it. So kindly help to check and update me on the same.

0 Karma