topic Re: Search Query to trigger an email of the host is not reporting in Splunk for last 15 minutes in Splunk Search

Search Query to trigger an email of the host is not reporting in Splunk for last 15 minutes

anandhalagaras1 — Mon, 29 Mar 2021 05:25:03 GMT

Hi Team

I have set of 5 hosts which are coming from an index=xyz and with sourcetype=iis so for example if any of the host logs from index=xyz and sourcetype=iis is getting stopped then we need to get an email notification from which server the logs has stopped getting ingested into splunk.

Timespan is for last 15 minutes.

5 host information:

abc, def, ijk, lmn, opq.

So can you kindly help to build the query.

Re: Search Query to trigger an email of the host is not reporting in Splunk for last 15 minutes

manjunathmeti — Mon, 29 Mar 2021 08:23:23 GMT

hi @anandhalagaras1,

You can schedule the below search which identifies hosts that are not sending any data in the last 15 minutes for every 15 minutes.

Configure email alert action with the message:
Data is not received in index=xyz and with sourcetype=iis from hosts: $result.host$.

If this reply helps you, a like would be appreciated.

Re: Search Query to trigger an email of the host is not reporting in Splunk for last 15 minutes

anandhalagaras1 — Tue, 30 Mar 2021 13:02:22 GMT

@manjunathmeti ,

Thanks for your reply.

But when i checked the query as you mentioned i am getting an error as below:

Error in 'eval' command: The arguments to the 'split' function are invalid.

So kindly help on the same.

Re: Search Query to trigger an email of the host is not reporting in Splunk for last 15 minutes

manjunathmeti — Wed, 31 Mar 2021 05:01:56 GMT

Post your query.

Re: Search Query to trigger an email of the host is not reporting in Splunk for last 15 minutes

anandhalagaras1 — Thu, 08 Apr 2021 05:56:47 GMT

Sorry for the delay.

This is the query which I have used to search the logs and which in turn throwed an error.

Error Information:

Error in 'eval' command: The arguments to the 'split' function are invalid.

The search job has failed due to an error. You may be able view the job in the Job Inspector.

So kindly help to correct me where i am wrong.

@manjunathmeti

Re: Search Query to trigger an email of the host is not reporting in Splunk for last 15 minutes

manjunathmeti — Thu, 08 Apr 2021 06:35:59 GMT

you need to provide a delimiter in the split function.

Re: Search Query to trigger an email of the host is not reporting in Splunk for last 15 minutes

anandhalagaras1 — Thu, 08 Apr 2021 12:59:32 GMT

@manjunathmeti ,

Thank you & now I can able to see some results. Here in my case I have ran the below mentioned query for last 60 minutes and currently I can see latest logs for sourcetype iis in splunk for all the hosts.

index=xyz sourcetype=iis

But if i use the query which you have provided I can see some result as below:

host event_count
ijk,lmn 0

But actually there are latest logs in splunk for ijk,lmn host for last 60 minutes but here in this case it shows the event_count as 0.

Not sure about it. So kindly help to check and update me on the same.

Re: Search Query to trigger an email of the host is not reporting in Splunk for last 15 minutes

96nick — Thu, 08 Apr 2021 13:32:26 GMT

I do this with sparkline. It's not based on time but on log count in the _internal index. No logs being sent by host in _internal generally means something is wrong. This isn't the 15m solution you were looking for but wanted to share in case you wanted to implement something similar. You could use bin to create 15m buckets of _time.

index=_internal sourcetype=splunkd destPort!="-" | stats sparkline count by hostname, sourceHost, host, destPort, version | rename destPort as "Destination Port", host as "Indexer", sourceHost as "Forwarder IP", version as "Splunk Forwarder Version", hostname as "Forwarder Host Name", sparkline as "Traffic Frequency" | dedup "Forwarder Host Name" | sort - count | where count < 20

You can play around with the count as well. I try to look for forwarders that aren't just completely dead but aren't communicating as much as normal as well.