If the host is not sending logs to Splunk

[anandhalagaras1]

Hi Team,

I want to create and schedule an alert  with two scenarios. In first case i have an ample of hosts for which if there is no logs getting ingested into Splunk for more than 15 minutes then it should trigger an email alert.

And another requirement is that the host may be any host (*) and if there are no alerts from any of the host then it should trigger an email to the team.

So for first case consider this data as example :

Host

abc, def, ijk, mne, zda, and so on.

So kindly help with the query.

[scelikok · Accepted solution]

There may be two options,

1- Case mismatch, let's try converting everything to lowercase.

index=_internal OR index=*
| eval host=lower(host)
| stats count by host
| append [
| inputlookup hosts.csv
| eval host=lower(host)
| eval count=0
]
| stats max(count) as count by host
| where count=0

2- Your hosts.csv file contains host field a FQDN, if this is the case you should update your hosts.csv with the same way Splunk shows.

[anandhalagaras1]

@scelikok,

Nice it works as expected. Thank you for helping me out.

[scelikok]

Hi @anandhalagaras1,

Please try below;

index=_internal OR index=*
| stats count by host
| append [
| inputlookup hosts.csv
| eval count=0
]
| stats max(count) as count by host
| where count=0

[anandhalagaras1]

Hi @scelikok ,

When i used your query as mentioned Still i can see 400 +hosts reporting with count as 0 for last 15 minutes. But when i checked those servers individually i can see the latest logs in Splunk. I am quite not sure where i am missing it.

[bowesmana]

Create a lookup file with the list of hosts, e.g. hosts.csv containing

host
abc
def
ghi
jkl
mno
pqr

Then your search can be

Your_base_search_to_get_data
| stats count by host
| append [
  | inputlookup hosts.csv
  | eval count=0
]
| stats values(count) as count by host
| where count=0

That will give you all the hosts in hosts.csv that have no data 

Then you can create the alert based on these results.

I am not sure I understand your second use case.

[anandhalagaras1]

@bowesmana ,

Apologies for the delayed response.  So today as you have mentioned I have created a hosts.csv file with 900+hosts  in it. And have ran the query as you have mentioned  for last 15 minutes:

index=_internal OR index=*
| stats count by host
| append [
| inputlookup hosts.csv
| eval count=0
]
| stats values(count) as count by host
| where count=0

I am getting results for 400+ hosts with count as 0. But for sample which I took one of the host from the output and checked  in the Search app and  I can see the host is reporting with latest timestamp itself without any issues. So why in this query it is getting captured with count as 0. Hence kindly help to guide me on the same.

[bowesmana]

I am not sure why this is the case,

Please replace the final stats with this

| stats max(count) as count by host

which will avoid creating multi value fields and see if this changes things.

Also, if you remove the 

| where count=0

from the search you will see the maximum value calculated for each search.

Make sure that when you are validating data, the time range you use is the same for each search you run.