Splunk Cloud Platform

If the host is not sending logs to Splunk

anandhalagaras1
Communicator

Hi Team,

I want to create and schedule an alert  with two scenarios. In first case i have an ample of hosts for which if there is no logs getting ingested into Splunk for more than 15 minutes then it should trigger an email alert.

And another requirement is that the host may be any host (*) and if there are no alerts from any of the host then it should trigger an email to the team.

So for first case consider this data as example :

Host

abc, def, ijk, mne, zda, and so on.

 

So kindly help with the query.

 

Labels (1)
0 Karma
1 Solution

scelikok
SplunkTrust
SplunkTrust

There may be two options,

1- Case mismatch, let's try converting everything to lowercase.

index=_internal OR index=*
| eval host=lower(host)
| stats count by host
| append [
| inputlookup hosts.csv
| eval host=lower(host)
| eval count=0
]
| stats max(count) as count by host
| where count=0

2- Your hosts.csv file contains host field a FQDN, if this is the case you should update your hosts.csv with the same way Splunk shows.

 

If this reply helps you an upvote and "Accept as Solution" is appreciated.

View solution in original post

scelikok
SplunkTrust
SplunkTrust

There may be two options,

1- Case mismatch, let's try converting everything to lowercase.

index=_internal OR index=*
| eval host=lower(host)
| stats count by host
| append [
| inputlookup hosts.csv
| eval host=lower(host)
| eval count=0
]
| stats max(count) as count by host
| where count=0

2- Your hosts.csv file contains host field a FQDN, if this is the case you should update your hosts.csv with the same way Splunk shows.

 

If this reply helps you an upvote and "Accept as Solution" is appreciated.

anandhalagaras1
Communicator

@scelikok,

Nice it works as expected. Thank you for helping me out.

0 Karma

scelikok
SplunkTrust
SplunkTrust

Hi @anandhalagaras1,

Please try below;

index=_internal OR index=*
| stats count by host
| append [
| inputlookup hosts.csv
| eval count=0
]
| stats max(count) as count by host
| where count=0
If this reply helps you an upvote and "Accept as Solution" is appreciated.
0 Karma

anandhalagaras1
Communicator

Hi @scelikok ,

When i used your query as mentioned Still i can see 400 +hosts reporting with count as 0 for last 15 minutes. But when i checked those servers individually i can see the latest logs in Splunk. I am quite not sure where i am missing it.

0 Karma

bowesmana
SplunkTrust
SplunkTrust

Create a lookup file with the list of hosts, e.g. hosts.csv containing

host
abc
def
ghi
jkl
mno
pqr

Then your search can be

Your_base_search_to_get_data
| stats count by host
| append [
  | inputlookup hosts.csv
  | eval count=0
]
| stats values(count) as count by host
| where count=0

That will give you all the hosts in hosts.csv that have no data 

Then you can create the alert based on these results.

I am not sure I understand your second use case.

 

0 Karma

anandhalagaras1
Communicator

@bowesmana ,

Apologies for the delayed response.  So today as you have mentioned I have created a hosts.csv file with 900+hosts  in it. And have ran the query as you have mentioned  for last 15 minutes:

index=_internal OR index=*
| stats count by host
| append [
| inputlookup hosts.csv
| eval count=0
]
| stats values(count) as count by host
| where count=0

I am getting results for 400+ hosts with count as 0. But for sample which I took one of the host from the output and checked  in the Search app and  I can see the host is reporting with latest timestamp itself without any issues. So why in this query it is getting captured with count as 0. Hence kindly help to guide me on the same.

 

 

 

0 Karma

bowesmana
SplunkTrust
SplunkTrust

I am not sure why this is the case,

Please replace the final stats with this

| stats max(count) as count by host

which will avoid creating multi value fields and see if this changes things.

Also, if you remove the 

| where count=0

from the search you will see the maximum value calculated for each search.

Make sure that when you are validating data, the time range you use is the same for each search you run.

 

0 Karma
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...