Splunk Search

How do I list machines that do not match my search?

cwheeler33
Explorer

How do I list machines that do not match my search?

"if" my script runs, a message is sent to splunk. The script runs once a week. I can easily see the details of my scripts, if it runs in splunk. So how do I list the machines that this script doesn't even start on? eg no entry sent to splunk.

This search does not list those who count is "zero", how do I list the "zero" machines?

 

 

"MyAppResults" | stats count by host | stats sum(count) as count by host

 


If I understood I should be using  "inputlookup hosts.csv" but I'm not sure how to use it properly. I still cannot get it to list "zero" machines.

Labels (1)
Tags (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @cwheeler33,

if you need to find the hosts in a monitoring perimeter that didin't send logs, you have to create a list of monitored hosts and use it for the control.

In few words:

  • create a lookup called e.g. perimeter.csv,
  • the lookup contains at least one field "host",
  • the run a search like this:
| metasearch index=_internal
| eval host=lower(host)
| stats count BY host
| append [ search
   | inputlookup perimeter.csv
   | eval host=lower(host), count=0
   | fields host count ]
| stats sum(count) AS total BY host
| where count=0

With this search you check the hosts with Forwarder, if you want also check hosts without forwarder (e.g. network appliance) you have to use a different index,

if you want to filter your logs using some field, you cannot use "| metasearch".

You can run this search all the times you want, also very frequently (e.g. 5 minutes) to control that you continously have logs. without them you're blind!

About the lookup, you have two way to populate it:

  • schedule a search with final outputlookup,
  • manually update it.

The first solution is easier and requires less job, but in this way you also have less control on your monitoring perimeter than the second one.

Ciao.

Giuseppe

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Finding something that is not there is not Splunk's strong suit.  See this blog entry for a good write-up on it.

https://www.duanewaddle.com/proving-a-negative/

Your lookup file will contain a list of all machines that run the script.  Compare that list to the list of results you get from the "MyAppResults" search to find out which machines did not run.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

CX Day is Coming!

Customer Experience (CX) Day is on October 7th!! We're so excited to bring back another day full of wonderful ...

Strengthen Your Future: A Look Back at Splunk 10 Innovations and .conf25 Highlights!

The Big One: Splunk 10 is Here!  The moment many of you have been waiting for has arrived! We are thrilled to ...

Now Offering the AI Assistant Usage Dashboard in Cloud Monitoring Console

Today, we’re excited to announce the release of a brand new AI assistant usage dashboard in Cloud Monitoring ...