Splunk Search

How do I list machines that do not match my search?

cwheeler33
Explorer

How do I list machines that do not match my search?

"if" my script runs, a message is sent to splunk. The script runs once a week. I can easily see the details of my scripts, if it runs in splunk. So how do I list the machines that this script doesn't even start on? eg no entry sent to splunk.

This search does not list those who count is "zero", how do I list the "zero" machines?

 

 

"MyAppResults" | stats count by host | stats sum(count) as count by host

 


If I understood I should be using  "inputlookup hosts.csv" but I'm not sure how to use it properly. I still cannot get it to list "zero" machines.

Labels (1)
Tags (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @cwheeler33,

if you need to find the hosts in a monitoring perimeter that didin't send logs, you have to create a list of monitored hosts and use it for the control.

In few words:

  • create a lookup called e.g. perimeter.csv,
  • the lookup contains at least one field "host",
  • the run a search like this:
| metasearch index=_internal
| eval host=lower(host)
| stats count BY host
| append [ search
   | inputlookup perimeter.csv
   | eval host=lower(host), count=0
   | fields host count ]
| stats sum(count) AS total BY host
| where count=0

With this search you check the hosts with Forwarder, if you want also check hosts without forwarder (e.g. network appliance) you have to use a different index,

if you want to filter your logs using some field, you cannot use "| metasearch".

You can run this search all the times you want, also very frequently (e.g. 5 minutes) to control that you continously have logs. without them you're blind!

About the lookup, you have two way to populate it:

  • schedule a search with final outputlookup,
  • manually update it.

The first solution is easier and requires less job, but in this way you also have less control on your monitoring perimeter than the second one.

Ciao.

Giuseppe

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Finding something that is not there is not Splunk's strong suit.  See this blog entry for a good write-up on it.

https://www.duanewaddle.com/proving-a-negative/

Your lookup file will contain a list of all machines that run the script.  Compare that list to the list of results you get from the "MyAppResults" search to find out which machines did not run.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Observability | How to Think About Instrumentation Overhead (White Paper)

Novice observability practitioners are often overly obsessed with performance. They might approach ...

Cloud Platform | Get Resiliency in the Cloud Event (Register Now!)

IDC Report: Enterprises Gain Higher Efficiency and Resiliency With Migration to Cloud  Today many enterprises ...

The Great Resilience Quest: 10th Leaderboard Update

The tenth leaderboard update (11.23-12.05) for The Great Resilience Quest is out >> As our brave ...