topic Re: Alerting in Splunk Search

How to find non-monitored hosts

Muthu_Vinith — Sat, 02 Dec 2023 11:47:59 GMT

Hey All,

I’m a splunk beginner I'm looking to create a query that to be used as an alert, specifically to identify servers not in the _inventory – those not being monitored by Splunk. If anyone could share insights, examples

Thank You

Re: Alerting

PickleRick — Sat, 02 Dec 2023 11:44:34 GMT

In splunk terminolgy it's not called "query" but "search".

Anyway, it's a common question how to "find" something that's not there.

See https://www.duanewaddle.com/proving-a-negative/

Re: Alerting

ITWhisperer — Sat, 02 Dec 2023 11:44:48 GMT

Splunk is not good at finding things that aren't there - essentially, you would have to provide a list of all the servers you expect to find and discount all those that you do find, leaving you a list of servers which haven't been found.

Re: Alerting

Muthu_Vinith — Mon, 04 Dec 2023 09:34:46 GMT

I’ve a scenario where I want to compare of events from index=abc host=_inventory and data from a lookup file that includes fields such as host, location, os, etc. The end goal is to point out servers that aren't being reported by Splunk. The structure of my Splunk events includes fields like location, tier, servers, and splunk_server. In the lookup file, I have fields like host, location, os, and more

I combined two data’s and what is the search condition to find out how servers are being monitored @ITWhisperer @PickleRick

Re: Alerting

ITWhisperer — Mon, 04 Dec 2023 10:13:21 GMT

Re: Alerting

Muthu_Vinith — Mon, 04 Dec 2023 11:49:38 GMT

This search will give results of servers that is not being reported Correct? @ITWhisperer

Re: Alerting

ITWhisperer — Mon, 04 Dec 2023 12:05:42 GMT

That's the idea - try it and see

Re: Alerting

Muthu_Vinith — Mon, 04 Dec 2023 12:23:07 GMT

Okay Thank you @ITWhisperer

Re: Alerting

Muthu_Vinith — Tue, 05 Dec 2023 04:01:25 GMT

I tired this method but it's giving me servers that is monitored @ITWhisperer

Re: Alerting

ITWhisperer — Tue, 05 Dec 2023 11:39:55 GMT

This sounds like a data issue - you should check which hosts are coming up as not being monitored and see why they are not showing up in your index.

Re: Alerting

Muthu_Vinith — Tue, 05 Dec 2023 17:47:26 GMT

Sure @ITWhisperer