I am running a lot of Splunkforwarders and use source=system sourcetype=foo for some custom Solaris OS metrics. All the nodes should have this source/sourcetype definition but I can see there is a gap between the total number of hosts and the number that send data as source=system sourcetype=foo. Therefore I would like to find all the nodes that are not sending data belonging to this source/sourcetype.
As far as I can see Deployment Monitor does not help here as it helps only to find hosts that suddenly stops reporting things, and I am pretty sure the nodes never sent sourcetype=foo.
The search should do a diff of the list of all the nodes (host=*) and a list of hosts returned by "source=system sourcetype=foo", tried various things but I am currently out of ideas.
Using dsh (or any other distributed SSH) and grep would help or course, but I am curious if there is any way doing it in SPL.
Any help appriciated.
I'm no fan of using
subsearch but I think this is one of the use cases where you have to use it. Try something like this:
source=system sourcetype=foo | stats count by sourcetype, host | search NOT [ search host=* | dedup host | table host ]
this should return a list of those
hosts not sending events with
Assuming there are some other sourcetype available per host, then there is a way to get this without a subsearch ... try this:
source=system | stats count(eval(sourcetype="foo")) AS c_foo count(eval(sourcetype!="foo")) AS c_others by host | search c_foo=0 AND c_others>=0 | dedup host | table host
hope this helps ...
thanks for your help, I was trying various things with subsearches but they didn't work. Your 2nd proposal is exactly what I was looking for - it works perfect!