I am running a lot of Splunkforwarders and use source=system sourcetype=foo for some custom Solaris OS metrics. All the nodes should have this source/sourcetype definition but I can see there is a gap between the total number of hosts and the number that send data as source=system sourcetype=foo. Therefore I would like to find all the nodes that are not sending data belonging to this source/sourcetype.
As far as I can see Deployment Monitor does not help here as it helps only to find hosts that suddenly stops reporting things, and I am pretty sure the nodes never sent sourcetype=foo.
The search should do a diff of the list of all the nodes (host=*) and a list of hosts returned by "source=system sourcetype=foo", tried various things but I am currently out of ideas.
Using dsh (or any other distributed SSH) and grep would help or course, but I am curious if there is any way doing it in SPL.
Any help appriciated.
Cheers Christian
Hi cruschke_bde,
I'm no fan of using join
or subsearch
but I think this is one of the use cases where you have to use it. Try something like this:
source=system sourcetype=foo | stats count by sourcetype, host | search NOT [ search host=* | dedup host | table host ]
this should return a list of those hosts
not sending events with sourcetype=foo
Update:
Assuming there are some other sourcetype available per host, then there is a way to get this without a subsearch ... try this:
source=system | stats count(eval(sourcetype="foo")) AS c_foo count(eval(sourcetype!="foo")) AS c_others by host | search c_foo=0 AND c_others>=0 | dedup host | table host
hope this helps ...
cheers, MuS
Hi cruschke_bde,
I'm no fan of using join
or subsearch
but I think this is one of the use cases where you have to use it. Try something like this:
source=system sourcetype=foo | stats count by sourcetype, host | search NOT [ search host=* | dedup host | table host ]
this should return a list of those hosts
not sending events with sourcetype=foo
Update:
Assuming there are some other sourcetype available per host, then there is a way to get this without a subsearch ... try this:
source=system | stats count(eval(sourcetype="foo")) AS c_foo count(eval(sourcetype!="foo")) AS c_others by host | search c_foo=0 AND c_others>=0 | dedup host | table host
hope this helps ...
cheers, MuS
Hi MuS,
thanks for your help, I was trying various things with subsearches but they didn't work. Your 2nd proposal is exactly what I was looking for - it works perfect!
Thanks again.
Cheers Christian
you're welcome 😉 please mark this as answered - thx
update ping ....