How to search for hosts that are not forwarding data of a specific source or sourcetype?

[cruschke_bde]

I am running a lot of Splunkforwarders and use source=system sourcetype=foo for some custom Solaris OS metrics. All the nodes should have this source/sourcetype definition but I can see there is a gap between the total number of hosts and the number that send data as source=system sourcetype=foo. Therefore I would like to find all the nodes that are not sending data belonging to this source/sourcetype.

As far as I can see Deployment Monitor does not help here as it helps only to find hosts that suddenly stops reporting things, and I am pretty sure the nodes never sent sourcetype=foo.

The search should do a diff of the list of all the nodes (host=*) and a list of hosts returned by "source=system sourcetype=foo", tried various things but I am currently out of ideas.

Using dsh (or any other distributed SSH) and grep would help or course, but I am curious if there is any way doing it in SPL.

Any help appriciated.

Cheers Christian

[MuS · Accepted solution]

Hi cruschke_bde,

I'm no fan of using join or subsearch but I think this is one of the use cases where you have to use it. Try something like this:

source=system sourcetype=foo | stats count by sourcetype, host | search NOT [ search host=* | dedup host | table host ] 

this should return a list of those hosts not sending events with sourcetype=foo

Update:

Assuming there are some other sourcetype available per host, then there is a way to get this without a subsearch ... try this:

source=system | stats count(eval(sourcetype="foo")) AS c_foo count(eval(sourcetype!="foo")) AS c_others by host | search c_foo=0 AND c_others>=0 | dedup host | table host

hope this helps ...

cheers, MuS

[cruschke_bde]

Hi MuS,

thanks for your help, I was trying various things with subsearches but they didn't work. Your 2nd proposal is exactly what I was looking for - it works perfect!

Thanks again.

Cheers Christian

[MuS]

you're welcome 😉 please mark this as answered - thx

[MuS]

update ping ....