Determine which Active servers with Universal Forwarder areNOT sending logs to Splunk

[geoffmx]

We have a bunch of servers with UFs installed. These servers may have different operational states. For example, "Active", "Build in Progress", "Decommissioned", and "Decom in Progress". We use ServiceNow for the asset inventory.

This is the search query used to determine the version of the UFs installed.

index=_internal source=*metrics.log component=Metrics group=tcpin_connections | dedup hostname | table hostname sourceIp os arch version

I would like to be able to get the Active servers that have splunk UF installed and NOT reporting to splunk.

Also, looking for a way to dynamically update the list of Active servers, as this list changes whenever there are new servers onboarded or old servers decommissioned.

I've looked at the Splunk App/Add-on for ServiceNow and could not find an option to do this.

I've also looked at https://docs.splunk.com/Documentation/Splunk/7.3.1/DMC/Configureforwardermonitoring, but am not sure how to go about configuring it to dynamically update the asset list with the Active servers.

Any help will be much appreciated.

[gcusello · Accepted solution]

Hi geoffmx,
the way to have the list of not reporting UFs is the following:
at first you have to create a lookup (e.g. called "perimeter.csv") containing the list of all hostnames of the UFs that you should have;
in this lookup there must be at least one field that you can call "host", but you could also insert other information to enrich your searches;
you can populate this lookup manually or from a CMDB;
after you have to run a search like this:

| metasearch index=_internal
| eval host=upper(host)
| stats count BY host
| append [ | inputlookup perimeter.csv | eval host=upper(host), count=0 | fields host, count ]
| stats sum(count) AS Total BY host
| where Total=0

In this way you have the list of all not reporting UFs and you can crete an alert.
If you don't use the last row, you can display in a dashboard the situation of all your forwarders.

Bye.
Giuseppe

[geoffmx]

Thanks gcusello! This is exactly what I was looking for 🙂

Would you happen to know a way to dynamically update the lookup via a CMDB? I can update manually, but automating it would be ideal.

[gcusello]

See DB-Connect App ( https://splunkbase.splunk.com/app/2686/ ) to execute queries on a DB and update a lookup.
Bye.
Giuseppe

[geoffmx]

Thank you, once again!