topic Re: Determine which Active servers with Universal Forwarder areNOT sending logs to Splunk in Getting Data In

Determine which Active servers with Universal Forwarder areNOT sending logs to Splunk

geoffmx — Tue, 03 Sep 2019 06:17:38 GMT

We have a bunch of servers with UFs installed. These servers may have different operational states. For example, "Active", "Build in Progress", "Decommissioned", and "Decom in Progress". We use ServiceNow for the asset inventory.

This is the search query used to determine the version of the UFs installed.

index=_internal source=*metrics.log component=Metrics group=tcpin_connections | dedup hostname | table hostname sourceIp os arch version

I would like to be able to get the Active servers that have splunk UF installed and NOT reporting to splunk.

Also, looking for a way to dynamically update the list of Active servers, as this list changes whenever there are new servers onboarded or old servers decommissioned.

I've looked at the Splunk App/Add-on for ServiceNow and could not find an option to do this.

I've also looked at https://docs.splunk.com/Documentation/Splunk/7.3.1/DMC/Configureforwardermonitoring, but am not sure how to go about configuring it to dynamically update the asset list with the Active servers.

Any help will be much appreciated.

Re: Determine which Active servers with Universal Forwarder areNOT sending logs to Splunk

gcusello — Tue, 03 Sep 2019 07:01:46 GMT

Hi geoffmx,
the way to have the list of not reporting UFs is the following:
at first you have to create a lookup (e.g. called "perimeter.csv") containing the list of all hostnames of the UFs that you should have;
in this lookup there must be at least one field that you can call "host", but you could also insert other information to enrich your searches;
you can populate this lookup manually or from a CMDB;
after you have to run a search like this:

| metasearch index=_internal
| eval host=upper(host)
| stats count BY host
| append [ | inputlookup perimeter.csv | eval host=upper(host), count=0 | fields host, count ]
| stats sum(count) AS Total BY host
| where Total=0

In this way you have the list of all not reporting UFs and you can crete an alert.
If you don't use the last row, you can display in a dashboard the situation of all your forwarders.

Bye.
Giuseppe

Re: Determine which Active servers with Universal Forwarder areNOT sending logs to Splunk

geoffmx — Tue, 03 Sep 2019 08:00:52 GMT

Thanks gcusello! This is exactly what I was looking for 🙂

Would you happen to know a way to dynamically update the lookup via a CMDB? I can update manually, but automating it would be ideal.

Re: Determine which Active servers with Universal Forwarder areNOT sending logs to Splunk

gcusello — Tue, 03 Sep 2019 08:06:15 GMT

See DB-Connect App ( https://splunkbase.splunk.com/app/2686/ ) to execute queries on a DB and update a lookup.
Bye.
Giuseppe

Re: Determine which Active servers with Universal Forwarder areNOT sending logs to Splunk

geoffmx — Tue, 03 Sep 2019 08:22:03 GMT

Thank you, once again!