Getting Data In

Determine which Active servers with Universal Forwarder areNOT sending logs to Splunk

geoffmx
Explorer

We have a bunch of servers with UFs installed. These servers may have different operational states. For example, "Active", "Build in Progress", "Decommissioned", and "Decom in Progress". We use ServiceNow for the asset inventory.

This is the search query used to determine the version of the UFs installed.

index=_internal source=*metrics.log component=Metrics group=tcpin_connections | dedup hostname | table hostname sourceIp os arch version

I would like to be able to get the Active servers that have splunk UF installed and NOT reporting to splunk.

Also, looking for a way to dynamically update the list of Active servers, as this list changes whenever there are new servers onboarded or old servers decommissioned.

I've looked at the Splunk App/Add-on for ServiceNow and could not find an option to do this.

I've also looked at https://docs.splunk.com/Documentation/Splunk/7.3.1/DMC/Configureforwardermonitoring, but am not sure how to go about configuring it to dynamically update the asset list with the Active servers.

Any help will be much appreciated.

0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi geoffmx,
the way to have the list of not reporting UFs is the following:
at first you have to create a lookup (e.g. called "perimeter.csv") containing the list of all hostnames of the UFs that you should have;
in this lookup there must be at least one field that you can call "host", but you could also insert other information to enrich your searches;
you can populate this lookup manually or from a CMDB;
after you have to run a search like this:

| metasearch index=_internal
| eval host=upper(host)
| stats count BY host
| append [ | inputlookup perimeter.csv | eval host=upper(host), count=0 | fields host, count ]
| stats sum(count) AS Total BY host
| where Total=0

In this way you have the list of all not reporting UFs and you can crete an alert.
If you don't use the last row, you can display in a dashboard the situation of all your forwarders.

Bye.
Giuseppe

View solution in original post

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi geoffmx,
the way to have the list of not reporting UFs is the following:
at first you have to create a lookup (e.g. called "perimeter.csv") containing the list of all hostnames of the UFs that you should have;
in this lookup there must be at least one field that you can call "host", but you could also insert other information to enrich your searches;
you can populate this lookup manually or from a CMDB;
after you have to run a search like this:

| metasearch index=_internal
| eval host=upper(host)
| stats count BY host
| append [ | inputlookup perimeter.csv | eval host=upper(host), count=0 | fields host, count ]
| stats sum(count) AS Total BY host
| where Total=0

In this way you have the list of all not reporting UFs and you can crete an alert.
If you don't use the last row, you can display in a dashboard the situation of all your forwarders.

Bye.
Giuseppe

0 Karma

geoffmx
Explorer

Thanks gcusello! This is exactly what I was looking for 🙂

Would you happen to know a way to dynamically update the lookup via a CMDB? I can update manually, but automating it would be ideal.

0 Karma

gcusello
SplunkTrust
SplunkTrust

See DB-Connect App ( https://splunkbase.splunk.com/app/2686/ ) to execute queries on a DB and update a lookup.
Bye.
Giuseppe

0 Karma

geoffmx
Explorer

Thank you, once again!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...