Deployment Architecture

Splunk Forwarder connection to Cluster Master

samadmemon
Explorer

Hi All,

I am trying to build a query through which we can track if all the Splunk forwarders are connected to Cluster Master. Wanted to create an alert if there are issues when forwarder is not able to connect with Cluster master.

Could you please help with the query.

0 Karma

codebuilder
SplunkTrust
SplunkTrust

The cluster master already provides the functionality that you're looking for.
From the web UI on the master, go to Settings > Monitoring Console > Forwarders > Forwarders:Deployment

alt text

This dashboards provides all the information you're looking for on your forwarders. Status, last connection time, thruput, and so on.
There's also a pre-built alert available for "Missing Forwarders". You simply have to enable it:
Settings > Searches, Reports, and Alerts > App: Monitoring Console > DMC Alert - Missing Forwarders

alt text

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

gcusello
Legend

Hi @samadmemon,
you could check if an Universal Forwarder is sending internal logs to Indexers and this means that the connection is ok.
To do this, you have to create a lookup with the server to monitor list (called e.g. perimeter.csv) and run something like this:

| metasearch index=_internal
| eval host=lower(host)
| stats count BY host
| append [ | inputlookup perimeter.csv | eval host=lower(host), count=0 | fields host count ]
| stats sum(count) AS total BY host
| where host=0

Ciao.
Giuseppe

0 Karma

samadmemon
Explorer

Hello Giuseppe,

Thanks for the response. Exact requirement is to track the forwarders which are not connected to Cluster master, which means that indexer discovery is set in outputs.conf of that forwarder and it is directly sending logs through IP address and not through the cluster master url

0 Karma

codebuilder
SplunkTrust
SplunkTrust

I'm a little confused by your last reply. You have indexer discovery configured on the forwarder(s) in question, but they are not communicating with the master? I may be misunderstanding your scenario, but with indexer discovery configured, forwarders will always communicate with the master to get a list of available indexers.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

gcusello
Legend

Hi @samadmemon,
sorry I wasn't so clear!
My hint is to to deploy (using Deployment Server) to all your Universal Forwarders a TA with outputs.conf.
In this way you can be sure about the configuration of your outputs.conf in all UFs.

Then in your outputs.conf you'll surely configure indexer discovery (it's a best practice), but the issue is that in this way you can have a better control on your outputs.conf and be sure that all the UFs are correctly configured.

Ciao.
Giuseppe

0 Karma

samadmemon
Explorer

Hi @codebuilder

Thanks for the response.

Exact requirement is to write a Splunk query to track that all the forwarders are always connected to cluster master.

0 Karma

gcusello
Legend

Hi @samadmemon,
sorry for the misunderstanding!
did you manage outputs.conf manually or by Deployment Server?
It's a good practice to create a Technical Add-On (called e.g. TA_Forwarders) containing only two files:

  • outputs.conf,
  • deploymentclient.conf;

and deploy it in all your UFs.

In this way you can centrally manage your UFs and you haven't the problem.

Ciao.
Giuseppe

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!