All Apps and Add-ons

Distribute Apps to multiple Indexers

zowa
Engager

What is the best way to distribute apps to multiple indexers?

For instance, if I have dozens of indexers (installed on Linux) in my distributed environment, and I want to monitor them with the Linux App, how should I install the Linux App TA on the indexers? Should I install a forwarder on each indexer, and then manage them all with the deployment server?

The point is: If I have to change the configuration of the TA, or even install new Apps, I would like to do it from a central point, instead of configuring server by server manually.

Is it ok to have the forwarder on the same machine as the indexer?

Thanks!

Tags (1)
0 Karma
1 Solution

rsennett_splunk
Splunk Employee
Splunk Employee

Using a forwarder on the same machine as the indexer is actually a very good practice, and very straight forward to arrange on Linux especially.

For the sake of others listening, I'll answer the burning question:Q: Why wouldn't you just make the indexer a deployment client and cut out the middle man?
A: Well, once the deployment-client.conf is in place, if you accidentally delete or fat finger something in $SPLUNK_HOME/etc/deployment-apps on the deployment server... you might temporarily have a forwarder that isn't sending.
Do that with an indexer (delete a deployment-app, or add a typo) and you could lose critical transforms or props on data that is STILL arriving. Also... think about how you set up an app on a forwarder... usually you tell the deployment server to issue a "restart" after the config is updated. You want you indexers restarting like that? Probably not...

Especially because people like to set up a server class called [all-UF] and something like [All-Indexers] would just follow suit... and with one key stroke you could lobotomize all your indexers.

Have I made my case? 🙂

So: Yes... it's a great idea to monitor your indexer boxes with a forwarder. The only thing you want to keep in mind, is that the forwarder will be configured by default for management port 8089. That's the same default as the indexer.

So, depending on how you're going to deploy/install/config all these forwarders, here's what you need to know:

The management port on a forwarder on a clean install will use the management port that is set in
$SPLUNK_HOME/etc/system/default/web.conf (yes, even though there is no GUI... those settings are in web.conf just like the indexer.

The setting looks like this: mgmtHostPort = 127.0.0.1:8089

To override that - you can interactively answer the question that comes up when the forwarder sees that port 8089 is already bound... OR

You can distribute pre-configured forwarders that have
$SPLUNK_HOME/etc/system/local/web.conf already created and edited to reflect the new port

That will set up a nice safe environment to monitor the indexer boxes...

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!

View solution in original post

0 Karma

rsennett_splunk
Splunk Employee
Splunk Employee

Using a forwarder on the same machine as the indexer is actually a very good practice, and very straight forward to arrange on Linux especially.

For the sake of others listening, I'll answer the burning question:Q: Why wouldn't you just make the indexer a deployment client and cut out the middle man?
A: Well, once the deployment-client.conf is in place, if you accidentally delete or fat finger something in $SPLUNK_HOME/etc/deployment-apps on the deployment server... you might temporarily have a forwarder that isn't sending.
Do that with an indexer (delete a deployment-app, or add a typo) and you could lose critical transforms or props on data that is STILL arriving. Also... think about how you set up an app on a forwarder... usually you tell the deployment server to issue a "restart" after the config is updated. You want you indexers restarting like that? Probably not...

Especially because people like to set up a server class called [all-UF] and something like [All-Indexers] would just follow suit... and with one key stroke you could lobotomize all your indexers.

Have I made my case? 🙂

So: Yes... it's a great idea to monitor your indexer boxes with a forwarder. The only thing you want to keep in mind, is that the forwarder will be configured by default for management port 8089. That's the same default as the indexer.

So, depending on how you're going to deploy/install/config all these forwarders, here's what you need to know:

The management port on a forwarder on a clean install will use the management port that is set in
$SPLUNK_HOME/etc/system/default/web.conf (yes, even though there is no GUI... those settings are in web.conf just like the indexer.

The setting looks like this: mgmtHostPort = 127.0.0.1:8089

To override that - you can interactively answer the question that comes up when the forwarder sees that port 8089 is already bound... OR

You can distribute pre-configured forwarders that have
$SPLUNK_HOME/etc/system/local/web.conf already created and edited to reflect the new port

That will set up a nice safe environment to monitor the indexer boxes...

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!

View solution in original post

0 Karma

rsennett_splunk
Splunk Employee
Splunk Employee

good job! Super easy on *nix. Glad it worked out!

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!
0 Karma

zowa
Engager

Thanks!

I had also to adapt the splunk script generated by "splunk enable boot-start" on the forwarder, renaming it to splunkforwarder (and adapting the script´s header changing splunk to splunkforwarder), and then enable it to run at boot with a update-rc.d splunkforwarder defaults (I am using Debian). With that, it was possible to have both indexer and forwarder starting after boot.

It looks ok now! 🙂

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.