How to implement Splunk Add-on for Unix and Linux to monitors hosts for search head clustering?

[lmjoin]

Hello
We have a search head clustering setup and deployment servers. We need to implement Splunk Add-on for Unix and Linux to monitors hosts. How can we do that?

[hurricanelabs]

1. install the Splunk_TA_nix on the splunk infrastructure as necessary
   • this will tell you where you need to install it:
   • http://docs.splunk.com/Documentation/AddOns/released/UnixLinux/Install
   • this will show you how to install it on the different servers:
   • https://docs.splunk.com/Documentation/AddOns/released/Overview/Distributedinstall
2. Enable data and scripted inputs as necessary in deployment-apps/Splunk_TA_nix/local/inputs.conf (or in another app) on the deployment server
   • I recommend creating a "baseline_nix_inputs" for this purpose
   • Push both the Splunk_TA_nix and your "baseline_nix_inputs" apps to your client machines using the deployment server
   • This will show you how to deploy apps to clients:
   • http://docs.splunk.com/Documentation/Splunk/latest/Updating/Updateconfigurations

[ggb667]

Do you guys have specific things you monitor though? 200MB per day per server seems really high.

[woodcock · Accepted solution]

The general approach for DS managed app is as follows:

Login to GUI of DS
Install app (e.g. Linux TA) on DS
Login to CLI of DS
su - splunk
mv $SPLUNK_HOME/etc/aps/Splunk_TA_nix $SPLUK_HOME/etc/deployment-apps/
cd $SPLUNK_HOME/etc/deployment-apps/Splunk_TA_nix/
rm -rf samples
mkdir local
cd local
cp ../default/inputs.conf .
vi inputs.conf

Remove every line EXCEPT for the stanza headers (lines that start with [ and end with ]) and the disabed = lines.
For any input that you desire, change the disabled = 1 (or disabled = true) lines to disabled = false.
Save the file.
Push it out.

[woodcock]

Also, set is_visible=false in app.conf in the local directory. This will keep people from interacting with it on your Search Head and messing up your settings.

[ggb667]

Do you guys have specific things you monitor though? 200MB per day per server seems really high.

[sloshburch]

Building on the brilliance of @woodcock, What are the best practices to create a local configuration file?

[woodcock]

I just use the CLI and copy app.conf from some other app. If the local directory does not exist in your app, just do mkdir local.

[koshyk]

I'm slightly confused by "monitor hosts" . Do you want to send the TA to the client systems or you thinking it just in your Clustering systems?

But in general, this is how you to do
1. Planning and Design => Decide your organisations Linux monitoring requirements and enable ONLY them in a separate app by copying the stanza from Splunk_TA_nix. (eg MY_nix_inputs/local/inputs.conf).
2. Endpoints/Clients via UF => Push Splunk_TA_nix && MY_nix_inputs app to relevant client systems which has Splunk Universal Forwarders (UF) installed and send the data to your Splunk cluster. Manage both the apps via your deployment server
3. Indexers => Install the Splunk_TA_nix in your indexers (slave-apps) or use the Splunk_TA_for_indexers (just the index time configs) to your Indexer via Cluster Master
4. Search Head Cluster => Install Splunk_TA_nix to your SH Cluster via deployer.
5. Heavy Forwarders => If you have HF in between clients and Indexers, install the TA via deployment server.