All Apps and Add-ons

How to install the Splunk Add-on for Unix and Linux and get it to work with the Splunk App for Unix and Linux?

neerajshah81
Path Finder

Hello, I have single server which has Splunk Enterprise installed. My requirement is to monitor some linux hosts in our network, have them send performance data like CPU/Memory/DISK stats etc to the Splunk server. I have installed the Splunk App and Add-on for Unix and Linux (*NIX App and *NIX Add-on) my Splunk server. I also went ahead and installed the Universal Forwarder on one of my Linux hosts. What's next ? I am not getting any CPU/MEMORY/DISK data in my Splunk dashboard from the linux host. When i click on the "Splunk App for Unix and Linux" app in my Splunk dashboard it shows empty .

I have looked at the official documentation for the Splunk Add-on for Unix and Linux. It talks about installing the Add-on on the Universal Forwarder. This did not make sense to me, as i have a dozen Linux hosts in my environment , so are we supposed to install add-on on each n every host we want to monitor? Also, the installation instructions for" add-on" say on one hand to install it on the universal forwarder and on the other hand it says post install login to the Splunk interface and enable/disable the parameters, scripted inputs etc you want to monitor. The Universal Forwarder does not even have a Splunk Web interface .

If the add-on does need to be installed on each and every device we want to monitor, how do we configure the options like what inputs to monitor when there is no web interface on the device (which also has the universal forwarder installed)?

As an alternate the document talks about running the setup of add-on via command line. So i went ahead and ran the below as shown in their documentation on my linux host :--

$SPLUNK_HOME/bin/splunk cmd $SPLUNK_HOME/etc/apps/Splunk_TA_nix/bin/setup.sh*
This command then asks for splunk username and password.

If i enter her my splunk server interface admin credentials it says LOGIN Failed. Also, I have not setup any credentials when i installed the univ. forwarder on this host, so if i leave the username/pwd empty, it says LOGIN Failed again. What creds is it really expecting ?

Any help will be appreciated.
Neeraj

0 Karma
1 Solution

nkhetia
Path Finder

DUThibault
Contributor

Turns out http://docs.splunk.com/Documentation/UnixAddOn/5.2.4/User/Platformandhardwarerequirements is lying. If you install Splunk_TA_nix on forwarders, you must also install it on the server(s) that have installed splunk_app_for_nix.

If, and only if, your server (the Splunk instance where splunk_app_for_nix is installed) is to be monitored in exactly the same way as the clients (the machines where a Universal Forwarder and Splunk_TA_nix are installed), then you can install splunk_app_for_nix and Splunk_TA_nix on the server and then deploy that to the clients. Be aware that despite having data inputs (scripts and monitored files and directories) listed separately in the Settings: (Data) Data inputs screen, you will NOT be able to change any setting (such as enabling/disabling) on one without having it change on the other.

If you want the server to merely run splunk_app_for_nix and receive Splunk_TA_nix data from forwarders, you must install splunk_app_for_nix AND Splunk_TA_nix on the server (leave Splunk_TA_nix's scripts and monitors disabled, their default setting), and then install Splunk_TA_nix on the forwarders (the clients). You will need to connect to each client and locally run Splunk_TA_nix/bin/setup.sh to enable the scripts and monitors. One key difference with this setup is that the forwarder data inputs will NOT show on the Settings: (Data) Data inputs screen.

Note that if you don't intend to change the settings often if at all, you can edit splunk-add-on-for-unix-and-linux_524.tgz to change the "disabled=1" lines in Splunk_TA_nix/default/inputs.conf before deploying the .tgz on the clients. This will save you from having to run setup.h.

DUThibault
Contributor

I'm having the same problem. I first had Splunk_TA_nix and splunk_app_for_nix deployed on my Splunk instance and its forwarders, and that worked fine. But I wanted to have the data inputs exclude the server (and if you disable the scripts on the server, the deployment service disables them on the forwarders too), so I now have a server with splunk_app_for_nix and the forwarders with Splunk_TA_nix. I've run Splunk_TA_nix/bin/setup.sh on the forwarders to enable just one source type (bandwidth) to start with. The Splunk server receives some data but throws it away with this message:

Received event for unconfigured/disabled/deleted index=os
with source="source::bandwidth" host="host::dut-centos7"
sourcetype="sourcetype::bandwidth". So far received events
from 1 missing index(es).

Unlike before, the Splunk_TA_nix scripts don't show up in the Source types screen. splunk_app_for_nix has been run and configured, so why is the 'os' index not created?

0 Karma

nkhetia
Path Finder

DUThibault
Contributor

I downvoted this post because this "solution" slaves the server and forwarders, which does not match the distribution recommended in the documentation (which states splunk_ta_nix should be installed only on the forwarders). in particular, one could not then enable or disable a script data input on the server without having the setting immediately propagated to the forwarders.

0 Karma

neerajshah81
Path Finder

Nilesh . Thanks for the response. Does the Add-on also need to be deployed on the central Splunk server ?

I have deployed the add-on in both places ( UF's and Central Splunk server). Will the input options not conflict with one another ? For instance, lets say if i enable scripted input of "cpu.sh" on Central but not on the UF side, what will be the end result ? Will i get the cpu metrics of the UF server ?

I am still not getting any performance metrics from my Universal forwarder. I have followed the steps as per the documetation. Not sure what's missing.

Thanks
Neeraj

0 Karma

nkhetia
Path Finder

Neeraj,

Add-on needs to be installed on UF only. Install Splunk app for unix/linux on central Splunk server. restart Splunk on UF and see if any errors in splunked.log. also check inputs.conf under local folder, it should have stanzas as follows:

[script://./bin/cpu.sh]
disabled = false
index = os

Thanks.

neerajshah81
Path Finder

Nilesh, i do not have any inputs.conf file under local folder. I am referring to /opt/splunkforwarder/etc/apps/Splunk_TA_nix/local folder on my UF server. I had enabled all scripted inputs via command line method. For instance, after installing the Add-on on my UF, i ran the below commands as per the official documentation. This did not create any inputs.conf file under /local.

$SPLUNK_HOME/bin/splunk cmd $SPLUNK_HOME/etc/apps/Splunk_TA_nix/bin/setup.sh
This prompted me for admin credentials and showed a menu. I selected the " enable-all" option which i assumed enables all the options.

I did not use the scripted file method.

0 Karma

nkhetia
Path Finder

Neeraj - can you post cpu.sh stanza from inputs.conf ? also do you see any messages related with this add-on/app in Splunkd.log?

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.