About the Splunk Technology Add-on (TA) for Unix and Linux
Complete newb with Splunk.
Running demo of Splunk on Linux CentOS 6.2 - 64-bit
VM Linux guest (CentOS 6.2-64bit) feeding data into Splunk using the Universal Forwarder - splunk-4.3.3-128297-linux-2.6-x86_64.rpm
Splunk indexer/receiver configured as below:
Data input type TCP > 514
Source > Accept connections from all hosts? Yes
Source Type > Manual
Source Type > syslog
This work just as expected, great!
I then tried out the Splunk Technology Add-on (TA) for Unix and Linux, as it would be nice to have performance metrics of various hosts in Splunk, and this is where I'm having problems.
I have downloaded *NIX 4.5 to Splunk and it is installed and it is available from the WebGUI under App > *NIX 4.5, and has an enabled status under Manager > Apps
I can collect and see performance data from the local host , the Splunk indexer/receiver, but not from the host I configured below.
I downloaded the Splunk Technology Add-on (TA) for Unix and Linux to VM Linux guest, taking note that it has to be installed manually, I used this guide, section "Install the TA on a universal forwarder"
http://docs.splunk.com/Documentation/UnixApp/latest/User/InstalltheSplunkTechnicalAddonforUnixandLin... - which I followed.
I also noted the comments about various typos in the howto.
I also noted that for Step 1 under section "Enable data and scripted inputs in the TA", it advises you to copy inputs.conf to a directory named "local', well on my install there is no directory named "local"
"1. Make a copy of $SPLUNK_HOME/etc/apps/Splunk_TA_nix/default/inputs.conf and place it into $SPLUNK_HOME/etc/apps/Splunk_TA_nix/local. "
So I had to create the directory and then assign splunk as owner and group perms, not sure this is a documentation error or a sign of trouble.
I then followed the rest of the guide, setting all of the disabled attributes to 0, thus enabling all the monitors, and restarted Splunk, it restarted without errors.
I got back to my Splunk receiver/indexer, using the Splunk WebGUI I select App > *NIX 4.5, expecting see my host added to it, but it is not, it still has only the local host.
What have I missed?
Hopefully this will help others take alot of surfing for answers out of this simple procedure:
rpm -Uvh splunkforwarder-5.0.2-149561.i386.rpm
tar xzvf $TMP/Splunk_TA_nix-4.7.0-156739.tgz
cp $SPLUNK_HOME/etc/apps/Splunk_TA_nix/default/inputs.conf $SPLUNK_HOME/etc/apps/Splunk_TA_nix/local/.
chown -R splunk.splunk $SPLUNK_HOME/etc/apps/Splunk_TA_nix
$SPLUNK_HOME/bin/splunk edit user admin -password 'skcorknulps' -role admin -auth admin:changeme
$SPLUNK_HOME/bin/splunk add forward-server hostname:9997
$SPLUNK_HOME/bin/splunk list forward-server
Username = admin
Passwd = skcorknulps
Also helpful to check web interface to verify: App-> *Nix-4.6 -> Configs -> Hardware Configurations by Host
NOTES: No firewall changes needed on clients running UF. Must open 9997/TCP on server (unless using a non standard port). Example iptables rule:
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp -s ip.of.client.here --dport 9997 -j ACCEPT
-INPUT -m state --state NEW -m tcp -p tcp -s ip.of.client.here --dport 9997 -j ACCEPT
Got it working, I think I got confused over the paths
I did this and it works fine now.
On Splunk server (receiver)
Download/install Splunk TA for Unix and Linux to the Splunk server (receiver) and enabled it by going to Manager|Apps|Enable
On host you want to collect data from (sender)
Download and install the Splunk Universal Forwarder to the Linux host - the assumption is made that this configured and sending data to Splunk, in brief it consisted of the following for RHEL6
rpm -ivh splunkforwarder-4.3.3-128297-linux-2.6-x86_64.rpm
Create init.d start-up script
/opt/splunkforwarder/bin/splunk enable boot-start
Enable start-up at boot
chkconfig splunk on
Open firewall port tcp 9997
iptables -I INPUT 4 -p tcp --dport 9997 -j ACCEPT
Copy the default inputs.conf file from $SPLUNK_HOME/splunkforwarder/etc/system/default to $SPLUNK_HOME/splunkforwarder/etc/system/local, check UID/GUIDs are splunk:splunk
Next edit the inputs.conf file located in $SPLUNK_HOME/splunkforwarder/etc/system/local to get something basic set-up.
host = mysender.local
disabled = false
Start the splunkforwarder
service start splunk
Tell Splunk that it now has a new sender
/opt/splunkforwarder/bin/splunk add forward-server mysender.local:9997
You should be getting data in Splunk now.
Now download and manually install Splunk Technology Add-on (TA) for Unix and Linux onto the host to collect data from (sender), it will end up here:
Copy the default inputs.conf file from $SPLUNK_HOME/splunkforwarder/etc/Splunk_TA_nix/default/inputs.conf to $SPLUNK_HOME/splunkforwarder/etc/apps/Splunk_TA_nix/local/
I had to create the "local" directory, also check UID/GUIDs are splunk:splunk
cp /opt/splunkforwarder/etc/apps/Splunk_TA_nix/default/inputs.conf /opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/
Next edit the inputs.conf file located in /opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/ and enable items you wish to monitor by setting the "disabled" attribute to "false" and then restart splunkforwarder.
Thanks for the reply.
I had a nagging feeling I omitted something in the post.
The VM host (the sender, now referred to as SenderA) does have the universal forwarder installed, I installed this first, configured it and get data sent out to Splunk, I then added Splunk TA for Unix and Linux, configured it on the hosts, SenderA, but the Splunk server is not getting anything.
SenderA sends data to Splunk using the universal forwarder without issue, I see syslog messages etc, it works fine.
I installed Splunk Technology Add-on (TA) for Unix and Linux to SenderA, this is not sending data to Splunk App for Unix and Linux to the Splunk server.
The Splunk server gets data from the locally installed Splunk TA for Unix and Linux.
Splunk server > local syslog data = OK
Splunk server > local data > Splunk TA for Unix and Linux = OK
Remote syslog data > univseral forwarder > Splunk server = OK
Remote syslog data > universal forwarder + Splunk TA for Unix and Linux > Splunk server = NOT OK
So I have to forward the output of Splunk TA for Unix and Linux on SenderA, to the universal forwarder on SenderA? This might be the part I've missed if this is the case, I was thinking that you configure the Splunk TA for Unix and Linux to send directly to Splunk, but what I think you are saying is that the data from Splunk TA for Unix and Linux has to be routed through the universal forwarder...?
It does seem I'm missing something here!
You should be set. If you're not seeing Splunk App for Unix and Linux data, there are a couple of possibilities as to why:
Check $SPLUNK_HOME/var/log/splunk/splunkd.log on the VM for clues as to why data is not getting into your full Splunk instance.
To get data from a remote Linux VM, you have to forward that data to your Splunk App for Unix and Linux instance using a universal forwarder.
On a secondary note, if you've got a fair number of servers you want to deploy the TA on, consider running a deployment server as well. I found it a touch tricky to start but is worth all the trouble of figuring out.