topic Re: Splunk Technology Add-on (TA) for Unix and Linux in All Apps and Add-ons

Splunk Technology Add-on (TA) for Unix and Linux

mikebrooker — Mon, 28 Sep 2020 12:15:25 GMT

Splunk v4.3.2
*NIX 4.5
About the Splunk Technology Add-on (TA) for Unix and Linux

Complete newb with Splunk.

Overview

Running demo of Splunk on Linux CentOS 6.2 - 64-bit

VM Linux guest (CentOS 6.2-64bit) feeding data into Splunk using the Universal Forwarder - splunk-4.3.3-128297-linux-2.6-x86_64.rpm

Splunk indexer/receiver configured as below:

Data input type TCP > 514
Source > Accept connections from all hosts? Yes
Source Type > Manual
Source Type > syslog

This work just as expected, great!

I then tried out the Splunk Technology Add-on (TA) for Unix and Linux, as it would be nice to have performance metrics of various hosts in Splunk, and this is where I'm having problems.

I have downloaded *NIX 4.5 to Splunk and it is installed and it is available from the WebGUI under App > *NIX 4.5, and has an enabled status under Manager > Apps

I can collect and see performance data from the local host , the Splunk indexer/receiver, but not from the host I configured below.

I downloaded the Splunk Technology Add-on (TA) for Unix and Linux to VM Linux guest, taking note that it has to be installed manually, I used this guide, section "Install the TA on a universal forwarder"
http://docs.splunk.com/Documentation/UnixApp/latest/User/InstalltheSplunkTechnicalAddonforUnixandLinux - which I followed.

I also noted the comments about various typos in the howto.

I also noted that for Step 1 under section "Enable data and scripted inputs in the TA", it advises you to copy inputs.conf to a directory named "local', well on my install there is no directory named "local"

Quote:
"1. Make a copy of $SPLUNK_HOME/etc/apps/Splunk_TA_nix/default/inputs.conf and place it into $SPLUNK_HOME/etc/apps/Splunk_TA_nix/local. "

So I had to create the directory and then assign splunk as owner and group perms, not sure this is a documentation error or a sign of trouble.

I then followed the rest of the guide, setting all of the disabled attributes to 0, thus enabling all the monitors, and restarted Splunk, it restarted without errors.

I got back to my Splunk receiver/indexer, using the Splunk WebGUI I select App > *NIX 4.5, expecting see my host added to it, but it is not, it still has only the local host.

What have I missed?

Re: Splunk Technology Add-on (TA) for Unix and Linux

malmoore — Mon, 13 Aug 2012 21:58:44 GMT

Hi,

To get data from a remote Linux VM, you have to forward that data to your Splunk App for Unix and Linux instance using a universal forwarder.

Configure the Splunk App for Unix and Linux instance to be a receiver.
Download and install the universal forwarder on the Linux server that you want to get data from.
Configure the universal forwarder to send data to your Splunk App for Unix and Linux instance.
Manually install (and configure) the Splunk TA for Unix and Linux on the Linux server that you want to get data from, as you attempted to before.
Your Linux server's data should then appear in the Splunk App for Unix and Linux instance.

Also:

I've fixed the typo in the documentation.
It's okay if the 'local' folder doesn't exist, you can create it and that doesn't mean there's anything wrong. 🙂

Relevant reading:

Cheers!

Re: Splunk Technology Add-on (TA) for Unix and Linux

iunderwood — Tue, 14 Aug 2012 01:13:05 GMT

On a secondary note, if you've got a fair number of servers you want to deploy the TA on, consider running a deployment server as well. I found it a touch tricky to start but is worth all the trouble of figuring out.

Re: Splunk Technology Add-on (TA) for Unix and Linux

mikebrooker — Tue, 14 Aug 2012 13:38:38 GMT

Thanks for the reply.

I had a nagging feeling I omitted something in the post.

The VM host (the sender, now referred to as SenderA) does have the universal forwarder installed, I installed this first, configured it and get data sent out to Splunk, I then added Splunk TA for Unix and Linux, configured it on the hosts, SenderA, but the Splunk server is not getting anything.

SenderA sends data to Splunk using the universal forwarder without issue, I see syslog messages etc, it works fine.

I installed Splunk Technology Add-on (TA) for Unix and Linux to SenderA, this is not sending data to Splunk App for Unix and Linux to the Splunk server.

The Splunk server gets data from the locally installed Splunk TA for Unix and Linux.

So:

Splunk server > local syslog data = OK
Splunk server > local data > Splunk TA for Unix and Linux = OK
Remote syslog data > univseral forwarder > Splunk server = OK
Remote syslog data > universal forwarder + Splunk TA for Unix and Linux > Splunk server = NOT OK

So I have to forward the output of Splunk TA for Unix and Linux on SenderA, to the universal forwarder on SenderA? This might be the part I've missed if this is the case, I was thinking that you configure the Splunk TA for Unix and Linux to send directly to Splunk, but what I think you are saying is that the data from Splunk TA for Unix and Linux has to be routed through the universal forwarder...?

It does seem I'm missing something here!

Cheers!

Re: Splunk Technology Add-on (TA) for Unix and Linux

malmoore — Mon, 28 Sep 2020 12:16:04 GMT

You should be set. If you're not seeing Splunk App for Unix and Linux data, there are a couple of possibilities as to why:

your forwarding and receiving is not set up correctly.
the Splunk TA for Unix and Linux is not installed correctly (it must be in $SPLUNK_HOME/etc/apps/Splunk_TA_nix on the computer that you want to collect data from)
you aren't looking in the right index for the data (the Splunk App for Unix and Linux puts data into an index called 'os')

Check $SPLUNK_HOME/var/log/splunk/splunkd.log on the VM for clues as to why data is not getting into your full Splunk instance.

Re: Splunk Technology Add-on (TA) for Unix and Linux

mikebrooker — Mon, 28 Sep 2020 12:30:38 GMT

Got it working, I think I got confused over the paths

I did this and it works fine now.

On Splunk server (receiver)
Download/install Splunk TA for Unix and Linux to the Splunk server (receiver) and enabled it by going to Manager|Apps|Enable

On host you want to collect data from (sender)
Download and install the Splunk Universal Forwarder to the Linux host - the assumption is made that this configured and sending data to Splunk, in brief it consisted of the following for RHEL6

Install
rpm -ivh splunkforwarder-4.3.3-128297-linux-2.6-x86_64.rpm

Create init.d start-up script
/opt/splunkforwarder/bin/splunk enable boot-start

Enable start-up at boot
chkconfig splunk on

Open firewall port tcp 9997
iptables -I INPUT 4 -p tcp --dport 9997 -j ACCEPT

Copy the default inputs.conf file from $SPLUNK_HOME/splunkforwarder/etc/system/default to $SPLUNK_HOME/splunkforwarder/etc/system/local, check UID/GUIDs are splunk:splunk

Next edit the inputs.conf file located in $SPLUNK_HOME/splunkforwarder/etc/system/local to get something basic set-up.

[default] host = mysender.local [monitor:///var/log/secure] disabled = false

Start the splunkforwarder
service start splunk
or
/etc/init.d/splunk start

Tell Splunk that it now has a new sender
/opt/splunkforwarder/bin/splunk add forward-server mysender.local:9997

You should be getting data in Splunk now.

Now download and manually install Splunk Technology Add-on (TA) for Unix and Linux onto the host to collect data from (sender), it will end up here:
$SPLUNK_HOME/splunkforwarder/etc/apps/Splunk_TA_nix

Copy the default inputs.conf file from $SPLUNK_HOME/splunkforwarder/etc/Splunk_TA_nix/default/inputs.conf to $SPLUNK_HOME/splunkforwarder/etc/apps/Splunk_TA_nix/local/

I had to create the "local" directory, also check UID/GUIDs are splunk:splunk
cp /opt/splunkforwarder/etc/apps/Splunk_TA_nix/default/inputs.conf /opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/

Next edit the inputs.conf file located in /opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/ and enable items you wish to monitor by setting the "disabled" attribute to "false" and then restart splunkforwarder.

Re: Splunk Technology Add-on (TA) for Unix and Linux

skcorknulps — Mon, 13 May 2013 17:57:20 GMT

Hopefully this will help others take alot of surfing for answers out of this simple procedure:

Download UF http://www.splunk.com/download/universalforwarder
Install UF rpm -Uvh splunkforwarder-5.0.2-149561.i386.rpm
Download Tech Add On http://splunk-base.splunk.com/apps/33800/splunk-add-on-for-unix-and-linux
Configure TA

cd $SPLUNK_HOME/etc/apps/ tar xzvf $TMP/Splunk_TA_nix-4.7.0-156739.tgz mkdir $SPLUNK_HOME/etc/apps/Splunk_TA_nix/local cp $SPLUNK_HOME/etc/apps/Splunk_TA_nix/default/inputs.conf $SPLUNK_HOME/etc/apps/Splunk_TA_nix/local/. vi $SPLUNK_HOME/etc/apps/Splunk_TA_nix/local/inputs.conf chown -R splunk.splunk $SPLUNK_HOME/etc/apps/Splunk_TA_nix

Configure UF

$SPLUNK_HOME/bin/splunk edit user admin -password 'skcorknulps' -role admin -auth admin:changeme $SPLUNK_HOME/bin/splunk add forward-server hostname:9997 $SPLUNK_HOME/bin/splunk/start

Verify UF

$SPLUNK_HOME/bin/splunk list forward-server Username = admin Passwd = skcorknulps

Also helpful to check web interface to verify: App-> *Nix-4.6 -> Configs -> Hardware Configurations by Host

NOTES: No firewall changes needed on clients running UF. Must open 9997/TCP on server (unless using a non standard port). Example iptables rule:

RHEL5:
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp -s ip.of.client.here --dport 9997 -j ACCEPT

RHEL6:
-INPUT -m state --state NEW -m tcp -p tcp -s ip.of.client.here --dport 9997 -j ACCEPT