How do I configure a Splunk Forwarder on Linux?

[MillerTime]

What is a good procure to follow for installing a Splunk Universal Forwarder on a Linux host for the first time? A step by step process might help first time users get data into Splunk and understand some of the ways Splunk can be managed and configured.

[gneumann_splunk]

Try these instructions to configure the universal forwarder. There are seven steps at the top of this page, with each step details listed below on the page. Possibly you need to perform Step 6 and the deploy-poll command:
http://docs.splunk.com/Documentation/Forwarder/6.5.2/Forwarder/HowtoforwarddatatoSplunkEnterprise

[DUThibault]

Indeed, that did the trick. Why is this evidently crucial step not marked as mandatory? The text gives the impression that the step is optional, merely for centralised configuration control.

Anyway, I'm glad I've finally cracked this nut!

[gneumann_splunk]

Glad it worked for you!

[phadnett_splunk]

Hi DUThibault. A Universal Forwarder does not need to be configured as a Deployment Client in order to forward data to Splunk Enterprise. Inputs can be configured locally, instead of being pushed out by a Deployment Server. This is why it is marked as Optional under the instructions for sending data to a Splunk Enterprise indexer or indexer cluster.

[DUThibault]

But...It DOES NOT WORK unless I add the deploy-poll step! Something must be off in our setup; one thing I noticed is that the selection I made of data to forward (splunk add monitor) was apparently completely ignored, since I had to specify it again from the central deployment server control interface when I 'added data'.

Note the inputstatus line: 9997:192.168.1.143:8089 ; it's as if 'splunk enable listen' by default went into 'deployment client' mode only.

[justinsimonelis]

correction to: "This will create a file: inputs.conf in /opt/splunk/etc/apps/search/local/ -- here is some documentation on inputs.conf:"

it's actually under /opt/splunkforwarder not /opt/splunk

[MillerTime]

fixed - thank you!

[DUThibault]

I just can't seem to get this started. I have a single-instance Splunk Enterprise environment, with a Universal Forwarder on another machine. But the Splunk Web interface stubbornly insists that "There are currently no forwarders configured as deployment clients to this instance."

On the forwarder:

splunk list forward-server
Active forwards:
192.168.1.136:9997
Configured but inactive forwards:
None
splunk list monitor
Monitored Directories:
$SPLUNK_HOME/var/log/splunk
...
/var/log
/var/log/anaconda.log
...
/var/log/messages
...
/var/log/yum.log
Monitored Files:
$SPLUNK_HOME/etc/splunk.version

On the Splunk Enterprise instance:

splunk list inputstatus
Cooked:tcp :
9997:10.116.2.206:8089
time opened = 2017-01-31T11:56:12-0500
9997:192.168.1.143:8089
time opened = 2017-01-31T11:56:02-0500
tcp
ExecProcessor:exec commands :
./bin/collector.path
time opened = 2017-01-31T11:55:59-0500
...
Raw:tcp :
tcp
TailingProcessor:FileStatus :
$SPLUNK_HOME/etc/splunk.version
...
/srv/elk/messages
file position = 5017121
file size = 5017121
percent = 100.00
type = finished reading
tcp_cooked:listenerports :
9997

Receive data: Forwarding and receiving: Receive data
shows that listening on 9997 is enabled.

But
Add Data: Select Forwarders
states "There are currently no forwarders configured as deployment clients to this instance."

Help?!?

[Thijxx]

Still a great tutorial after 1.5 years!

@Anonymous 8: there are two almost identical unix apps.

After installation of a unix app, there is nu unix folder in /opt/splunk/etc/apps/
There are: splunk_app_for_nix and Splunk_TA_nix

You should copy the Splunk_TA_nix as described here: http://docs.splunk.com/Documentation/UnixApp/5.0.2TA/User/InstalltheSplunkAdd-onforUnixandLinux

[MillerTime · Accepted solution]

Splunk Command Line Reference:
http://docs.splunk.com/Documentation/Splunk/latest/Admin/AccessandusetheCLIonaremoteserver

Note: the CLI may ask you to authenticate – it’s asking for the LOCAL credentials, so if you haven’t changed the admin password on the forwarder, you should use admin/changeme



Steps for Installing/Configuring Linux forwarders:

Step 1: Download Splunk Universal Forwarder:

http://www.splunk.com/download/universalforwarder

(64bit package if applicable!)

Step 2: Install Forwarder

Step 3: Enable boot-start/init script:

/opt/splunkforwarder/bin/splunk enable boot-start

(start splunk: /opt/splunkforwarder/splunk start)

Step 4: Enable Receiving input on the Index Server

Configure the Splunk Index Server to receive data, either in the manager:

Manager -> sending and receiving -> configure receiving -> new

or via the CLI:

/opt/splunk/bin/splunk enable listen 9997

Where 9997 (default) is the receiving port for Splunk Forwarder connections

Step 5: Configure Forwarder connection to Index Server:

/opt/splunkforwarder/bin/splunk add forward-server hostname.domain:9997

(where hostname.domain is the fully qualified address or IP of the index server (like indexer.splunk.com), and 9997 is the receiving port you create on the Indexer:
Manager -> sending and receiving -> configure receiving -> new)

Step 6: Test Forwarder connection:

/opt/splunkforwarder/bin/splunk list forward-server

Step 7: Add Data:

/opt/splunkforwarder/bin/splunk add monitor /path/to/app/logs/ -index main -sourcetype %app%

Where /path/to/app/logs/ is the path to application logs on the host that you want to bring into Splunk, and %app% is the name you want to associate with that type of data


This will create a file: inputs.conf in /opt/splunkforwarder/etc/apps/search/local/ -- here is some documentation on inputs.conf:

http://docs.splunk.com/Documentation/Splunk/latest/admin/Inputsconf


Note: System logs in /var/log/ are covered in the configuration part of Step 7. If you have application logs in /var/log/*/

Step 8 (Optional): Install and Configure UNIX app on Indexer and *nix forwarders:

On the Splunk Server, go to Apps -> Manage Apps -> Find more Apps Online -> Search for ‘Splunk App for Unix and Linux’ -> Install the "Splunk App for Unix and Linux'

Restart Splunk if prompted, Open UNIX app -> Configure


Once you’ve configured the UNIX app on the server, you'll want to install the related Add-on: "Splunk Add-on for Unix and Linux" on the Universal Forwarder. Go to http://apps.splunk.com/ and find the "Splunk Add-on for Unix and Linux" (Note you want the ADD-ON, not the App - there is a difference!).
Copy the contents of the Add-On zip file to the Universal Forwarder, in: /opt/splunkforwarder/etc/apps/. If done correctly, you will have the directory "/opt/splunkforwarder/etc/apps/Splunk_TA_nix" and inside it will be a few directories along with a README & license files.

Restart the Splunk forwarder (/opt/splunkforwarder/bin/splunk restart)


Note: The data collected by the unix app is by default placed into a separate index called ‘os’ so it will not be searchable within splunk unless you either go through the UNIX app, or include the following in your search query: “index=os” or “index=os OR index=main” (don’t paste doublequotes)

Step 9 (Optional): Customize UNIX app configuration on forwarders:

Look at inputs.conf in /opt/splunkforwarder/etc/apps/unix/local/ and /opt/splunkforwarder/etc/apps/unix/default/

The ~default/inputs. path shows what the app can do, but everything is disabled. The ~local/inputs.conf shows what has been enabled – if you want to change polling intervals or disable certain scripts, make the changes in ~local/inputs.conf.

Step 10 (Optional): Configure File System Change Monitoring (for configuration files):

http://docs.splunk.com/Documentation/Splunk/4.3.2/Data/Monitorchangestoyourfilesystem

Note that Splunk also has a centralized configuration management server called Deployment Server. This can be used to define server classes and push out specific apps and configurations to those classes. So you may want to have your production servers class have the unix app configured to execute those scripts listed in ~local/inputs at the default values, but maybe your QA servers only need a few of the full stack, and at longer polling intervals. Using Deployment Server, you can configure these classes, configure the app once centrally, and push the appropriate app/configuration to the right systems.

[nk-1]

How about a GUI Installer, like the Windows Splunk UniversalForwarder has?
That was an easy install.

I haven't used Linux in about 2 years, and all this is vaguely familiar.
If I mess up the Oracle Linux server this Splunk forwarder needs to be installed on, the Oracle DBA will be annoyed (to put it mildly) with me.

[oraant]

Thanks very much!!!
Much better than the official documentation!!!

[pelin_kurt_2]

It works!
Thank you very much for a great tutorial.

[flakrat]

The contents of this comment should replace: http://docs.splunk.com/Documentation/Splunk/6.2.3/Data/Useforwardingagentstogetdata

I spent hours trying to get the universal forwarder working on a linux box using that link, it only took a few minutes using MillerTime's instructions.

[amnonh]

Yes, Splunk could do with more clear documentation and examples for sure.

[mnatalier]

You need to ensure that the sysstat package is installed on the forwarder if you are running ubuntu. This add-on makes use of sar to provide data.

[MillerTime]

search $SPLUNK_HOME/etc/system/local/ and $SPLUNK_HOME/etc/apps (recursively) for "inputs.conf". If the command to add the input was successful there should be an associated inputs.conf with the specifications set by the command. If you're getting an error, what is the exact text?

[windyita]

I met with the same problem in step 7. after leaving off the '-index main' part, still no data is written in the config file. What's wrong? besides step 7 should be executed in which host?? thank you!

[nashish]

It's good Article

[MillerTime]

If the command is giving you an error then it likely won't write to the inputs.conf file. Strange that the main index doesn't exist yet...try leaving off the '-index main' part. The main index is where new data goes by default anyways.