install splunk receiver and forwarder in 2 different OSX 10.6 servers

[cthacker]

I've downloaded Splunk for the first time and am trying to get it working to evaluate it.

I've installed it on one OS X server as a receiver and configured receiving to use the default port 9997. I've installed the *nix (unix) app as well. (server1.example.com:9997)

I then downloaded the universalforwarder on the other OS X 10.6 server and ran "./splunk add forward-server server1.example.com:9997" to configure it to forward to the first box.

I can't figure out how to view this 2nd server in the first one's interface. if i go to: http://localhost:8000/en-US/app/unix/about_dashboard, i only see hosts: server1.example.com:9997 and can view its logs but can't find the 2nd server.

[tgow]

I just added the port 9997 in Manager-->Forwarding & Receiving-->Receive Data-->New

The $SPLUNK_HOME/etc/apps/search/local/inputs.conf file was updated with the following:

[splunktcp://9997]

Make sure you restart the Splunk processes after updating the inputs.conf file on the Indexer.

[tgow]

Configure the $SPLUNK_HOME/etc/system/local/inputs.conf on the Forwarder (server2) with the following:

[monitor:///var/log/system.log]
disabled = false
followTail = 0
sourcetype = system

Restart Splunk

$SPLUNK_HOME/bin/splunk restart

Restart the Splunk Indexer (server1)

$SPLUNK_HOME/bin/splunk restart

Now log back in as the "admin" user and select "Launch the search app".

Does "server2" show up in the lists of hosts?

[cthacker]

it's showing up now... the indexer is using an old, invalid host name for this forwarder. i ran 'changeip' on the forwarder and fixed it. i'm curious where the indexer is grabbing this name from.

[tgow]

Your inputs.conf and outputs.conf files look fine. I would start with looking at the default Search app. Is the system listed in the Hosts section? Can you run the following search:

host=server2.example.com

Does any data come back?

[cthacker]

thanks for the suggestion. i tried this and nothing is returned. do i need to tell the indexer to grab the lone test forwarder? i don't see any indication that it sees it.

[kdenton]

if you run

Splunk list monitor to see if the files you are monitoring are actually showing up.

Also run the command Splunk list forward-server to make sure the server is showing up correctly.

[cthacker]

both of these seem to indicate that the lone test forwarder is configured correctly. i still don't see it mentioned anywhere within the splunk indexer gui.

[cthacker]

thanks for the suggestion. i did that and now my inputs.conf file contains:

[default]
[monitor:///var/log]

and output.conf contains:

[tcpout]
defaultGroup = server1.example.com_9997
disabled = false
[tcpout:server1.example.com_9997]
server = server1.example.com:997
[tcpout-server://server1.example.com:9997]

is this correct?
am i looking for the data in the right place on the indexer? within the unix app under "hosts"?

[tgow · Accepted solution]

Make sure that you add at least one data input on the 2nd server that you installed the Universal Forwarder on. The hosts will not show up until data is being monitored on the Universal Forwarder.

Just edit/create a $SPLUNK_HOME/etc/system/local/inputs.conf file and add the following stanza:

[monitor:///var/log]

Restart the Forwarder if you don't see any data coming into the 1st Server (Indexer).