Cannot forward data to Splunk Server

[triptrops]

Hi All,

I am a newbie on Splunk and I am trying to setup a Splunk server and a Splunk Light forwarder to forward data to it, here are the things that I did during the setup:

1) I installed a full Splunk on a Linux server(Server1). Started the Splunk daemon and installed *nix app.
2) I enabled a script 'df' to test if it is workin. Indeed the monitor is working.
3) I then enabled the receiving by go to Manager->Configure Receiving and Forwarding-> set port 9997 as the listening port.
4) Installed another full Splunk on another Linux server (Server2). Installed the *nix app also and have it working.

I then configured it to be a forwarder via the following commands:

cd /opt/splunk/bin
./splunk start
./splunk enable app SplunkLightForwarder
./splunk restart
./splunk add forward-server server1:9997

My inputs.conf (/opt/splunk/etc/apps/unix/local/inputs) is liked this one:

    [script://./bin/cpu.sh]
    [script://./bin/df.sh]
    disabled = 0
    [script://./bin/hardware.sh]
    [script://./bin/interfaces.sh]
    [script://./bin/iostat.sh]
    [script://./bin/lastlog.sh]
    [script://./bin/lsof.sh]
    [script://./bin/netstat.sh]
    [script://./bin/openPorts.sh]
    [script://./bin/package.sh]
    [script://./bin/protocol.sh]
    [script://./bin/ps.sh]
    [script://./bin/rlog.sh]
    [script://./bin/time.sh]
    [script://./bin/top.sh]
    [script://./bin/usersWithLoginPrivs.sh]
    [script://./bin/vmstat.sh]
    [script://./bin/who.sh]

My outputs.conf (/opt/splunk/etc/apps/unix/local/outputs.conf) is liked this one:

 [tcpout]
defaultGroup = server1.domain.com_9997
disabled = false

[tcpout:server1.domain.com_9997]
server = server1.domain.com:9997

[tcpout-server://server1.domain.com:9997]

I restarted both servers Splunk Daemon but still the server1 cannot see the data from server2.

Did I miss any configurations? please advise, thank you very much.

[triptrops · Accepted solution]

The forwarding of data from the client is now working. It just happened that I have my Splunk server as a client and have my outputs forwarded to another Splunk server. I am not 100% sure this was the cause but this was the only thing I changed before the forwarding worked.

Thanks everyone for the help.

[jasonnadeau]

You may want to check that IPTables is off or allowing TCP 9997 outbound from Server 2 and inbound to server 1.

You may find a useful message in the splunkd.log on your light forwarder. Normally it will indicate if a sucessful network connection has been made. Look here: /opt/splunk/var/log/splunk/splunkd.log

[tdinh]

You should add the following line -on the client side-

splunk add monitor /var/log

where /var/log is the directory of logs files.
Good luck

[tdinh]

I've same problems with splunk v 4.2.3