How to edit my configurations when setting up a forwarder?

[cboard]

I'm finding the instructions a little confusing but my understanding is I can have Splunk Enterprise on server 1 which is the indexer and the web interface etc.

Server 2 has the forwarder, so I can set up logs that are monitored and they are forwarded so they can be displayed on server 2. ]

Server 1 according to netstat has 9997 established from the forwarder and the forwarder is also showing the same thing so it looks like the network connectivity between my two servers are working fine.

However, in the log file I am receiving the following:

03-19-2017 15:39:00.074 +0000 WARN  TcpOutputFd - Connect to 192.168.172.212:9997 failed. Connection refused
03-19-2017 15:39:00.074 +0000 ERROR TcpOutputFd - Connection to host=192.168.172.212:9997 failed
03-19-2017 15:39:01.907 +0000 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
03-19-2017 15:39:12.458 +0000 WARN  HttpListener - Socket error from 192.168.172.212 while idling: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
03-19-2017 15:39:13.907 +0000 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected

In /opt/splunkforwarder/etc/system/local/outputs.conf I have the following content:

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = 192.168.172.212:9997

#[tcpout-server://192.168.172.212:9997]

[monitor:/var/log/httpd/access_log]
server = 192.168.172.212:9997
disabled = 0

[monitor:/var/log/httpd/error_log]
server = 192.168.172.212:9997
disabled = 0

Both servers are running CentOS 7 x64

I'm not sure what I'm doing wrong so any help would be greatly appreciated.

I think I've made some progress, I think I had a forwarder added to the indexer the wrong way round so it was effectively forwarding to itself and failing.

In the indexer log, I am now seeing the following

03-19-2017 16:23:28.440 +0000 ERROR TcpInputProc - Message rejected. Received unexpected message of size=369295617 bytes from src=192.168.166.56:33078 in streaming mode. Maximum message size allowed=67108864. (::) Possible invalid source sending data to splunktcp port or valid source sending unsupported payload.

And on the forwarder I am seeing the following:

03-19-2017 16:28:27.099 +0000 WARN  HttpPubSubConnection - Unable to parse message from PubSubSvr:
03-19-2017 16:28:27.099 +0000 INFO  HttpPubSubConnection - Could not obtain connection, will retry after=65.832 seconds.
03-19-2017 16:28:33.324 +0000 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
03-19-2017 16:28:33.324 +0000 INFO  DC:PhonehomeThread - Attempted handshake 30 times. Will try to re-subscribe to handshake reply
03-19-2017 16:28:45.325 +0000 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected

[gcusello]

Hi cboard,
did you test connection between servers? (from server 2 telnet server1_IP 9997)
did you disabled iptables on both the servers for ports 9997 and 8089?
Bye.
Giuseppe

[woodcock · Accepted solution]

You need to remove this part from outputs.conf:

[monitor:/var/log/httpd/access_log]
server = 192.168.172.212:9997
disabled = 0

[monitor:/var/log/httpd/error_log]
server = 192.168.172.212:9997
disabled = 0

Then add this to inputs.conf:

[monitor:/var/log/httpd/access_log]
disabled = 0

[monitor:/var/log/httpd/error_log]
disabled = 0

As far as the DS logs, you have not said that you setup a Deployment Server so you can ignore those.

[cboard]

Thanks I've updated both config files on the forwarder and restarted but the web interface is still showing there's no forwarder.

I did set the deploy-server using the following command sudo -u splunk ./splunk set deploy-poll 192.168.172.212:9997 which was pointing at the indexer, but from your answer, i've disabled it and restarted the forwarder and the indexer.

I am no longer getting the errors in the log file apart from 03-19-2017 19:54:52.052 +0000 WARN UserManagerPro - Can't find [distributedSearch] stanza in distsearch.conf, using default authtoken HTTP timeouts

[woodcock]

You used the wrong port; do this from the forwarder:

 sudo -u splunk ./splunk set deploy-poll 192.168.172.212:8089

Is are you doing an all-in-one (Splunk Indexer=Search Head=Deployment Server)? This is fine for PoC but really not a great idea for any kind of production situation.

[cboard]

Ah that's done it, I've managed to add a file/directory monitor in the web GUI now pointing it at the forwarder. Does it take some time before it arrives as I've when going into the search page and viewing the data summary, it only shows 1 host, the host of the indexer not the host for the forwarder so I'm not seeing the logs for my second server.

[woodcock]

Try these searches for time All time.

See if there are any errors:

index=_* YOUR_FILE_NAME_HERE

Look for your data:

 |tstats count where index=* BY sourcetype

[cboard]

Thanks, it looks like something else might be broken as I don't seem to be getting any logs at all now, last entry is the 19th, but I know the logs have been updated so I'm guessing I might have broken something trying to get the forwarder working

[cboard]

I've just found in the Monitoring console on the indexer under Forwarder > Forwarder Instances it is now listing my forwarder as a universal forwarder type. So if the forwarder is working why can't I add data sources from the forwarder as the page for adding sources from a forwarder is still saying there are no forwarders.

[woodcock]

Because the your forwarder is not a Deployment Client because you used the wrong port. If you fix that (see my previous comment), then you should be able to use Settings -> Add data -> Forward.