Index from old Splunk Heavy Forwarder

[emccaslin]

Setup currently I have the newest version of Splunk (6.0) running as my main Splunk server with several universal forwarders v 6.0 sending logs to the server to be indexed.

I have another box that the v 6.0 forwarders are incompatible with so I need to install Splunk version 3.14 onto the box. I see in the documentation that I can make the full installation a heavy forwarder to push to my regular indexer, but it is not working for me.

Steps Taken:

1. I Installed the full Splunk v 3.14 on the box I want to use a forwarder
2. Then enabled the forwarder: ./splunk enable app SplunkForwarder -auth <username>:<password>
3. Started forwarding activity: ./splunk add forward-server <host>:<port> -auth <username>:<password>
4. Added deploy server: ./splunk set deploy-poll <host>:<port>
5. Retarted splunk: ./splunk restart
6. Waited but the forwarder never appears in the list under Forwarder Management on the Splunk Server

I assume this has something to do with the different versions of Splunk that I am using, but the documentation says:

"All indexers are backwards compatible
with any forwarder and can receive
data from any earlier version
forwarder."

Anyone else have this problem or know how to better implement this?

Documentation:

• Heavy Forwarder Doc
• Forwarder and Indexer Compatibility

[Lowell]

Any chance that this is your issue?

http://answers.splunk.com/answers/115495/i-upgraded-my-distributed-environment-to-splunk-60-and-now-...

Basically, try negotiateNewProtocol = false

[sciurus · Accepted solution]

Start with the assumption that it's compatible, and something else is broken. Check basic TCP - can you see the connection in netstat? Is it successfully connecting? If so, check splunkd.log, if not, check routes and firewalls, etc.

If it ISN'T compatible, then you've got something which is being rejected by the v6 server - in which case it will show in logs somewhere. If it IS compatible but it's being rejected due to a configuration issue, that will also show up, etc. Also deploy-poll is different to forwarding, so troubleshoot that separately.

[emccaslin]

So I eventually got this working and now I am able to get it working on multiple Windows 2000 servers. One of the main differences I noticed it working is when I enabled the SplunkLightForwarder instead of SplunkForwarder.

Also, because a compatibility issue, Splunk cannot send the configurations through a deployment app as the Universal Forwarders do. So I have to manually put the configurations in $SPLUNK_HOME/etc/system/local and restart the forwarder. Seems to be working well now.

[sciurus]

If it's receiving back HTML, are you sure you're pointing it to the Splunk log port (default 9997), not the management (default 8089) or user interface (default 8000)? I'm not sure why you'd get HTML back from the log port.

[emccaslin]

On the v6 Server in splunkd.log I am getting the following about the v3 forwarder: "DEBUG RPCDispatcher - Request from 3.x deployment client : <ip address> received. <some html code>"

I believe the forwarder is connecting to the server. I'm not seeing anything in logs on the sever that indicates incompatibility, but on the forwarder I see a message along the lines of "possible server compatibility issue". I have tried getting the forwarder to monitor a log by placing the configuration in ./etc/system/local instead of having it pull the config from the server but this is still not working.

[emccaslin]

Great suggestions for me to start looking for a solution.