topic Re: Index from old Splunk Heavy Forwarder in Getting Data In

Index from old Splunk Heavy Forwarder

emccaslin — Mon, 23 Dec 2013 22:02:36 GMT

Setup currently I have the newest version of Splunk (6.0) running as my main Splunk server with several universal forwarders v 6.0 sending logs to the server to be indexed.

I have another box that the v 6.0 forwarders are incompatible with so I need to install Splunk version 3.14 onto the box. I see in the documentation that I can make the full installation a heavy forwarder to push to my regular indexer, but it is not working for me.

Steps Taken:

I Installed the full Splunk v 3.14 on the box I want to use a forwarder
Then enabled the forwarder: ./splunk enable app SplunkForwarder -auth <username>:<password>
Started forwarding activity: ./splunk add forward-server <host>:<port> -auth <username>:<password>
Added deploy server: ./splunk set deploy-poll <host>:<port>
Retarted splunk: ./splunk restart
Waited but the forwarder never appears in the list under Forwarder Management on the Splunk Server

I assume this has something to do with the different versions of Splunk that I am using, but the documentation says:

"All indexers are backwards compatible
with any forwarder and can receive
data from any earlier version
forwarder."

Anyone else have this problem or know how to better implement this?

Documentation:

Re: Index from old Splunk Heavy Forwarder

sciurus — Thu, 26 Dec 2013 09:54:38 GMT

Start with the assumption that it's compatible, and something else is broken. Check basic TCP - can you see the connection in netstat? Is it successfully connecting? If so, check splunkd.log, if not, check routes and firewalls, etc.

If it ISN'T compatible, then you've got something which is being rejected by the v6 server - in which case it will show in logs somewhere. If it IS compatible but it's being rejected due to a configuration issue, that will also show up, etc. Also deploy-poll is different to forwarding, so troubleshoot that separately.

Re: Index from old Splunk Heavy Forwarder

emccaslin — Thu, 26 Dec 2013 20:51:00 GMT

Great suggestions for me to start looking for a solution.

Re: Index from old Splunk Heavy Forwarder

emccaslin — Thu, 26 Dec 2013 20:51:05 GMT

On the v6 Server in splunkd.log I am getting the following about the v3 forwarder: "DEBUG RPCDispatcher - Request from 3.x deployment client : <ip address> received. <some html code>"

I believe the forwarder is connecting to the server. I'm not seeing anything in logs on the sever that indicates incompatibility, but on the forwarder I see a message along the lines of "possible server compatibility issue". I have tried getting the forwarder to monitor a log by placing the configuration in ./etc/system/local instead of having it pull the config from the server but this is still not working.

Re: Index from old Splunk Heavy Forwarder

Lowell — Thu, 26 Dec 2013 21:53:26 GMT

Any chance that this is your issue?

http://answers.splunk.com/answers/115495/i-upgraded-my-distributed-environment-to-splunk-60-and-now-my-indexers-are-crashing

Basically, try negotiateNewProtocol = false

Re: Index from old Splunk Heavy Forwarder

sciurus — Fri, 27 Dec 2013 09:59:25 GMT

If it's receiving back HTML, are you sure you're pointing it to the Splunk log port (default 9997), not the management (default 8089) or user interface (default 8000)? I'm not sure why you'd get HTML back from the log port.

Re: Index from old Splunk Heavy Forwarder

emccaslin — Fri, 04 Apr 2014 16:46:51 GMT

So I eventually got this working and now I am able to get it working on multiple Windows 2000 servers. One of the main differences I noticed it working is when I enabled the SplunkLightForwarder instead of SplunkForwarder.

Also, because a compatibility issue, Splunk cannot send the configurations through a deployment app as the Universal Forwarders do. So I have to manually put the configurations in $SPLUNK_HOME/etc/system/local and restart the forwarder. Seems to be working well now.