New to splunk.....I have two servers in a distributed deployment. S1 has Indexer, License Master, and Search Head roles. S2 is just an indexer. Can I have a splunk server (S1) index and forward data? When I attempt to set up forwarding to both servers, using Splunk load balancing, I receive errors:
0800 ERROR TcpOutputFd - Connection to host=xxx.xx.xx.xx:xxxx failed
As soon as I remove S1 from splunk lb (forwarding), S2 does all of the indexing. I'd like to have S1 index data as well. Is this possible and how can this be accomplished?
Make sure your forwarders know both indexers, in their outputs.conf there should be a line like this:
[tcpout...] server = S1:9997, S2:9997
Without setting anything else that'll default to load balancing between the two indexers.
Data generated on S1 will be indexed by S1 locally, there is no partial forwarding to the other indexer. If you have significant data being generated on S1 (DB Connect, network inputs, etc.) consider moving them to a heavy forwarder (DB Connect) or other dedicated system that then gets a universal forwarder (network inputs sending to syslog-ng, http://www.georgestarcher.com/splunk-success-with-syslog/)
Check if S1 has port 9997 (default) enabled:
go to settings/data/forwarding and receiving/ configure receiving
you don't need to forward data from S1 to S2 because you will end up with duplicated data(considering you are not setting up a cluster)
Thanks for your reply! Port 9997 is enabled on both S1 and S2. If I don't need to forward data on S1, how does splunk know how to forward data to itself since S1 is also a forwarder? The only way I could get indexing to work properly was to remove S1 from Forwarding, but S1 is not indexing data.