topic Re: How do I set up indexing across 2 servers in Getting Data In

How do I set up indexing across 2 servers

jjlambre — Tue, 03 Feb 2015 14:10:22 GMT

New to splunk.....I have two servers in a distributed deployment. S1 has Indexer, License Master, and Search Head roles. S2 is just an indexer. Can I have a splunk server (S1) index and forward data? When I attempt to set up forwarding to both servers, using Splunk load balancing, I receive errors:

0800 ERROR TcpOutputFd - Connection to host=xxx.xx.xx.xx:xxxx failed

As soon as I remove S1 from splunk lb (forwarding), S2 does all of the indexing. I'd like to have S1 index data as well. Is this possible and how can this be accomplished?

Re: How do I set up indexing across 2 servers

martin_mueller — Thu, 05 Feb 2015 00:43:59 GMT

Are you trying to accomplish load balancing between the two indexers or HA/failover with duplicated data on both?

Re: How do I set up indexing across 2 servers

aalanisr26 — Thu, 05 Feb 2015 01:19:21 GMT

Check if S1 has port 9997 (default) enabled:
go to settings/data/forwarding and receiving/ configure receiving

you don't need to forward data from S1 to S2 because you will end up with duplicated data(considering you are not setting up a cluster)

Re: How do I set up indexing across 2 servers

jjlambre — Thu, 05 Feb 2015 02:32:07 GMT

load balancing between the two indexers

Re: How do I set up indexing across 2 servers

jjlambre — Thu, 05 Feb 2015 02:40:27 GMT

Thanks for your reply! Port 9997 is enabled on both S1 and S2. If I don't need to forward data on S1, how does splunk know how to forward data to itself since S1 is also a forwarder? The only way I could get indexing to work properly was to remove S1 from Forwarding, but S1 is not indexing data.

Re: How do I set up indexing across 2 servers

martin_mueller — Thu, 05 Feb 2015 02:55:59 GMT

Make sure your forwarders know both indexers, in their outputs.conf there should be a line like this:

[tcpout...]
server = S1:9997, S2:9997

Without setting anything else that'll default to load balancing between the two indexers.

Data generated on S1 will be indexed by S1 locally, there is no partial forwarding to the other indexer. If you have significant data being generated on S1 (DB Connect, network inputs, etc.) consider moving them to a heavy forwarder (DB Connect) or other dedicated system that then gets a universal forwarder (network inputs sending to syslog-ng, http://www.georgestarcher.com/splunk-success-with-syslog/)

Re: How do I set up indexing across 2 servers

Sourabhv05 — Thu, 05 Feb 2015 13:09:37 GMT

In Forwarder, in outputs.conf file

forward data to all your peers. mention IP's of all peers.
autoLBFrequency=40
server = Peer1:9997, peer2:9997
useACK = true

Re: How do I set up indexing across 2 servers

jjlambre — Thu, 05 Feb 2015 20:40:34 GMT

Thanks! Attempted this change and now i receive this error message:
Forwarding to indexer group default-autolb-group blocked for 5000 seconds.

Re: How do I set up indexing across 2 servers

Sourabhv05 — Fri, 06 Feb 2015 05:19:14 GMT

On the indexer(s), you must configure a receiving port in inputs.conf.

Or from GUI, go to settings> Forward and receive > in receive add port 9997 at each indexer.