topic Re: Distribution of Load and Splunk Performance in Splunk Search

Distribution of Load and Splunk Performance

nikhilmehra79 — Mon, 07 Apr 2014 17:55:26 GMT

As a quick check can some one suggest me if we have a 2 indexer envirornment with 2 search heads - does it make sense for all universal forwarders to send events to both indexers?

Question i am asking is i am looking to improve performance at my dashboards run from search head and if we are sending same data to both indexers how does it help performance? ( i know it helps from load balancing and DR prospective but not how it helps in seek time/performance)?

Anyone?

Re: Distribution of Load and Splunk Performance

martin_mueller — Mon, 07 Apr 2014 19:39:33 GMT

Usually you'll want to have each UF send to both indexers in a load-balanced fashion, so they switch between one and the other. That way almost every search can pull data from both indexers and you get decent performance.
I don't see how cloning the data to both indexers would help performance.

Re: Distribution of Load and Splunk Performance

nikhilmehra79 — Mon, 07 Apr 2014 20:36:29 GMT

when you say load balnced way you mean output.conf has ?

[tcpout]
defaultGroup = productionSplunk1, productionSplunk2

[tcpout:productionSplunk1]
server = X.X.X.X:9997

[tcpout:productionSplunk2]
server = Y.Y.Y.Y:9997

Re: Distribution of Load and Splunk Performance

linu1988 — Mon, 07 Apr 2014 20:44:53 GMT

that would send everything to both, use autolb=true or use a DNS load balancer

Re: Distribution of Load and Splunk Performance

martin_mueller — Mon, 07 Apr 2014 20:46:38 GMT

Defining two groups would indeed clone data to both indexers. To get load balancing, add two servers to one group like this:

[tcpout]
defaultGroup = productionSplunk

[tcpout:productionSplunk]
server = X.X.X.X:9997, Y.Y.Y.Y:9997

Re: Distribution of Load and Splunk Performance

nikhilmehra79 — Mon, 07 Apr 2014 21:18:20 GMT

Thanks so does either of the two configs will be ok?
Option 1
[tcpout]
defaultGroup = productionSplunk

[tcpout:productionSplunk]
server = X.X.X.X:9997, Y.Y.Y.Y:9997

Option 2
autolb=true
[tcpout] defaultGroup = productionSplunk1, productionSplunk2

[tcpout:productionSplunk1] server = X.X.X.X:9997

[tcpout:productionSplunk2] server = Y.Y.Y.Y:9997

Re: Distribution of Load and Splunk Performance

martin_mueller — Mon, 07 Apr 2014 21:21:08 GMT

They do entirely different things, so which one is okay for you depends on what you want.

Re: Distribution of Load and Splunk Performance

linu1988 — Mon, 07 Apr 2014 21:28:42 GMT

Martin

[tcpout:productionSplunk] server = X.X.X.X:9997, Y.Y.Y.Y:9997
will not this clone data as well?

i thought only the below would send load balanced

[tcpout:productionSplunk] server = X.X.X.X:9997, Y.Y.Y.Y:9997 autoLB=True

[tcpout:productionSplunk] server = some.splunk.com:9997

load balancer handles the traffic.

Re: Distribution of Load and Splunk Performance

nikhilmehra79 — Mon, 07 Apr 2014 21:30:59 GMT

Can you please explain or refer me to doc -

Isn't in both cases UF sent data to one indexer but other indexer recieve it from its first peer at port 8080?

Re: Distribution of Load and Splunk Performance

martin_mueller — Mon, 07 Apr 2014 21:33:47 GMT

@linu1988: That will load-balance rather than clone - note, autoLB=true is the default.

@nikhilmehra79: http://docs.splunk.com/Documentation/Splunk/6.0.2/Admin/outputsconf

If you're talking about one indexer receiving data from the other, that's clustering - are you using clustering?

Re: Distribution of Load and Splunk Performance

nikhilmehra79 — Mon, 07 Apr 2014 21:52:09 GMT

Ok i think i was not clear with my requirements:

I need to send data in Load Blanced method (round Robin) to one indexer at a time
But data should synch between two indexers (by peer clustering at port 8080 default). First - am i doing something crazy?, all i am thinking to use efficiently networks and also not give up on a DR.

Will following load balance and send one event to one indexer only on network, and since both indexers are cluster peers will ultimately both indexers have same set of events?
[tcpout:productionSplunk] server = X.X.X.X:9997, Y.Y.Y.Y:9997

Re: Distribution of Load and Splunk Performance

martin_mueller — Mon, 07 Apr 2014 21:55:10 GMT

If both indexers are peers in the same cluster and your replication factor is 2 then yes, both peers will store each event even if your forwarders only send it to one using load balancing.

Re: Distribution of Load and Splunk Performance

nikhilmehra79 — Mon, 07 Apr 2014 22:00:41 GMT

Yes both peers are in same cluster controlled by same master node and i have a replication factor 2. Thanks for explanations guys.