topic Splunk Performance in Monitoring Splunk

Splunk Performance

lspringer — Thu, 27 Sep 2012 17:54:17 GMT

We started our setup with a standalone Splunk server. Now that we have a second standalone Splunk server next to it, we'd like to share the load across both machines. We'd prefer to do this in a way where we don't have to share an index between the machines, for the least amount of disruption to users.

What would be the best way to go about this?

Re: Splunk Performance

chris — Thu, 27 Sep 2012 19:52:27 GMT

I'm not sure I quite understood what you mean by "not sharing an index between the machines". But I'll have a go at it:

Splunk will perform best if you scale horizontally, if you have a lot of dedicated indexers, they can all do the work at the same time. So for your scenario you could use one server as a combined indexer and search head and the other machine as a dedicated indexer. Both servers should have the same indexes configured (they will store the data received locally and do not have a shared filesystem or anything like that). Then you set up the forwarders to auto load balance their data to both servers (There is an example here: http://docs.splunk.com/Documentation/Splunk/latest/Admin/Outputsconf)

If for some reason you do not want to have half the data of a specific index on one server and the other half on the other, you could use both servers as indexers & search heads and the split up the data you index among them ( You could habe application1 and application2 on server1 and application3 and application4 on server2. If there is a team looking after application1 &2 and another one looking after application3 & 4 that might make sense). But to me the first setup makes more sense.

If you can explain a little what you mean by the least amount of disruption to users, we might be able to give you better assistance

Re: Splunk Performance

lspringer — Thu, 27 Sep 2012 20:02:26 GMT

Regarding "not sharing an index between the machines", I read somewhere that I should set up a share and have the indexers access the shared location.

Your solution is what I'm looking for I just want the solution to be a config change as opposed to standing up a share which would take more time and require more configuration.

For clarification the specific example you are referring to is:

[tcpout]
heartbeatFrequency=15
indexAndForward=true

[tcpout:indexer1]
server=Y.Y.Y.Y:9997

[tcpout:indexer2]
server=X.X.X.X:6666