Splunk Performance

[lspringer]

We started our setup with a standalone Splunk server. Now that we have a second standalone Splunk server next to it, we'd like to share the load across both machines. We'd prefer to do this in a way where we don't have to share an index between the machines, for the least amount of disruption to users.

What would be the best way to go about this?

[chris · Accepted solution]

I'm not sure I quite understood what you mean by "not sharing an index between the machines". But I'll have a go at it:

Splunk will perform best if you scale horizontally, if you have a lot of dedicated indexers, they can all do the work at the same time. So for your scenario you could use one server as a combined indexer and search head and the other machine as a dedicated indexer. Both servers should have the same indexes configured (they will store the data received locally and do not have a shared filesystem or anything like that). Then you set up the forwarders to auto load balance their data to both servers (There is an example here: http://docs.splunk.com/Documentation/Splunk/latest/Admin/Outputsconf)

If for some reason you do not want to have half the data of a specific index on one server and the other half on the other, you could use both servers as indexers & search heads and the split up the data you index among them ( You could habe application1 and application2 on server1 and application3 and application4 on server2. If there is a team looking after application1 &2 and another one looking after application3 & 4 that might make sense). But to me the first setup makes more sense.

If you can explain a little what you mean by the least amount of disruption to users, we might be able to give you better assistance

[lspringer]

Regarding "not sharing an index between the machines", I read somewhere that I should set up a share and have the indexers access the shared location.

Your solution is what I'm looking for I just want the solution to be a config change as opposed to standing up a share which would take more time and require more configuration.

For clarification the specific example you are referring to is:

[tcpout]
heartbeatFrequency=15
indexAndForward=true

[tcpout:indexer1]
server=Y.Y.Y.Y:9997

[tcpout:indexer2]
server=X.X.X.X:6666