Multitple Splunk instances needed?

[bckq]

I'm using Splunk in work for monitoring servers and services and I get information from users that it is running quite slow. I'm wondering if it need some upgrade, for example by adding another server with Splunk. Currently I'm using Dell PowerEdge R710 with Intel Xeon L5609 and 24GB RAM.

Daily volume is: ~800MB
Daily indexes number is: ~1,400,000 events
Below I attach charts presented Splunk and server load:

Chart 1

Chart 2

Do you think that second instance of Splunk is needed?

[Drainy · Accepted solution]

So according to your second chart you are only using up to 5 cores which is alright. A Splunk search will lock a core entirely for the duration of that search. This sounds dramatic but searches are fast and then there is the scheduler on the system that also helps, but regardless its always a good indicator of a system where you need to start distributing.

At 800MB a day I would be surprised if you have any bottlenecks but one thing to do would be to install the Splunk on Splunk app (SoS. I can't remember if those screenshots are from that app or not 🙂 ).
This will give you some more detail. Also do a search on index=_internal for the word blocked. If any queues are blocking then it could be a sign of poor IO, which as Takajian says above, is important.

[Takajian]

Daily volume less than 800MB is supposed to be small enviornment. In genral, splunk need high disk io to get search result faster. As for Xeon L5609, clock size 1.86 GHz is not fast. Are you using recommened server suggested by splunk? Please see as bellow.

http://docs.splunk.com/Documentation/Splunk/latest/installation/capacityplanningforalargersplunkdepl...

In typical, you do not need second splunk instance when daily volume is less than 100GB. In your case, you may need to upgrade your hardware spec CPU and DISK in order to make search faster. Please verify your environmnet.

[Drainy]

Yes, on the flip side, everything up until the 100GB part I agree with wholeheartedly and should definitely be followed 🙂

[Takajian]

My suggestion is certainly rough, but it could be kind of guideline. I hope this will help something.

[Drainy]

Actually this isn't quite true, Yes the standard spec is based on a 100gb/day indexer however if you are indexing 100gb/day I would rather have 2 or 3 indexers running and distribute the search. This will result in much better performance and fewer bottlenecks. Infact I have one customer with 20gb of data running 3 indexes to allow for future capacity and to give greater performance now.