Deployment Architecture

Multitple Splunk instances needed?

bckq
Path Finder

I'm using Splunk in work for monitoring servers and services and I get information from users that it is running quite slow. I'm wondering if it need some upgrade, for example by adding another server with Splunk. Currently I'm using Dell PowerEdge R710 with Intel Xeon L5609 and 24GB RAM.

Daily volume is: ~800MB
Daily indexes number is: ~1,400,000 events
Below I attach charts presented Splunk and server load:

Chart 1

Chart 2

Do you think that second instance of Splunk is needed?

1 Solution

Drainy
Champion

So according to your second chart you are only using up to 5 cores which is alright. A Splunk search will lock a core entirely for the duration of that search. This sounds dramatic but searches are fast and then there is the scheduler on the system that also helps, but regardless its always a good indicator of a system where you need to start distributing.

At 800MB a day I would be surprised if you have any bottlenecks but one thing to do would be to install the Splunk on Splunk app (SoS. I can't remember if those screenshots are from that app or not 🙂 ).
This will give you some more detail. Also do a search on index=_internal for the word blocked. If any queues are blocking then it could be a sign of poor IO, which as Takajian says above, is important.

View solution in original post

0 Karma

Drainy
Champion

So according to your second chart you are only using up to 5 cores which is alright. A Splunk search will lock a core entirely for the duration of that search. This sounds dramatic but searches are fast and then there is the scheduler on the system that also helps, but regardless its always a good indicator of a system where you need to start distributing.

At 800MB a day I would be surprised if you have any bottlenecks but one thing to do would be to install the Splunk on Splunk app (SoS. I can't remember if those screenshots are from that app or not 🙂 ).
This will give you some more detail. Also do a search on index=_internal for the word blocked. If any queues are blocking then it could be a sign of poor IO, which as Takajian says above, is important.

0 Karma

Takajian
Builder

Daily volume less than 800MB is supposed to be small enviornment. In genral, splunk need high disk io to get search result faster. As for Xeon L5609, clock size 1.86 GHz is not fast. Are you using recommened server suggested by splunk? Please see as bellow.

http://docs.splunk.com/Documentation/Splunk/latest/installation/capacityplanningforalargersplunkdepl...

In typical, you do not need second splunk instance when daily volume is less than 100GB. In your case, you may need to upgrade your hardware spec CPU and DISK in order to make search faster. Please verify your environmnet.

0 Karma

Drainy
Champion

Yes, on the flip side, everything up until the 100GB part I agree with wholeheartedly and should definitely be followed 🙂

0 Karma

Takajian
Builder

My suggestion is certainly rough, but it could be kind of guideline. I hope this will help something.

0 Karma

Drainy
Champion

Actually this isn't quite true, Yes the standard spec is based on a 100gb/day indexer however if you are indexing 100gb/day I would rather have 2 or 3 indexers running and distribute the search. This will result in much better performance and fewer bottlenecks. Infact I have one customer with 20gb of data running 3 indexes to allow for future capacity and to give greater performance now.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...