Increasing Splunks Search performance on Linux

[dragmore]

Hi. We have several big Splunk installations and im working on trying to increase the search performance on them. Unfortunatly ive come to and end and i could really use some input/suggestions on where to fix this.

Info:
1. Splunk 4.3.2 x64 REDHAT @ RHEL 5.7 X64
2. HOT/WARM IDX @ 2x120GB SSD in RAID1 mounted volume
3. COLD & Thawed @a 14x300GB RAID6-ADG mounted volume
4. 2x6CPU Cores and 48GB MEM (HP DL380g7)

So when i do a search i often see almost all my cpu's ad idle, but the one im using for search..
I got no IO-Waiting on my Disk-IO subsystem so i know this issue is CPU bound.

So the BIG question is : Is there a way to enable a search to span over multiple cpu cores? Multithreaded/processed searches?

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 196 812064 1117692 32270052 0 0 6 50 2 2 6 0 94 0 0
2 0 196 809468 1117708 32272140 0 0 0 257 1280 1743 9 0 90 0 0
7 0 196 660532 1117800 32270748 0 0 62 1726 1602 3894 25 2 72 0 0
7 0 196 556972 1117920 32274096 0 0 1 1690 1648 21236 50 1 48 0 0
3 0 196 687980 1117952 32258168 0 0 0 428 1424 10324 40 1 59 0 0

br TE

[twkan]

Personally, I would install multiple Splunk Indexers listening on different ports with the aim of saturating the CPU cores as well as Disk I/O. Given that you have 12 CPU cores, I would start with perhaps 2 to 3 Splunk instances, and monitor the health status via iostat, top etc. to make sure that I am not overloading the box, and subsequently validate the improved utilisation of the hardware resources.

[MuS]

okay at first I disagreed on this but after reading http://splunk-base.splunk.com/answers/5202/how-do-i-get-the-most-out-of-a-16-core-server I think you can improve search performance this way.

[twkan]

Generally speaking, search performance will increase along with indexing performance. This is where the multiple indexers with MapReduce will come into play to increase the search performance.

[MuS]

Hi twkan, then you would 'only' increase the index performance but not the search performance.

[MuS]

Hi dragmore

please read this answer to find out more about search performance.

regards