So i am trying to find the bottleneck in our hardware layout as i am running into a lot of slowdown in realtime searches. They can sometime backfill for 2-3 minutes as i dont think my indexers can keep up with the data and search usage. My hardware layout is as follows:
5 dedicated search heads with 8 cores and 12 gig of ram
8 dedicated indexers with 32 cores and 16 gb of memory
150-200 GB of data usage per day
20-25 realtime (5 minute) searches running 24/7
an additional 5-10 users searching data 24/7 as well
From what i have poked around on here and found. It looks like my indexers cant keep up with the IO's of all the realtime searches and logging the information at the same time.
Any idea's?
I do the following to make realtime searches perform. This does result in a bit of a delay -- which is ok for most the my needs.
Real-time searches cause an delay in the indexers because of the way the fit into the indexers queues. Basically, real-time searches make the events available for searching before they hit the disk (the indexing queues).
Splunk 6.0 introduces "indexed real-time" which functions mostly the same but is dramatically easier on your indexers. I have been using it and I prefer it.
I do the following to make realtime searches perform. This does result in a bit of a delay -- which is ok for most the my needs.
Giving it a shot, we'll see how it goes.
Also going to run some at -6m ending -1m.