Splunk Search

Realtime Search backfilling and slowdown

jmheaton
Path Finder

So i am trying to find the bottleneck in our hardware layout as i am running into a lot of slowdown in realtime searches. They can sometime backfill for 2-3 minutes as i dont think my indexers can keep up with the data and search usage. My hardware layout is as follows:

5 dedicated search heads with 8 cores and 12 gig of ram
8 dedicated indexers with 32 cores and 16 gb of memory

150-200 GB of data usage per day

20-25 realtime (5 minute) searches running 24/7
an additional 5-10 users searching data 24/7 as well

From what i have poked around on here and found. It looks like my indexers cant keep up with the IO's of all the realtime searches and logging the information at the same time.

Any idea's?

0 Karma
1 Solution

joebensimo
Path Finder

I do the following to make realtime searches perform. This does result in a bit of a delay -- which is ok for most the my needs.

  1. Rather than realtime, create scheduled searches that run once per minute and aggregate the data of a single minute (usually -2m@m to -1m@m, in order to allow sufficient indexing time -- you may need more or less depending on indexing latency) using stats or sistats, and save the results of these scheduled searches into a summary index.
  2. Create a saved search that references the summary indexes rather than the raw logs.
  3. Run the saved searches as realtime searches.

View solution in original post

0 Karma

LukeMurphey
Champion

Real-time searches cause an delay in the indexers because of the way the fit into the indexers queues. Basically, real-time searches make the events available for searching before they hit the disk (the indexing queues).

Splunk 6.0 introduces "indexed real-time" which functions mostly the same but is dramatically easier on your indexers. I have been using it and I prefer it.

0 Karma

joebensimo
Path Finder

I do the following to make realtime searches perform. This does result in a bit of a delay -- which is ok for most the my needs.

  1. Rather than realtime, create scheduled searches that run once per minute and aggregate the data of a single minute (usually -2m@m to -1m@m, in order to allow sufficient indexing time -- you may need more or less depending on indexing latency) using stats or sistats, and save the results of these scheduled searches into a summary index.
  2. Create a saved search that references the summary indexes rather than the raw logs.
  3. Run the saved searches as realtime searches.

View solution in original post

0 Karma

jmheaton
Path Finder

Giving it a shot, we'll see how it goes.
Also going to run some at -6m ending -1m.

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!