Splunk Search

Realtime Search backfilling and slowdown

jmheaton
Path Finder

So i am trying to find the bottleneck in our hardware layout as i am running into a lot of slowdown in realtime searches. They can sometime backfill for 2-3 minutes as i dont think my indexers can keep up with the data and search usage. My hardware layout is as follows:

5 dedicated search heads with 8 cores and 12 gig of ram
8 dedicated indexers with 32 cores and 16 gb of memory

150-200 GB of data usage per day

20-25 realtime (5 minute) searches running 24/7
an additional 5-10 users searching data 24/7 as well

From what i have poked around on here and found. It looks like my indexers cant keep up with the IO's of all the realtime searches and logging the information at the same time.

Any idea's?

0 Karma
1 Solution

joebensimo
Path Finder

I do the following to make realtime searches perform. This does result in a bit of a delay -- which is ok for most the my needs.

  1. Rather than realtime, create scheduled searches that run once per minute and aggregate the data of a single minute (usually -2m@m to -1m@m, in order to allow sufficient indexing time -- you may need more or less depending on indexing latency) using stats or sistats, and save the results of these scheduled searches into a summary index.
  2. Create a saved search that references the summary indexes rather than the raw logs.
  3. Run the saved searches as realtime searches.

View solution in original post

0 Karma

LukeMurphey
Champion

Real-time searches cause an delay in the indexers because of the way the fit into the indexers queues. Basically, real-time searches make the events available for searching before they hit the disk (the indexing queues).

Splunk 6.0 introduces "indexed real-time" which functions mostly the same but is dramatically easier on your indexers. I have been using it and I prefer it.

0 Karma

joebensimo
Path Finder

I do the following to make realtime searches perform. This does result in a bit of a delay -- which is ok for most the my needs.

  1. Rather than realtime, create scheduled searches that run once per minute and aggregate the data of a single minute (usually -2m@m to -1m@m, in order to allow sufficient indexing time -- you may need more or less depending on indexing latency) using stats or sistats, and save the results of these scheduled searches into a summary index.
  2. Create a saved search that references the summary indexes rather than the raw logs.
  3. Run the saved searches as realtime searches.
0 Karma

jmheaton
Path Finder

Giving it a shot, we'll see how it goes.
Also going to run some at -6m ending -1m.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...