https://community.splunk.com/t5/Splunk-Search/Realtime-Search-backfilling-and-slowdown/m-p/170834#M48928 <P>Real-time searches cause an delay in the indexers because of the way the fit into the indexers queues. Basically, real-time searches make the events available for searching before they hit the disk (the indexing queues).</P> <P>Splunk 6.0 introduces "<A href="http://docs.splunk.com/Documentation/Splunk/latest/Search/Realtimeperformanceandlimitations#Indexed_real-time_searches">indexed real-time</A>" which functions mostly the same but is dramatically easier on your indexers. I have been using it and I prefer it.</P> Mon, 09 Dec 2013 04:33:17 GMT LukeMurphey 2013-12-09T04:33:17Z https://community.splunk.com/t5/Splunk-Search/Realtime-Search-backfilling-and-slowdown/m-p/170831#M48925 <P>So i am trying to find the bottleneck in our hardware layout as i am running into a lot of slowdown in realtime searches. They can sometime backfill for 2-3 minutes as i dont think my indexers can keep up with the data and search usage. My hardware layout is as follows:</P> <P>5 dedicated search heads with 8 cores and 12 gig of ram<BR /> 8 dedicated indexers with 32 cores and 16 gb of memory</P> <P>150-200 GB of data usage per day</P> <P>20-25 realtime (5 minute) searches running 24/7<BR /> an additional 5-10 users searching data 24/7 as well</P> <P>From what i have poked around on here and found. It looks like my indexers cant keep up with the IO's of all the realtime searches and logging the information at the same time.</P> <P>Any idea's?</P> Sun, 08 Dec 2013 19:48:07 GMT https://community.splunk.com/t5/Splunk-Search/Realtime-Search-backfilling-and-slowdown/m-p/170831#M48925 jmheaton 2013-12-08T19:48:07Z https://community.splunk.com/t5/Splunk-Search/Realtime-Search-backfilling-and-slowdown/m-p/170832#M48926 <P>I do the following to make realtime searches perform. This does result in a bit of a delay -- which is ok for most the my needs.</P> <OL> <LI>Rather than realtime, create scheduled searches that run once per minute and aggregate the data of a single minute (usually -2m@m to -1m@m, in order to allow sufficient indexing time -- you may need more or less depending on indexing latency) using stats or sistats, and save the results of these scheduled searches into a summary index.</LI> <LI>Create a saved search that references the summary indexes rather than the raw logs.</LI> <LI>Run the saved searches as realtime searches.</LI> </OL> Sun, 08 Dec 2013 21:46:27 GMT https://community.splunk.com/t5/Splunk-Search/Realtime-Search-backfilling-and-slowdown/m-p/170832#M48926 joebensimo 2013-12-08T21:46:27Z https://community.splunk.com/t5/Splunk-Search/Realtime-Search-backfilling-and-slowdown/m-p/170833#M48927 <P>Giving it a shot, we'll see how it goes.<BR /> Also going to run some at -6m ending -1m.</P> Sun, 08 Dec 2013 22:31:23 GMT https://community.splunk.com/t5/Splunk-Search/Realtime-Search-backfilling-and-slowdown/m-p/170833#M48927 jmheaton 2013-12-08T22:31:23Z https://community.splunk.com/t5/Splunk-Search/Realtime-Search-backfilling-and-slowdown/m-p/170834#M48928 <P>Real-time searches cause an delay in the indexers because of the way the fit into the indexers queues. Basically, real-time searches make the events available for searching before they hit the disk (the indexing queues).</P> <P>Splunk 6.0 introduces "<A href="http://docs.splunk.com/Documentation/Splunk/latest/Search/Realtimeperformanceandlimitations#Indexed_real-time_searches">indexed real-time</A>" which functions mostly the same but is dramatically easier on your indexers. I have been using it and I prefer it.</P> Mon, 09 Dec 2013 04:33:17 GMT https://community.splunk.com/t5/Splunk-Search/Realtime-Search-backfilling-and-slowdown/m-p/170834#M48928 LukeMurphey 2013-12-09T04:33:17Z