How to run repeated searches by time period in a loop?

[rgustin]

Normally, we populate search results in real time into the summary index. But a research question came up about historical data that resides in the main index. Trying to run a search over a period of one month results in "Timed out waiting for peer xxx.xxx.com. If this occurs frequently, receiveTimeout in distsearch.conf may need to be increased. Search results might be incomplete!" The same search runs fine in 24 hours.

So the thought I had would be to run a search by calendar day or less, put the result in the summary index, go back one day, repeat for 90 past days.

Once the single count by day or less is in the summary index, then I can easily graph it.

The is called running repeated searches in a loop over different time periods: looping repeated searches by time.

I may have to do it by hour, if I get timeouts. I don't have access to change the search timeouts.

[marcoscala]

yYu can design a saved search that extract the data you need from the main index and add them to the summary index for a fixed time-frame (1hour for instance) and set it to be scheduled each hour.

Then you have the "fill_summary_index.py" python script that allows you to run back in time the search for the desired period of time.

For details, see http://docs.splunk.com/Documentation/Splunk/6.0.3/Knowledge/Managesummaryindexgapsandoverlaps#Use_th...

Marco

[marcoscala]

It this answers your question, don't forget to mark it as valid here on answers.com!

And happy Splunking!

Marco

[rgustin]

Thank you!