I have a scheduled search that runs each minute, and the basic premise is that the resulting events get passed to a Python script for external action by another system. I need to create a timechart or other graph that trends the number of events returned by this scheduled search over time. I don't see a way to identify how many results were returned during each run, and also ensuring that data remains available indefinitely. Perhaps I'm missing it?
Have a look at the scheduler.log file.
index=_internal source="/opt/splunk/var/log/splunk/scheduler.log" app="foo" | timechart avg(result_count) by savedsearch_name
Try something like this
index=_internal sourcetype=scheduler savedsearch_name="ProvideYourSearchNameHere" | eval _time=scheduled_time | table _time savedsearch_name result_count
This should give you the scheduled time of the search, name, and the result returned by the execution.
Please note that the internal index has retention period of 30 days, so if you need to have trends for more that 30 days period, I would suggest to use a summary index search to saved this data to a summary index with desired retention period.
Thanks for the tip about the retention period. For the summary index, that would just be by enabling summary indexing for that saved search correct (through the GUI)?