Chart number of results returned from scheduled searches

[IngloriousSplun]

I have a scheduled search that runs each minute, and the basic premise is that the resulting events get passed to a Python script for external action by another system. I need to create a timechart or other graph that trends the number of events returned by this scheduled search over time. I don't see a way to identify how many results were returned during each run, and also ensuring that data remains available indefinitely. Perhaps I'm missing it?

[somesoni2 · Accepted solution]

Try something like this

index=_internal sourcetype=scheduler savedsearch_name="ProvideYourSearchNameHere" | eval _time=scheduled_time | table _time savedsearch_name result_count

This should give you the scheduled time of the search, name, and the result returned by the execution.

Please note that the internal index has retention period of 30 days, so if you need to have trends for more that 30 days period, I would suggest to use a summary index search to saved this data to a summary index with desired retention period.

[IngloriousSplun]

Thanks for the tip about the retention period. For the summary index, that would just be by enabling summary indexing for that saved search correct (through the GUI)?

[richgalloway]

Have a look at the scheduler.log file.

index=_internal source="/opt/splunk/var/log/splunk/scheduler.log" app="foo" | timechart avg(result_count) by savedsearch_name