topic Re: How do I created a scheduled report using the collect command without populating it with partial data in Knowledge Management

How do I created a scheduled report using the collect command without populating it with partial data

michaeltokar — Sat, 04 Mar 2017 22:08:03 GMT

I'm just starting to get into summary indexes and changing over some reports that were previously long-running to use a summary index to speed them up. I'm aware that I need to have two parts to producing and consuming the summary data:

A search that runs regularly on fixed intervals (in my case, a 24 hour period) that populates the summary index;
A search that runs as often as I want that consumes the produced summary data.

I have devised the search that I want to use to populate my summary index. I've tested this with the collect command to make sure the data comes in correctly, using different marker values for test data. Now I want to schedule this search to run every day, to populate data from the previous 24 hour day.

But the way Splunk Reports work, I can't seem to define the Report without executing the report - the Save button is not available until you've executed the search (or if you have an existing Report, it isn't available until you change the definition of the original report and then execute it). And from basic testing, it looks like the collect command will partially populate results even if you terminate the job half-way through.

So what is the way to solve this? Ideally, something like the Dashboard's source editor for Reports would be available, where I can alter the Report's commands without actually executing them.

For reference, my search command to populate the summary index looks something like this:

"prd-safe" env="prd-safe" dbCapable query eventtype="unicorn_jira" 
| ... | stats latest(_time) AS _time, latest(mMode) AS mMode, count AS totalRequests, sum(totalSearches) AS totalSearches, sum(numCapable) AS totalCapable, sum(numEquivTrue) AS totalEquivTrue, median(totalDbDelta) AS p50Delta, perc99(totalDbDelta) AS p99Delta, max(totalDbDelta) AS p100Delta by hostname 
| ... | collect index=summary_team marker="report=jvs_migration_daily"

And ideally the Report would be scheduled to run Daily at 10am, for the period of the previous day i.e. -1d@d to -0d@d

Re: How do I created a scheduled report using the collect command without populating it with partial data

niketn — Sun, 05 Mar 2017 09:06:41 GMT

While saving the search as Scheduled Search take out the collect piece i.e. | collect index=summary_team marker="report=jvs_migration_daily"

Then, to edit the Saved Search without running it, you can open the same from Settings --> Searches, reports, and alerts (this would be similar to editing dashboard source code as you need). In the edit mode add the collect command in the end.

http://<Your_Splunk_URL>/manager/launcher/saved/searches

[Update as per new question]

No you need not enable summary indexing for that scheduled Search, collect command itself is doing this for you. If you do not want to use collect and want Splunk to take care of Summaries, then you choose the other Summary Indexing option.

Collect will give more control to your summary in terms of defining your own index (Summaries enabled through acceleration go to index="summary") and also dummy sourcetype field like orig_sourcetype=<My_Original_Sourcetype> etc (Sourcetype for summarized events is by default always stash).

In the search and alerts menu, you just need to define Schedule for your Scheduled Report.

Re: How do I created a scheduled report using the collect command without populating it with partial data

michaeltokar — Sun, 05 Mar 2017 22:12:51 GMT

Thank you! I didn't know about this alternative editor for Reports (Saved Searches).

So if I add the | collect ... command to the end of my search, do I also need to enable Summary Indexing at the bottom of the form? Or is it one or the other (i.e. they both do the same thing)?

screenshot of form: https://drive.google.com/file/d/0B79DKuM-KaKKS3ZKRlRTZDdfemc/view?usp=sharing