Summary Indexing for dashboard

[mwdbhyat]

Hi,

I want to use summary indexing for my dashboard and was wondering what the best approach would be? The following searches I am currently running are:

-Count of daily hosts
-Count of daily events
EG - | tstats count WHERE index=dig* OR index=mybbc* earliest=-30d@d latest=@d by _time span=1d | eval count=round(count/1000000)

-Daily license usage
EG - index=_internal source=license_usage.log type=Usage pool= idx=corp* earliest=-30d@d latest=@d | timechart span=1d sum(b) as Bytes |eval Bytes=(Bytes/1024/1024/1024)

-Monthly license usage

Could anyone advise on the best and most efficient approach to building my summary indexes in order to achieve the daily/monthly views for my dash? I was thinking of my initial scheduled search to run every 10min. Are there any commands I should remove from the initial search and only use when searching my summary index?

Thanks!

[skoelpin · Accepted solution]

Hello @mwdbhyat

This all depends on how much data is coming through and how long you're willing to wait to be able to search the past 30 days. You have a transform command in your second search so that would be eligible for accelerated searching. But to answer your question, you can set it up for 10 minutes if that's what your looking for.

The factors that will determine this are
1) The amount of saved searches you already have
2) The amount of CPU your indexer(s) have available
3) The need to have your summary index populated every 10 minutes or 1 hour

If you have enough CPU and you have a need to re-index this data into a SI every 10 minutes then yes

[mwdbhyat]

Thanks - I will look into the factors.. Would you say its better to use the new way of creating SI's? with the si or use the old way of ? .. Also I read this in the splunk manual "When you define summary-index-populating searches, do not pipe other search operators after the main summary indexing reporting command. In other words, don't include additional | eval commands and the like. Save the extra search operators for the searches you run against the summary indexes, not the search you use to populate it." ... Does this mean i should remove all my eval commands from the SI populating search and only use them in the search on the indexed data when I am populating my dashboard view?

[skoelpin]

You are correct, you're populating search should not have an eval command, but you can use the eval command after the data's been indexed in your Summary Index.

What version of Splunk are you running? Creating a SI is pretty straight forward

http://docs.splunk.com/Documentation/Splunk/6.4.2/Knowledge/Configuresummaryindexes

[mwdbhyat]

Cool thanks for the help.. Is there any reason why you shouldnt have eval in the summary index populating search?

[skoelpin]

Yes because you are essentially creating a new field and adding extra workload while the data is being indexed. It's better to add that workload after it's been re-indexed

[mwdbhyat]

To add to this the counts need to be counts for the past 30days..