Using multiple summary indexes

[sc0tt]

Our reporting needs are starting to grow so I am planning on creating new summaries and would like to use best practices to manage these summaries while trying to plan ahead as best as possible. I came across another post (here) about using multiple indexes for managing summaries. Based on the answer, I plan use the same structure and create 3 separate indexes (summary_5m, summary_1h, summary_1d).

Is this a good practice? Are there any other methods that may be better?

[emotz · Accepted solution]

Yes, it is a good practice. The objective of summary indexing in general is to reduce the amount of data to be searched/processed by an order of magnitude. Assuming that you would be using these summaries to provide dashboards and in the future trending either month over month or year over year, you will need some level of granularity to support those use cases.

In general data that will be searched together and is of the same type should only be put into another index for security (one group can see and and others should not) or for retention (you want to keep certain data longer than others, maybe for compliance reasons). Given that, if you need to keep the less than 1 hour summary information for 90 days and the 1 hour summary for 2 years and the 1d summary forever, they should all be put into different indexes.

Happy summarizing,
reduce, reuse, recycle

[sc0tt]

Thanks for your response and insight.