Reporting

Data model, saved search or summary index?

javiierg14
New Member

I need to know which of these methods is better for this scenario:

I have a big log of events that index 2.5 million of events every day, this log is a raw text that require a complex Regular Expression to get the fields and values, i have like 10 dashboard feeding from this log, one of them is a report view where me and my team search event with multiples filters that are dinamilly choose from tokens.

these reports takes to much time when the time range is seven day ago or more, it's very hard generate a report of the top 10 events, or the distributions of errors.

the problem is that the time range selected is very random, one day we need a today report, then a 3 months ago or especific day, I need a method to optimize this reports and reduce the duration of the jobs.

I have tried with make all the dashboard run a base search and then post process the results on each panel, this did'nt reduce the duration.

So, what you recommend, use a saved search, a summary index or data model?

keep in mind, the time range selected it's very variable

0 Karma

mayurr98
Super Champion

Saved search does not make any sense here as there are many reports and some of them might be token based which you can not accelerate.
Based on my experience, I would recommend you to use data model, as it is meant to process large amount of data in a rapid and efficient way. After building a data model you can accelerate it and make as many reports/dashboards you want.
To accelerate data model follow these steps:
To accelerate the data model go to the Data Model Manager page (it says "Data Models" at the top and has an Actions column; you get to it from the Data Model Editor page by clicking "Back to Data Models").

Click Edit and select Edit Permissions. Share the object with the App or All Apps. (Only shared objects can be accelerated.)

Click Edit again and click Edit Acceleration.

In the Edit Acceleration dialog select Accelerate and then select a Summary Range. Summary range is the amount of time that you need to be accelerated. The bigger the range, the more space the acceleration summary will take up on disk and the longer it will take to create, so don't choose a range that is longer than you need it to be. For example, if you don't plan to search over more than the last week or two, select a range of 1 Month.

I hope this helps you!

Save your acceleration changes. Your model is now accelerated.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In December, the Splunk Threat Research Team had 1 release of new security content via the Enterprise Security ...

Why am I not seeing the finding in Splunk Enterprise Security Analyst Queue?

(This is the first of a series of 2 blogs). Splunk Enterprise Security is a fantastic tool that offers robust ...

Index This | What are the 12 Days of Splunk-mas?

December 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...