Reporting

Caution on Retention – Impact of Accelerated Data Model and Report acceleration when using volume based retention policies

sat94541
Communicator

Be careful when you set size-based retention limits for your indexes so they do not take up too much disk storage space. By default, report acceleration summaries can theoretically take up an unlimited amount of disk space. This can be a problem if you're also locking down the maximum data size of your indexes or index volumes.

1 Solution

rbal_splunk
Splunk Employee
Splunk Employee

The good news is that you can optionally configure retention limits for your report acceleration summaries or Data Model acceleration.

Note: Although report acceleration summaries are unbounded in size by default, they are tied to raw data in your warm and hot index buckets and will age along with it. When events pass out of the hot/warm buckets into cold buckets, they are likewise removed from the related summaries. Same is true for Data Model Acceleration.

For example, by default, report acceleration summaries live alongside the hot and warm buckets in your index at homePath/../summary/. In other words, if in indexes.conf the homePath for the hot and warm buckets in your index is:

homePath = /opt/splunk/var/lib/splunk/index1/db
Then summaries that map to buckets in that index will be created at:
homePath/opt/splunk/var/lib/splunk/index1/summary

for example you can have index like
[winevents]
coldPath= volume:Seconday/winevents/colddb
homePath= volume:primary/winevents/db
tstatsHomePath= volume:_Data_Model\$_index_name\datamodel_summary
summaryHomePath== volume:_reportacc_summaries\$_index_name\datamodel_summary

So to manage disk utilization better, you will need to define separate volume for

homepath>hot and warm bucket
coldPath>Cold Buckets
Data Model Acceleration
Report Acceleration

And set each of the volume like
[volume:primary]
path =

maxVolumeDataSizeMB= < allowed size>

[volume:seconday]
path =

maxVolumeDataSizeMB= < allowed size>

[volume:_ Data_Model]
path = $SPLUNK_DB
maxVolumeDataSizeMB= < allowed size>

[volume: :_reportacc_summaries]
path = /Data/report_acceleration/
maxVolumeDataSizeMB= < allowed size>

This information is documented at location --http://docs.splunk.com/Documentation/Splunk/6.2.2/Knowledge/Manageacceleratedsearchsummaries and look link http://docs.splunk.com/Documentation/Splunk/6.2.2/Knowledge/Acceleratedatamodels Section “Configure size-based retention for data model summaries”

View solution in original post

rbal_splunk
Splunk Employee
Splunk Employee

The good news is that you can optionally configure retention limits for your report acceleration summaries or Data Model acceleration.

Note: Although report acceleration summaries are unbounded in size by default, they are tied to raw data in your warm and hot index buckets and will age along with it. When events pass out of the hot/warm buckets into cold buckets, they are likewise removed from the related summaries. Same is true for Data Model Acceleration.

For example, by default, report acceleration summaries live alongside the hot and warm buckets in your index at homePath/../summary/. In other words, if in indexes.conf the homePath for the hot and warm buckets in your index is:

homePath = /opt/splunk/var/lib/splunk/index1/db
Then summaries that map to buckets in that index will be created at:
homePath/opt/splunk/var/lib/splunk/index1/summary

for example you can have index like
[winevents]
coldPath= volume:Seconday/winevents/colddb
homePath= volume:primary/winevents/db
tstatsHomePath= volume:_Data_Model\$_index_name\datamodel_summary
summaryHomePath== volume:_reportacc_summaries\$_index_name\datamodel_summary

So to manage disk utilization better, you will need to define separate volume for

homepath>hot and warm bucket
coldPath>Cold Buckets
Data Model Acceleration
Report Acceleration

And set each of the volume like
[volume:primary]
path =

maxVolumeDataSizeMB= < allowed size>

[volume:seconday]
path =

maxVolumeDataSizeMB= < allowed size>

[volume:_ Data_Model]
path = $SPLUNK_DB
maxVolumeDataSizeMB= < allowed size>

[volume: :_reportacc_summaries]
path = /Data/report_acceleration/
maxVolumeDataSizeMB= < allowed size>

This information is documented at location --http://docs.splunk.com/Documentation/Splunk/6.2.2/Knowledge/Manageacceleratedsearchsummaries and look link http://docs.splunk.com/Documentation/Splunk/6.2.2/Knowledge/Acceleratedatamodels Section “Configure size-based retention for data model summaries”

Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...