Reporting

Caution on Retention – Impact of Accelerated Data Model and Report acceleration when using volume based retention policies

sat94541
Communicator

Be careful when you set size-based retention limits for your indexes so they do not take up too much disk storage space. By default, report acceleration summaries can theoretically take up an unlimited amount of disk space. This can be a problem if you're also locking down the maximum data size of your indexes or index volumes.

1 Solution

rbal_splunk
Splunk Employee
Splunk Employee

The good news is that you can optionally configure retention limits for your report acceleration summaries or Data Model acceleration.

Note: Although report acceleration summaries are unbounded in size by default, they are tied to raw data in your warm and hot index buckets and will age along with it. When events pass out of the hot/warm buckets into cold buckets, they are likewise removed from the related summaries. Same is true for Data Model Acceleration.

For example, by default, report acceleration summaries live alongside the hot and warm buckets in your index at homePath/../summary/. In other words, if in indexes.conf the homePath for the hot and warm buckets in your index is:

homePath = /opt/splunk/var/lib/splunk/index1/db
Then summaries that map to buckets in that index will be created at:
homePath/opt/splunk/var/lib/splunk/index1/summary

for example you can have index like
[winevents]
coldPath= volume:Seconday/winevents/colddb
homePath= volume:primary/winevents/db
tstatsHomePath= volume:_Data_Model\$_index_name\datamodel_summary
summaryHomePath== volume:_reportacc_summaries\$_index_name\datamodel_summary

So to manage disk utilization better, you will need to define separate volume for

homepath>hot and warm bucket
coldPath>Cold Buckets
Data Model Acceleration
Report Acceleration

And set each of the volume like
[volume:primary]
path =

maxVolumeDataSizeMB= < allowed size>

[volume:seconday]
path =

maxVolumeDataSizeMB= < allowed size>

[volume:_ Data_Model]
path = $SPLUNK_DB
maxVolumeDataSizeMB= < allowed size>

[volume: :_reportacc_summaries]
path = /Data/report_acceleration/
maxVolumeDataSizeMB= < allowed size>

This information is documented at location --http://docs.splunk.com/Documentation/Splunk/6.2.2/Knowledge/Manageacceleratedsearchsummaries and look link http://docs.splunk.com/Documentation/Splunk/6.2.2/Knowledge/Acceleratedatamodels Section “Configure size-based retention for data model summaries”

View solution in original post

rbal_splunk
Splunk Employee
Splunk Employee

The good news is that you can optionally configure retention limits for your report acceleration summaries or Data Model acceleration.

Note: Although report acceleration summaries are unbounded in size by default, they are tied to raw data in your warm and hot index buckets and will age along with it. When events pass out of the hot/warm buckets into cold buckets, they are likewise removed from the related summaries. Same is true for Data Model Acceleration.

For example, by default, report acceleration summaries live alongside the hot and warm buckets in your index at homePath/../summary/. In other words, if in indexes.conf the homePath for the hot and warm buckets in your index is:

homePath = /opt/splunk/var/lib/splunk/index1/db
Then summaries that map to buckets in that index will be created at:
homePath/opt/splunk/var/lib/splunk/index1/summary

for example you can have index like
[winevents]
coldPath= volume:Seconday/winevents/colddb
homePath= volume:primary/winevents/db
tstatsHomePath= volume:_Data_Model\$_index_name\datamodel_summary
summaryHomePath== volume:_reportacc_summaries\$_index_name\datamodel_summary

So to manage disk utilization better, you will need to define separate volume for

homepath>hot and warm bucket
coldPath>Cold Buckets
Data Model Acceleration
Report Acceleration

And set each of the volume like
[volume:primary]
path =

maxVolumeDataSizeMB= < allowed size>

[volume:seconday]
path =

maxVolumeDataSizeMB= < allowed size>

[volume:_ Data_Model]
path = $SPLUNK_DB
maxVolumeDataSizeMB= < allowed size>

[volume: :_reportacc_summaries]
path = /Data/report_acceleration/
maxVolumeDataSizeMB= < allowed size>

This information is documented at location --http://docs.splunk.com/Documentation/Splunk/6.2.2/Knowledge/Manageacceleratedsearchsummaries and look link http://docs.splunk.com/Documentation/Splunk/6.2.2/Knowledge/Acceleratedatamodels Section “Configure size-based retention for data model summaries”

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...