Reporting

Caution on Retention – Impact of Accelerated Data Model and Report acceleration when using volume based retention policies

sat94541
Communicator

Be careful when you set size-based retention limits for your indexes so they do not take up too much disk storage space. By default, report acceleration summaries can theoretically take up an unlimited amount of disk space. This can be a problem if you're also locking down the maximum data size of your indexes or index volumes.

1 Solution

rbal_splunk
Splunk Employee
Splunk Employee

The good news is that you can optionally configure retention limits for your report acceleration summaries or Data Model acceleration.

Note: Although report acceleration summaries are unbounded in size by default, they are tied to raw data in your warm and hot index buckets and will age along with it. When events pass out of the hot/warm buckets into cold buckets, they are likewise removed from the related summaries. Same is true for Data Model Acceleration.

For example, by default, report acceleration summaries live alongside the hot and warm buckets in your index at homePath/../summary/. In other words, if in indexes.conf the homePath for the hot and warm buckets in your index is:

homePath = /opt/splunk/var/lib/splunk/index1/db
Then summaries that map to buckets in that index will be created at:
homePath/opt/splunk/var/lib/splunk/index1/summary

for example you can have index like
[winevents]
coldPath= volume:Seconday/winevents/colddb
homePath= volume:primary/winevents/db
tstatsHomePath= volume:_Data_Model\$_index_name\datamodel_summary
summaryHomePath== volume:_reportacc_summaries\$_index_name\datamodel_summary

So to manage disk utilization better, you will need to define separate volume for

homepath>hot and warm bucket
coldPath>Cold Buckets
Data Model Acceleration
Report Acceleration

And set each of the volume like
[volume:primary]
path =

maxVolumeDataSizeMB= < allowed size>

[volume:seconday]
path =

maxVolumeDataSizeMB= < allowed size>

[volume:_ Data_Model]
path = $SPLUNK_DB
maxVolumeDataSizeMB= < allowed size>

[volume: :_reportacc_summaries]
path = /Data/report_acceleration/
maxVolumeDataSizeMB= < allowed size>

This information is documented at location --http://docs.splunk.com/Documentation/Splunk/6.2.2/Knowledge/Manageacceleratedsearchsummaries and look link http://docs.splunk.com/Documentation/Splunk/6.2.2/Knowledge/Acceleratedatamodels Section “Configure size-based retention for data model summaries”

View solution in original post

rbal_splunk
Splunk Employee
Splunk Employee

The good news is that you can optionally configure retention limits for your report acceleration summaries or Data Model acceleration.

Note: Although report acceleration summaries are unbounded in size by default, they are tied to raw data in your warm and hot index buckets and will age along with it. When events pass out of the hot/warm buckets into cold buckets, they are likewise removed from the related summaries. Same is true for Data Model Acceleration.

For example, by default, report acceleration summaries live alongside the hot and warm buckets in your index at homePath/../summary/. In other words, if in indexes.conf the homePath for the hot and warm buckets in your index is:

homePath = /opt/splunk/var/lib/splunk/index1/db
Then summaries that map to buckets in that index will be created at:
homePath/opt/splunk/var/lib/splunk/index1/summary

for example you can have index like
[winevents]
coldPath= volume:Seconday/winevents/colddb
homePath= volume:primary/winevents/db
tstatsHomePath= volume:_Data_Model\$_index_name\datamodel_summary
summaryHomePath== volume:_reportacc_summaries\$_index_name\datamodel_summary

So to manage disk utilization better, you will need to define separate volume for

homepath>hot and warm bucket
coldPath>Cold Buckets
Data Model Acceleration
Report Acceleration

And set each of the volume like
[volume:primary]
path =

maxVolumeDataSizeMB= < allowed size>

[volume:seconday]
path =

maxVolumeDataSizeMB= < allowed size>

[volume:_ Data_Model]
path = $SPLUNK_DB
maxVolumeDataSizeMB= < allowed size>

[volume: :_reportacc_summaries]
path = /Data/report_acceleration/
maxVolumeDataSizeMB= < allowed size>

This information is documented at location --http://docs.splunk.com/Documentation/Splunk/6.2.2/Knowledge/Manageacceleratedsearchsummaries and look link http://docs.splunk.com/Documentation/Splunk/6.2.2/Knowledge/Acceleratedatamodels Section “Configure size-based retention for data model summaries”

Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) v3.54.0

The Splunk Threat Research Team (STRT) recently released Enterprise Security Content Update (ESCU) v3.54.0 and ...

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...