Reporting
Highlighted

How can I change the location of accelerated data model summary data?

Explorer

Hi,

I've currently got a few accelerated data models set up, however it seems that the summary data is always written to the same place: $SPLUNKHOME/var/lib/splunk/indexname/datamodel_summary

How can I alter the location where these are written as I need to spread them over a few different disks?

Highlighted

Re: How can I change the location of accelerated data model summary data?

Motivator

Datamodel summaries are kept with the indexes they are generated from.

The .tsidx files that make up a high-performance analytics store for a single data model are always distributed across one or more of your indexers. This is because Splunk Enterprise creates .tsidx files on the indexer, parallel to the buckets that contain the events referenced in the file and which cover the range of time that the summary spans.

The best you can do is split out each index to different locations.

You can't split individual hot/warm indexes for a single specific index to multiple disks (unless you do some funky symlinking). You can split out hot/warm and cold however.

Simple define your disks into volumes and then define your index per volume.

http://docs.splunk.com/Documentation/Splunk/6.1.3/admin/Indexesconf

Example. indexes.conf

[volume:fastdisk1]
path=/dev/fastdisk1

[volume:fastdisk2]
path=/dev/fastdisk2

[volume:cold1]
path=/dev/slowdisk1

[volume:cold2]
path=/dev/slowdisk2

[idx1]
homePath = volume:fastdisk1/idx1
coldPath = volume:cold2/idx1
thawedPath = $SPLUNK_DB/idx1/thaweddb
...

[idx2]
homePath = volume:fastdisk2/idx2
coldPath = volume:cold2/idx2
thawedPath = $SPLUNK_DB/idx2/thaweddb
...
Highlighted

Re: How can I change the location of accelerated data model summary data?

Explorer

This isn't actually what's happening. The indexes are already on completely different disks, but the datamodel summaries are still created on the disk which splunk is installed on.

If our datamodel with constaint "index=video" went onto the same disk as where that index is actually located then that would be fine, but it doesn't.

0 Karma
Highlighted

Re: How can I change the location of accelerated data model summary data?

Motivator

Really? hmm. So its not "parallel to the buckets that contain the events referenced in the file"? What version of 6 is that happening on?

0 Karma
Highlighted

Re: How can I change the location of accelerated data model summary data?

Explorer

This is on 6.0.1

0 Karma
Highlighted

Re: How can I change the location of accelerated data model summary data?

Explorer

To give you more info to help me out, here's the index in the indexes.conf:

[videoviewraw]
coldPath = /content4/videoviewraw/colddb
thawedPath = /content4/videoviewraw/thaweddb
homePath = /content4/videoviewraw/db
maxTotalDataSizeMB = 950000

However, the datamodel summary is appearing in:

/opt/splunk/var/lib/splunk/videoviewraw/datamodel_summary

The constaints in the summary objects only refer to this index, so what you're saying is it should appear in the content4 disk?

0 Karma
Highlighted

Re: How can I change the location of accelerated data model summary data?

Motivator

as per mattness's post

it should be

[videoviewraw]
coldPath = /content4/videoviewraw/colddb
thawedPath = /content4/videoviewraw/thaweddb
homePath = /content4/videoviewraw/db
tstatsHomePath = /content4/videoviewraw/datamodel_summary
maxTotalDataSizeMB = 950000

Highlighted

Re: How can I change the location of accelerated data model summary data?

Splunk Employee
Splunk Employee

Try taking advantage of the tstatshomepath setting on indexes.conf by defining a different one for each of your indexes. From indexes.conf.spec:

tstatsHomePath = <path on index server>
* Location where datamodel acceleration TSIDX data for this index should be stored
* If specified, MUST be defined in terms of a volume definition (see volume section below)
* If not specified it defaults to volume:_splunk_summaries/$_index_name/datamodel_summary, where $_index_name is the name of the index
* CAUTION: Path must be writable.
* Must restart splunkd after changing this parameter; index reload will not suffice.

This subtopic, which covers setup of size-based retention across indexers, provides an example configuration.

Highlighted

Re: How can I change the location of accelerated data model summary data?

Motivator

The data model storage path/tstatsHomePath for all indexes by default points to volume:splunksummaries/$indexname/datamodel_summary so it's just a matter of changing the storage path of the splunksummaries volume:

My preferred way of changing the default behaviour is the following:

In $SPLUNK_HOME/etc/system/local/indexes.conf:

[volume:_splunk_summaries]
# 60 GB
maxVolumeDataSizeMB = 60000
# Preferred partition for summaries
path = /splunk_summaries

This way you change the default location of the splunksummaries volume out of the Splunk home dir and you won't have to change it per index. If you want more granular control per index you'd have to change tstatsHomePath for each one as stated in the other answer.