How can I change the location of accelerated data model summary data?

[anthony_copus]

Hi,

I've currently got a few accelerated data models set up, however it seems that the summary data is always written to the same place: $SPLUNK_HOME/var/lib/splunk/index_name/datamodel_summary

How can I alter the location where these are written as I need to spread them over a few different disks?

[mikaelbje]

The data model storage path/tstatsHomePath for all indexes by default points to volume:_splunk_summaries/$_index_name/datamodel_summary so it's just a matter of changing the storage path of the _splunk_summaries volume:

My preferred way of changing the default behaviour is the following:

In $SPLUNK_HOME/etc/system/local/indexes.conf:

[volume:_splunk_summaries]
# 60 GB
maxVolumeDataSizeMB = 60000
# Preferred partition for summaries
path = /splunk_summaries

This way you change the default location of the _splunk_summaries volume out of the Splunk home dir and you won't have to change it per index. If you want more granular control per index you'd have to change tstatsHomePath for each one as stated in the other answer.

[mattness]

Try taking advantage of the tstatshomepath setting on indexes.conf by defining a different one for each of your indexes. From indexes.conf.spec:

tstatsHomePath = <path on index server>
* Location where datamodel acceleration TSIDX data for this index should be stored
* If specified, MUST be defined in terms of a volume definition (see volume section below)
* If not specified it defaults to volume:_splunk_summaries/$_index_name/datamodel_summary, where $_index_name is the name of the index
* CAUTION: Path must be writable.
* Must restart splunkd after changing this parameter; index reload will not suffice.

This subtopic, which covers setup of size-based retention across indexers, provides an example configuration.

[Lucas_K]

Datamodel summaries are kept with the indexes they are generated from.

The .tsidx files that make up a high-performance analytics store for a single data model are always distributed across one or more of your indexers. This is because Splunk Enterprise creates .tsidx files on the indexer, parallel to the buckets that contain the events referenced in the file and which cover the range of time that the summary spans.

The best you can do is split out each index to different locations.

You can't split individual hot/warm indexes for a single specific index to multiple disks (unless you do some funky symlinking). You can split out hot/warm and cold however.

Simple define your disks into volumes and then define your index per volume.

http://docs.splunk.com/Documentation/Splunk/6.1.3/admin/Indexesconf

Example. indexes.conf

[volume:fastdisk1]
path=/dev/fastdisk1

[volume:fastdisk2]
path=/dev/fastdisk2

[volume:cold1]
path=/dev/slowdisk1

[volume:cold2]
path=/dev/slowdisk2

[idx1]
homePath = volume:fastdisk1/idx1
coldPath = volume:cold2/idx1
thawedPath = $SPLUNK_DB/idx1/thaweddb
...

[idx2]
homePath = volume:fastdisk2/idx2
coldPath = volume:cold2/idx2
thawedPath = $SPLUNK_DB/idx2/thaweddb
...

[Lucas_K]

as per mattness's post

it should be

[video_view_raw]
coldPath = /content4/video_view_raw/colddb
thawedPath = /content4/video_view_raw/thaweddb
homePath = /content4/video_view_raw/db
tstatsHomePath = /content4/video_view_raw/datamodel_summary
maxTotalDataSizeMB = 950000

[anthony_copus]

To give you more info to help me out, here's the index in the indexes.conf:

[video_view_raw]
coldPath = /content4/video_view_raw/colddb
thawedPath = /content4/video_view_raw/thaweddb
homePath = /content4/video_view_raw/db
maxTotalDataSizeMB = 950000

However, the datamodel summary is appearing in:

/opt/splunk/var/lib/splunk/video_view_raw/datamodel_summary

The constaints in the summary objects only refer to this index, so what you're saying is it should appear in the content4 disk?

[anthony_copus]

This is on 6.0.1

[Lucas_K]

Really? hmm. So its not "parallel to the buckets that contain the events referenced in the file"? What version of 6 is that happening on?

[anthony_copus]

This isn't actually what's happening. The indexes are already on completely different disks, but the datamodel summaries are still created on the disk which splunk is installed on.

If our datamodel with constaint "index=video" went onto the same disk as where that index is actually located then that would be fine, but it doesn't.