topic Re: What is the best practice: Implicit or Explicit Index Path Locations? in Splunk Dev

What is the best practice: Implicit or Explicit Index Path Locations?

paimonsoror — Tue, 29 Sep 2020 16:03:38 GMT

Curious on what is the recommended? I know the second one makes sense for readability, but the first one i feel would greatly reduce retyping and indexes.conf file size:

Practice 1
[default]
coldPath=$SPLUNK_DB/$_index_name/colddb
homePath=$SPLUNK_DB/$_index_name/db
thawedPath=$SPLUNK_DB/$_index_name/thaweddb
frozenTimePeriodInSecs = 200000

[foo]
frozenTimePeriodInSecs = 100000

[bar]

Practice 2
[default]
frozenTimePeriodInSecs = 200000

[foo]
coldPath=$SPLUNK_DB/foo/colddb
homePath=$SPLUNK_DB/foo/db
thawedPath=$SPLUNK_DB/foo/thaweddb
frozenTimePeriodInSecs = 100000

[bar]
coldPath=$SPLUNK_DB/bar/colddb
homePath=$SPLUNK_DB/bar/db
thawedPath=$SPLUNK_DB/bar/thaweddb

Re: What is the best practice: Implicit or Explicit Index Path Locations?

inventsekar — Sat, 30 Sep 2017 04:14:52 GMT

Practice 1 looks perfect, as you said - the first one i feel would greatly reduce retyping and indexes.conf file size.
easy to manage.

also (we can assume 😉 ) the indexes.conf suggest practice 1

homePath = <path on index server>
* An absolute path that contains the hotdb and warmdb for the index.
* It is recommended that you specify the path with the following syntax: 
     homePath = $SPLUNK_DB/$_index_name/db
  At runtime, Splunk expands "$_index_name" to the name of the index. For example, 
  if the index name is "newindex", homePath becomes "$SPLUNK_DB/newindex/db".
* Splunkd keeps a file handle open for warmdbs at all times.
* May contain a volume reference (see volume section below) in place of $SPLUNK_DB.
* CAUTION: The parent path "$SPLUNK_DB/$_index_name/" must be writable.
* Required. Splunk will not start if an index lacks a valid homePath.
* Must restart splunkd after changing this parameter; index reload will not
  suffice.
* We strongly recommend that you avoid the use of other environment variables in
  index paths, aside from the possible exception of SPLUNK_DB.
  * As an exception, SPLUNK_DB is explicitly managed by the provided software,
    so most possible downsides here do not exist.
  * Environment variables could be different from launch to launch of the
    software, causing severe problems with management of indexed data,
    including:
    * Data in the prior location will not be searchable.
    * The indexer may not be able to write to the new location, causing outages
      and/or data loss.
    * Writing to a new, unexpected location could lead to disk exhaustion
      causing additional operational problems.
    * Recovery from such a scenario will require manual intevention and bucket
      renaming, especially difficult in an index clustered environment.
    * In all circumstances, Splunk Diag, the diagnostic tool we use to support
      you, will have no way to determine the correct values for the environment
      variables, so cannot reliably operate.  You may need to manually acquire
      information about your index buckets in troubleshooting scenarios.
  * Generally speaking, volumes provide a more appropriate way to control the
    storage location for indexes in a general way.

Re: What is the best practice: Implicit or Explicit Index Path Locations?

paimonsoror — Tue, 03 Oct 2017 12:42:45 GMT

Thanks for the response. Good call out on the docs as well 🙂 Should have done more reading. Anyway, I switched over to the mentioned nomenclature and love it since it reduced my indexes.conf file by nearly half and really made it so much cleaner looking.

Re: What is the best practice: Implicit or Explicit Index Path Locations?

sloshburch — Fri, 06 Oct 2017 21:55:04 GMT

I embraced this approach and have reduced my index stanzas to just the stanza header in some cases. I think it's gonna make this easier for administration too.

Did you learn about this during a .conf2017 session? The timing of this post and my discussion of it at .conf2017 is interesting...

Re: What is the best practice: Implicit or Explicit Index Path Locations?

paimonsoror — Sat, 07 Oct 2017 01:52:04 GMT

Hehe busted! Your talk is exactly what got me saying to myself "whyyy am I not doing thisssssssss"

Re: What is the best practice: Implicit or Explicit Index Path Locations?

sloshburch — Mon, 09 Oct 2017 12:59:56 GMT

Huzzah! Glad I could help!

I asked myself the same thing, and I think it's cause we didn't really promote that $_index_name was created. So without that, you were handcuffed from this.

Anyway, depending on your version, you may see an empty $_index_name folder created. But fear not, it is known by the product side and it appears to really have no impact (no data stored in there). Just didn't want you to fear if you saw that.

Enjoy and glad I could help!
(also, make sure you rated the session!)

Re: What is the best practice: Implicit or Explicit Index Path Locations?

ddrillic — Mon, 09 Oct 2017 13:33:42 GMT

Plain gorgeous !!!!

[default]
coldPath=$SPLUNK_DB/$_index_name/colddb
homePath=$SPLUNK_DB/$_index_name/db
thawedPath=$SPLUNK_DB/$_index_name/thaweddb
frozenTimePeriodInSecs = 200000

Re: What is the best practice: Implicit or Explicit Index Path Locations?

sloshburch — Tue, 10 Oct 2017 14:29:30 GMT

You know how to make an engineer blush! 🙂

Re: What is the best practice: Implicit or Explicit Index Path Locations?

ddrillic — Fri, 20 Oct 2017 15:40:44 GMT

; - ) you know @SloshBurch, I was about to implement it when I realized that for the out-of-the-box indexes we don't specify coldPath, homePath etc. Would I break anything, if I add this default stanza?

We have them defined as -

[_internal]
repFactor = auto
frozenTimePeriodInSecs = 34186698
maxTotalDataSizeMB = 4000000
enableTsidxReduction = true
timePeriodInSecBeforeTsidxReduction = 3024000
maxDataSize = auto_high_volume

# 395.679375 Days
# Minify .tsidx files 35 Days
[_audit]
repFactor = auto
frozenTimePeriodInSecs = 34186698
enableTsidxReduction = true
timePeriodInSecBeforeTsidxReduction = 3024000
maxDataSize = auto_high_volume

Re: What is the best practice: Implicit or Explicit Index Path Locations?

sloshburch — Mon, 23 Oct 2017 17:05:00 GMT

I don't believe so but you should really throw down some btool to be sure. Fair?

Re: What is the best practice: Implicit or Explicit Index Path Locations?

ddrillic — Mon, 23 Oct 2017 23:08:48 GMT

Perfect @SloshBurch.