How to add additional SPLUNK_COLD_DB path?

[sim_tcr]

We are running splunk 6.3.3 with a clustered environment (index cluster and search head cluster)
Below is how our indexes are configured,

[indexname]
repFactor=auto
homePath = $SPLUNK_DB/indexname/db
coldPath = $SPLUNK_COLD_DB/indexname/colddb
thawedPath = $SPLUNK_DB/indexname/thaweddb

maxWarmDBCount = 60

frozenTimePeriodInSecs = 2592000

$SPLUNK_DB is a physical drive directly mounted on the indexer /Splunk
$SPLUNK_COLD_DB is NFS volume mounted on the indexers at /Data

$SPLUNK_COLD_DB is already at its max (15TB) we cannot increase the size of it any more.

How can we add another $SPLUNK_COLD_DB and start sending events to that at the same $SPLUNK_COLD_DB is still available for users to search events from.

Thanks.

[nickhills]

Not that it helps in your situation (because you are on 6.3.x) but for other users who read this post in the future.

You could also consider moving your data to SmartStore which effectively replaces your cold and warm database locations with an Amazon S3 bucket (or compatible service/appliance).

https://docs.splunk.com/Documentation/Splunk/8.0.2/Indexer/MigratestandalonetoSmartStore

It also has the huge benefit of decoupling the compute of your indexing tier from the inherent challenges of local (or near) storage, making adding and perhaps more importantly removing indexers from your cluster vastly faster and simpler.

Whilst Splunk 7.2 introduces support for SmartStore, there have been a large number of enhancements in later versions. - I strongly suggest you consider upgrading to Splunk 8 if you plan to use SS

[gcusello · Accepted solution]

Hi @sim_tcr,
for my knowledge, it isn't possible to have two coldPath folders.
The best approach is to move the cold folder of some indexes in a different storage (e.g. /Data2) using an absolute path, having in this way free storage in the main cold storage.
e.g.:

[indexname1]
repFactor=auto
homePath = $SPLUNK_DB/indexname/db
coldPath = $SPLUNK_COLD_DB/indexname/colddb
thawedPath = $SPLUNK_DB/indexname/thaweddb
##maxWarmDBCount = 60
frozenTimePeriodInSecs = 2592000
[indexname2]
repFactor=auto
homePath = $SPLUNK_DB/indexname/db
coldPath = /Data2/indexname/colddb
thawedPath = $SPLUNK_DB/indexname/thaweddb
##maxWarmDBCount = 60
frozenTimePeriodInSecs = 2592000

Ciao.
Giuseppe

[sim_tcr]

So are you suggesting, we add another volume /Data2
And on any net new indexes send them to /Data2?

Do you know the procedure to move existing index data to /Data2?

[gcusello]

No, sorry I wasn't clear:

• I suggest to add a new volume /Data2
• and move to them some of your indexes cold folders not the new ones,
• in this way you'll have some indexes that have cold folders in Data1 (using the $SPLUNK_COLD_DB variable in coldPath option) and some others with cold folders in /Data2 (using the absolute path /Data2 in coldPath option),
• in this way you have some free storage in /Data1 (the space of the moved indexes) and your indexes (located in this storage) can grow relatively to your needs and the other indexes in /Data2.

You can do the choose of indexes to move to the new location analyzing the cold folder dimension and growth.

In other words, if you have e.g the following indexes, you can divide them in this way:

• index1, 5TB, /Data1
• Index2, 5TB, /Data2
• Index3, 3TB, /Data1
• Index4, 2TB. /Data2

So you'll use in /Data1 8TB and in /Data2 7TB that can grow to the maximun storage dimension.

Ciao.
Giuseppe