Getting Data In

Configuring Splunk for Multiple Indexer Partitions

gbowden_pheaa
Path Finder

I am not sure how to configure the indexes.conf AND the splunk-launch.conf. I understand multiple volumes in indexes.conf, such as:

[volume:hotwarm]
path = /splunkindexes/hot

[volume:cold]
path = /splunkdata

and using this in the index definition in indexes.conf:

[main]
homePath = volume:hotwarm/defaultdb/db
coldPath = volume:cold/defaultdb/colddb
thawedPath = $SPLUNK_DB/defaultdb/thaweddb
repFactor = auto

I have the splunk-launch.conf set as:

SPLUNK_HOME=/splunk
SPLUNK_DB = /splunkindexes/hot

setting the SPLUNK_DB to the hotwarm volume.

I am not seeing the cold volume data. I just made these changes, and have the hot to cold rollover set "disk size", so it will stay hot until the disk is close to full then it will roll to cold.

Any ideas on way I cannot see the existing cold data? Thanks...

0 Karma

gbowden_pheaa
Path Finder

solarboyz1 provided the answer - I was not referencing the correct path in my cold path, so the data was not being seen. Thank you!

0 Karma

solarboyz1
Builder

Do you have the Splunk process running as a non-root user, and does that non-root user have access to /splunkdata

Do you see any errors in splunkd.log or your internal index referencing the cold location.

Are there currently buckets in the cold location?

0 Karma

gbowden_pheaa
Path Finder

We have a non-root user that owns the /splunkdata partition, with 700 rights. We are not seeing errors regarding the cold location. The cold location was the previous primary partition of all indexes, we've added a "flash" drive to handle the hot/warm buckets. Hot/Warm appears to be running fine. There are numerous buckets in the various indexes in the cold partition.

0 Karma

solarboyz1
Builder

So, /splunkdata used to be the hot/warm (homepath)

If so, is the directory structure still /splunkdata/${indexname}/db or did you rename the db direcotries to colddb?

If they are still named db and your indexes.conf reference colddb...that could be the issue

0 Karma

gbowden_pheaa
Path Finder

You are right on - I tried both changing the colddb name from index/db to index/colddb, and changing the indexes.conf to use coldpath = volume:cold/index/db, and both worked. Thank you!

Now the big question, we do not freeze data, we roll it off. Will there be advantage to one method over the other in data retention and retention processing? I need to dig more into this to determine of data can roll from volume:hotwarm/index/db to volume:cold/index/db to see if there are issues, or if the data has to roll from the colddb directory.

Thank you, this is a huge help!

0 Karma

solarboyz1
Builder

So, volumes provide a better way to manage space when its shared among indexes.

We set a frozen time per index, then we set a maxVolumeDataSize for the volume that is set to ~90% of the storage partition size.

When the storage hits 90%, Splunk will go through ALL indexes on the volume and start rolling buckets until the size is below 90%.

Basically, we use the volume setting as the failsafe to ensure the storage doesn't get exhausted. We set high homepath and maxdb size per index, managing the total storage by the volume settings.

This works for us, there are advantages/disadvantages to this approach.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...