Deployment Architecture

Is there a way to separate the hot and warm bucket path?

PPape
Contributor

Hi,

i found this in the Docs: Use multiple partitions for index data
But there i can only configure

  • homePath = ... This is the path that contains the hot and warm databases for the index. Caution: The path must be writable.
  • coldPath = ... This is the path that contains the cold databases for the index. Caution: The path must be writable. Is there a way to separate the hot and the warm Path? I have a system with SSD, SAS15k, and SAS10k, and I only want the hot Bucket on the SSD.

Thanks for your help!

Tags (4)
1 Solution

dwaddle
SplunkTrust
SplunkTrust

No it's not possible to separate hot and warm.

The fundamental difference between a hot bucket and a warm bucket is "am I writing to it?" When a bucket is hot, there are open file descriptors into the files comprising the bucket and the files are being written to. When searches launch, the search process forked inherits those open file descriptors.

When a bucket is warm, the file descriptors remain open just as in hot - but there is no writing.

The open descriptors are why hot and warm must coexist on the same filesystem ; you cannot move a file from one filesystem to another and maintain an open descriptor against it. A cross-filesystem move is really a "copy and delete" so trying to do this with open files results in the descriptors being pointed at deleted files and the space for those deleted files not being reclaimed.

So, architecturally, Splunk puts hot and warm in the same filesystem and you cannot separate them.

View solution in original post

dwaddle
SplunkTrust
SplunkTrust

No it's not possible to separate hot and warm.

The fundamental difference between a hot bucket and a warm bucket is "am I writing to it?" When a bucket is hot, there are open file descriptors into the files comprising the bucket and the files are being written to. When searches launch, the search process forked inherits those open file descriptors.

When a bucket is warm, the file descriptors remain open just as in hot - but there is no writing.

The open descriptors are why hot and warm must coexist on the same filesystem ; you cannot move a file from one filesystem to another and maintain an open descriptor against it. A cross-filesystem move is really a "copy and delete" so trying to do this with open files results in the descriptors being pointed at deleted files and the space for those deleted files not being reclaimed.

So, architecturally, Splunk puts hot and warm in the same filesystem and you cannot separate them.

View solution in original post

PPape
Contributor

Thanks for this detailed Answer 🙂

0 Karma

jeffland
Champion

I don't think it's possible since the documentation always mentions the hot and warm buckets in some kind of union. It's not the same, but you could simply reduce the number of warm buckets to a lower number (default is 300) so that data starts leaving your hot/warm SSD sooner (which I think should have the same effect).
See http://docs.splunk.com/Documentation/Splunk/6.2.2/Indexer/Configureindexstorage

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!