Deployment Architecture

Is there a way to separate the hot and warm bucket path?

Contributor

Hi,

i found this in the Docs: Use multiple partitions for index data
But there i can only configure

  • homePath = ... This is the path that contains the hot and warm databases for the index. Caution: The path must be writable.
  • coldPath = ... This is the path that contains the cold databases for the index. Caution: The path must be writable. Is there a way to separate the hot and the warm Path? I have a system with SSD, SAS15k, and SAS10k, and I only want the hot Bucket on the SSD.

Thanks for your help!

Tags (4)
1 Solution

SplunkTrust
SplunkTrust

No it's not possible to separate hot and warm.

The fundamental difference between a hot bucket and a warm bucket is "am I writing to it?" When a bucket is hot, there are open file descriptors into the files comprising the bucket and the files are being written to. When searches launch, the search process forked inherits those open file descriptors.

When a bucket is warm, the file descriptors remain open just as in hot - but there is no writing.

The open descriptors are why hot and warm must coexist on the same filesystem ; you cannot move a file from one filesystem to another and maintain an open descriptor against it. A cross-filesystem move is really a "copy and delete" so trying to do this with open files results in the descriptors being pointed at deleted files and the space for those deleted files not being reclaimed.

So, architecturally, Splunk puts hot and warm in the same filesystem and you cannot separate them.

View solution in original post

SplunkTrust
SplunkTrust

No it's not possible to separate hot and warm.

The fundamental difference between a hot bucket and a warm bucket is "am I writing to it?" When a bucket is hot, there are open file descriptors into the files comprising the bucket and the files are being written to. When searches launch, the search process forked inherits those open file descriptors.

When a bucket is warm, the file descriptors remain open just as in hot - but there is no writing.

The open descriptors are why hot and warm must coexist on the same filesystem ; you cannot move a file from one filesystem to another and maintain an open descriptor against it. A cross-filesystem move is really a "copy and delete" so trying to do this with open files results in the descriptors being pointed at deleted files and the space for those deleted files not being reclaimed.

So, architecturally, Splunk puts hot and warm in the same filesystem and you cannot separate them.

View solution in original post

Contributor

Thanks for this detailed Answer 🙂

0 Karma

Champion

I don't think it's possible since the documentation always mentions the hot and warm buckets in some kind of union. It's not the same, but you could simply reduce the number of warm buckets to a lower number (default is 300) so that data starts leaving your hot/warm SSD sooner (which I think should have the same effect).
See http://docs.splunk.com/Documentation/Splunk/6.2.2/Indexer/Configureindexstorage

0 Karma