Deployment Architecture

Max warm settings exceeded, but cold is still empty

oliverj
Communicator

I am looking through my indexes, and I see that my busiest one is not responding at all how I thought I had it configured.

I am hoping I have some sort of settings precedence overriding the behavior I expected....

indexes.conf

#Unlimited storage overall
maxTotalDataSizeMB = 1000000000
#Once my hot/warm index reaches 500GB, send them off to cold
homePath.maxDataSizeMB = 500000
#Purge data older than 5.1 years
frozenTimePeriodInSecs=160833600

[volume:hot]
path = E:/splunk-hot

[volume:cold]
path = F:/splunk-hot

[busyIndex]
repFactor = auto
homePath = volume:hot/busyIndex/db
coldPath = volume:cold/busyIndex/colddb

The problem:
Looking at my IndexDetail page from the splunk monitoring console, I see that:
Warm Index Size = 552GB -- Why did it not start rolling already? It has exceeded the maxDataSizeMb
Cold Index Size = 0
Total buckets: 1747 (Max buckets is 300, per this same page) -- Why did it not start rolling already?
Cold Path -- I have checked, and it seems fine. The dummy folders have been created by splunk so It has permissions. Per "Index Detail" page, maxColdDb is 0 (for unlimited!)

The settings from my indexes.conf are reflected properly in this "Index Detail" screen, so I assume my indexes.conf has valid stanzas.

Second question.....
My goal:
For each index, store 500GB of data on hot storage before pushing off to cold, where it will sit. Overall data will be purged after 5.1 years.
I think my settings are not at all in line with this though. If my max bucket size is not configured, it would default to "auto" (750MB), meaning no matter how high I set my homePath.maxDataSizeMB to, it can never exceed ~230GB.
So, I need to:

  1. Change my max bucket count to 675 (leaving bucketSize at auto 750)
  2. Change my homePath.maxDataSizeMB to something much larger, because it applies to all indexes as a group, not a single index

Correct?

0 Karma
1 Solution

oliverj
Communicator

I gave up and contacted support I just couldn't figure it out, especially because the same configs were working elsewhere. Turns out, I did not have "system" permissions on the folder.
Splunk was able to create the folder tree in the directory with no premissions, but was unable to put any actual files (buckets) in.
The moment I changed permissions, the folders started updating, buckets started rolling, etc. All is well now.
Note: the splunkd log DID have errors about inflight data and permission errors. I just missed them, unfortunately.

View solution in original post

0 Karma

oliverj
Communicator

I gave up and contacted support I just couldn't figure it out, especially because the same configs were working elsewhere. Turns out, I did not have "system" permissions on the folder.
Splunk was able to create the folder tree in the directory with no premissions, but was unable to put any actual files (buckets) in.
The moment I changed permissions, the folders started updating, buckets started rolling, etc. All is well now.
Note: the splunkd log DID have errors about inflight data and permission errors. I just missed them, unfortunately.

0 Karma
Get Updates on the Splunk Community!

Get the T-shirt to Prove You Survived Splunk University Bootcamp

As if Splunk University, in Las Vegas, in-person, with three days of bootcamps and labs weren’t enough, now ...

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...

Wondering How to Build Resiliency in the Cloud?

IT leaders are choosing Splunk Cloud as an ideal cloud transformation platform to drive business resilience,  ...