I am looking through my indexes, and I see that my busiest one is not responding at all how I thought I had it configured.
#Unlimited storage overall
maxTotalDataSizeMB = 1000000000
#Once my hot/warm index reaches 500GB, send them off to cold
homePath.maxDataSizeMB = 500000
#Purge data older than 5.1 years
frozenTimePeriodInSecs=160833600
[volume:hot]
path = E:/splunk-hot
[volume:cold]
path = F:/splunk-hot
[busyIndex]
repFactor = auto
homePath = volume:hot/busyIndex/db
coldPath = volume:cold/busyIndex/colddb
The problem:
Looking at my IndexDetail page from the splunk monitoring console, I see that:
Warm Index Size = 552GB -- Why did it not start rolling already? It has exceeded the maxDataSizeMb
Cold Index Size = 0
Total buckets: 1747 (Max buckets is 300, per this same page) -- Why did it not start rolling already?
Cold Path -- I have checked, and it seems fine. The dummy folders have been created by splunk so It has permissions. Per "Index Detail" page, maxColdDb is 0 (for unlimited!)
The settings from my indexes.conf are reflected properly in this "Index Detail" screen, so I assume my indexes.conf has valid stanzas.
Second question.....
My goal:
For each index, store 500GB of data on hot storage before pushing off to cold, where it will sit. Overall data will be purged after 5.1 years.
I think my settings are not at all in line with this though. If my max bucket size is not configured, it would default to "auto" (750MB), meaning no matter how high I set my homePath.maxDataSizeMB to, it can never exceed ~230GB.
So, I need to:
Correct?
I gave up and contacted support I just couldn't figure it out, especially because the same configs were working elsewhere. Turns out, I did not have "system" permissions on the folder.
Splunk was able to create the folder tree in the directory with no premissions, but was unable to put any actual files (buckets) in.
The moment I changed permissions, the folders started updating, buckets started rolling, etc. All is well now.
Note: the splunkd log DID have errors about inflight data and permission errors. I just missed them, unfortunately.
I gave up and contacted support I just couldn't figure it out, especially because the same configs were working elsewhere. Turns out, I did not have "system" permissions on the folder.
Splunk was able to create the folder tree in the directory with no premissions, but was unable to put any actual files (buckets) in.
The moment I changed permissions, the folders started updating, buckets started rolling, etc. All is well now.
Note: the splunkd log DID have errors about inflight data and permission errors. I just missed them, unfortunately.