Deployment Architecture

Max warm settings exceeded, but cold is still empty

oliverj
Communicator

I am looking through my indexes, and I see that my busiest one is not responding at all how I thought I had it configured.

I am hoping I have some sort of settings precedence overriding the behavior I expected....

indexes.conf

#Unlimited storage overall
maxTotalDataSizeMB = 1000000000
#Once my hot/warm index reaches 500GB, send them off to cold
homePath.maxDataSizeMB = 500000
#Purge data older than 5.1 years
frozenTimePeriodInSecs=160833600

[volume:hot]
path = E:/splunk-hot

[volume:cold]
path = F:/splunk-hot

[busyIndex]
repFactor = auto
homePath = volume:hot/busyIndex/db
coldPath = volume:cold/busyIndex/colddb

The problem:
Looking at my IndexDetail page from the splunk monitoring console, I see that:
Warm Index Size = 552GB -- Why did it not start rolling already? It has exceeded the maxDataSizeMb
Cold Index Size = 0
Total buckets: 1747 (Max buckets is 300, per this same page) -- Why did it not start rolling already?
Cold Path -- I have checked, and it seems fine. The dummy folders have been created by splunk so It has permissions. Per "Index Detail" page, maxColdDb is 0 (for unlimited!)

The settings from my indexes.conf are reflected properly in this "Index Detail" screen, so I assume my indexes.conf has valid stanzas.

Second question.....
My goal:
For each index, store 500GB of data on hot storage before pushing off to cold, where it will sit. Overall data will be purged after 5.1 years.
I think my settings are not at all in line with this though. If my max bucket size is not configured, it would default to "auto" (750MB), meaning no matter how high I set my homePath.maxDataSizeMB to, it can never exceed ~230GB.
So, I need to:

  1. Change my max bucket count to 675 (leaving bucketSize at auto 750)
  2. Change my homePath.maxDataSizeMB to something much larger, because it applies to all indexes as a group, not a single index

Correct?

0 Karma
1 Solution

oliverj
Communicator

I gave up and contacted support I just couldn't figure it out, especially because the same configs were working elsewhere. Turns out, I did not have "system" permissions on the folder.
Splunk was able to create the folder tree in the directory with no premissions, but was unable to put any actual files (buckets) in.
The moment I changed permissions, the folders started updating, buckets started rolling, etc. All is well now.
Note: the splunkd log DID have errors about inflight data and permission errors. I just missed them, unfortunately.

View solution in original post

0 Karma

oliverj
Communicator

I gave up and contacted support I just couldn't figure it out, especially because the same configs were working elsewhere. Turns out, I did not have "system" permissions on the folder.
Splunk was able to create the folder tree in the directory with no premissions, but was unable to put any actual files (buckets) in.
The moment I changed permissions, the folders started updating, buckets started rolling, etc. All is well now.
Note: the splunkd log DID have errors about inflight data and permission errors. I just missed them, unfortunately.

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...