Determining indexes.conf settings for all indexes combined

[teedilo]

I've spent hours studying the documentation and articles outside of splunkbase about configuring indexing, and I'm still confused, and our indexing isn't working as expected. This shouldn't be that difficult. Hot + warm + cold usage is way beyond what I have configured for maxTotalDataSizeMB for the main index, but the volume for hot + warm is only at about 69% utilization, whereas the volume for cold is at 100%. Why did the cold volume fill up? I.e., why isn't cold going to frozen soon enough?

I'm thinking now that it might be that I have only taken the main index into account. That's probably because it seems that most of the documentation and articles just talk about maxTotalDataSizeMB and other indexes.conf settings in reference to main. We have maxTotalDataSizeMB set to 160000, which is sufficiently low for main (hot+warm volume size is 100000, cold volume size is 100000). However, maxTotalDataSizeMB is set to the default value of 500000 for the other indexes (history, summary, etc.), which is way beyond the size of the two volumes combined. Don't I need to take those into account as well? That is, don't I need to keep the total of the maxTotalDataSizeMB values for all indexes below our total volume size for this to work? The splunk documentation isn't at all clear about this.

I may file a support case for this, but I figured I'd try my luck on the forum first.

[teedilo]

I think I'll try using maxVolumeDataSizeMB and the other volume related settings to achieve better control, and possibly the homePath.maxDataSizeMB and coldPath.maxDataSizeMB settings as well. For some reason the person who originally configured our Splunk installations didn't use any of these settings -- maybe they weren't available when our installations were first configured.

In any case, it's stlil a mystery to me why the maxTotalDataSizeMB value for our main index has been exceeded by quite a bit. The setting is 160000 (MB). There's about 69000 MB of hot + warm, and 100000 MB of cold. Our cold volume size is 100000, so it's completely filled. Even if you don't consider the values for the other indexes, it doesn't seem that this should ever happen. If anyone has any ideas how this may be happening, I'd still like to know.

(In my humble opinion, Splunk should be able to handle all of this automatically. There should be no need to set any of this stuff. Splunk should be able to see when it's running out of volume space and just do the right thing.)

I included the contents of our current indexes.conf file below. Also, the value of minFreeSpace in our server.conf file is 1024. Also, as explained earlier, the size of the hot+warm volume (in its default location) is 100 GB, and the size of the cold volume (in /var/lib/splunk/audit/colddb) is also 100 GB. The only thing I see in the cold location is buckets for main (in /var/lib/splunk/defaultdb/colddb). There's nothing anywhere else that could be chewing up space in cold. But that's beside the point -- Splunk shouldn't have allowed the total hot+warm+cold to reach 169000 (MB) when maxTotalDataSizeMB for main is 160000.

[default]

[_audit]
coldPath = /var/lib/splunk/audit/colddb

[_blocksignature]
coldPath = /var/lib/splunk/blockSignature/colddb

[_internal]
coldPath = /var/lib/splunk/_internaldb/colddb

[_thefishbucket]
coldPath = /var/lib/splunk/fishbucket/colddb

[history]
coldPath = /var/lib/splunk/historydb/colddb

[main]
maxHotBuckets = 5
maxDataSize = 1024
maxHotIdleSecs = 0
coldPath = /var/lib/splunk/defaultdb/colddb
maxTotalDataSizeMB = 160000
maxWarmDBCount = 76

[summary]
coldPath = /var/lib/splunk/summarydb/colddb
<<<<<<

[MuS · Accepted solution]

Hi teedilo,

I will answer the question you posted as topic, even you don't referre to it later.

Do get all configs for all indexes run this search in the UI:

| rest /services/configs/conf-indexes

This will give you table of all the settings for all indexes.

Regarding your other questions; this mainly up to you to get it right...but take a good read here http://wiki.splunk.com/Deploy:BucketRotationAndRetention and the dice on indexes.conf and the option frozenTimePeriodeInSecs

Hope this helps ...

cheers, MuS

[teedilo]

Thanks much for your response, MuS! I was about to delete my question at least until I could research it further, but once you had responded it was too late. 🙂 Anyway, good tip on the rest query -- thanks for that. Wondering whether you know of another query that gives the actual size that Splunk thinks each index is currently occupying? I don't see that in the results returned by that rest query.

I've studied the documentation at both of the links you mention. However, neither article really makes it clear how you need to account for the total space occupied by all indexes combined. I realize now that that probably shoud be intuitive, but it's just not spelled out very well in this documentation.

I also realize that a point made in my original question was flawed. While it's true that there's 69000 (MB) being occupied in my hot+warm volume, that's the same volume where all of my indexes have their hot+warm data (history, summary, etc.), and I haven't attempted to determine exactly how much of that 69000 can be attributed to the hot+warm for my main index. So maybe Splunk is behaving as expected after all if I determine that only ~60000 MB is being occupied by the hot+warm for main, and the other 9000 MB being occupied by the hot+ warm for the other indexes.

That's where it would be handy if you knew of a query that would give me the actual size that Splunk thinks each index is occupying.

Thanks again for your help.

[teedilo]

I think I answered my own question about a query to get current index sizes. The currentDBSizeMB values in the query below (which I found in an answer to another question on this forum) should give me what I need. Just from an quick analysis I get the sense that Splunk is respecting the 160000 value for maxTotalDataSizeMB for the main index, and that the other space being occupied is from the other indexes. Mystery solved, so I know where to go from here. Thanks again for your help, MuS.

| rest /services/data/indexes

[martin_mueller]

You could just open the 6.3 Distributed Management Console and use those pre-built dashboards...

[teedilo]

Thanks for the tip, Martin. Sounds like something for us to look forward to, but we haven't had time to upgrade Splunk in a while.