Index over-consumption of Disk

[GaryChan1]

Hi all,

I noticed my Splunk instance wasn't indexing data this afternoon. I looked at the server and one of the disks that hosts some of my indexes was full.

I looked at the individual size of each index on disk and two of them are consuming disk space far in excess of the limits that I have set on the index properties.

Index : wineventlog
Max size: 200 GB
Max bucket size: 10000 MB
Current Size: 199.25 GB
Size of index on disk: 430 GB

Index : windows
Max size: 200 GB
Max bucket size: 10000 MB
Current Size: 75.65 GB
Size of index on disk: 231 GB

As a temporary fix I've increased the size of the VHD (the instance is virtualized) but ideally I'd like to reduce the size of the data on disk. Any pointers on how I should tackle this?

Thanks
Gary

[woodcock]

On any indexer, use this command to get a grip on what settings are in effect:

${SPLUNK_HOME}/etc/bin/splunk btool indexes list --debug

Then xref against the docs and set the options correctly as necessary:
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Indexesconf

[adonio · Accepted solution]

your max size is the default 500gb ... as its unspecified in your indexes.conf

you should fix it and splunk will happily comply to your size and retention wishes

[GaryChan1]

Thank you. It took a little while but I've reclaimed some of my disk space now.

Gary

[broberg]

Can you show the indexes.conf config and how you set the max index size?
Are you sure it is not bucketsize you have set?
Do you have warm and cold on the same partition?

[GaryChan1]

Hi broberg,

I think wineventlog and windows are the default indexes created by the Splunk_TA_Windows addon. The indexes.conf only contains the following:

[windows]
homePath = $SPLUNK_DB/windows/db
coldPath = $SPLUNK_DB/windows/colddb
thawedPath = $SPLUNK_DB/windows/thaweddb

[wineventlog]
homePath = $SPLUNK_DB/wineventlog/db
coldPath = $SPLUNK_DB/wineventlog/colddb
thawedPath = $SPLUNK_DB/wineventlog/thaweddb

[perfmon]
homePath = $SPLUNK_DB/perfmon/db
coldPath = $SPLUNK_DB/perfmon/colddb
thawedPath = $SPLUNK_DB/perfmon/thaweddb

Via settings > indexes, I have the following configured:

windows index: https://imgur.com/NqaO8vr
wineventlog index: https://imgur.com/ZuKj6aZ