High disk space utilization on indexer

[Reethika]

Hi.

 I have disk space issue with indexer. where there is 92% utilization in opt/splunkdata dir.  and most space consuming files in this directory are db files, such as "_internal_db" and some other temp folders, which also contain dbs. I'm not sure which of them to clear. Almost all files in directory are db. 

could please suggest want kind of data can deleted to free some space without loosing important data. 

Thanks in advance. 

[richgalloway]

Everything in /opt/splunkdata is important data. Don't touch any of it.
Either add storage to the indexer or reduce the amount of data you retain in your indexes.

[Reethika]

Thankyou @richgalloway .

/opt/splunkdata have "temp" directory, which consumes most data. cleaning this directory is suggested?

[richgalloway]

I don't recall ever seeing a 'temp' directory in $SPLUNK_DB. What's in it?

[Reethika]

Hi @richgalloway @gcusello ,

So In my case, I have reduced retention period from 1 year to 3 months for an index. And after restarting splunk, its still the same. and after a day the utilization have increased.

In my scenario, /opt/splunkdata/temp/                      filepath,

• db
• datamodel
• summary

 are present in /temp.

Thanks.

[gcusello]

Hi @Reethika ,

temp seems to be an index, do you see it in the indexes.conf or in web interface?

If it's an index, see if you can reduce retention on this index.

If it isn't an index, see which data go in it, maybe there's a script or other.

Ciao.

Giuseppe

[Reethika]

@gcusello, @richgalloway 

"temp "its an index, can't find it on web interface though. 

cat /opt/splunk/etc/apps/Axxxxxxxxxxxxxxxxxxxx/default/indexes.conf
[_internal]
maxTotalDataSizeMB = 70000
homePath.maxDataSizeMB = 10000
homePath = $SPLUNK_DB/_internaldb/db
coldPath.maxDataSizeMB = 60000
coldPath = $SPLUNK_DB/_internaldb/colddb
thawedPath = $SPLUNK_DB/_internaldb/thaweddb
frozenTimePeriodInSecs = 7776000

These are parameters used, and restarted. but didn't work.

earlier frozenTimePeriodInSecs was about an year.

coldPath.maxDataSizeMB > frozenTimePeriodInSecs ? 

maxDataSizeMB rules over frozenTimePeriodInSecs ? 

Reducing coldPath.maxDataSizeMB can help?

Thanks. 

[richgalloway]

The temp index may be defined in a different indexes.conf file.  Try this command to find it.

splunk btool --debug indexes list temp

Or run this search from the GUI

| rest /services/data/indexes | dedup title | table title

[gcusello]

Hi @Reethika ,

enlarge the storage (as suggested by @richgalloway ) is always the best solution.

If you cannot do this, you could also reduce the disk occupation of _internal data reducing the retention on this Index: instead of one month set e.g. 15 days:

• open indexes.conf in $SPLUNK_HOME/etc/system/local, if you haven't it, create it and copy the _internal stanza from the default folder.
• modify the parameter FrozenTimePeriodInSecs  = 1296000,
• restart Splunk.

In this way the disk occupation of this index will be reduced.

Ciao.

Giuseppe

[Reethika]

Thanks @gcusello .

As suggested, data retention period is reduced for internal index.

But the utilization is same.

New  FrozenTimePeriodInSecs  parameter is applicable only  future to be indexed data. And old index data would be same.

Please can you clear this out. 

[gcusello]

hi @Reethika ,

retention is appliad on the full index, so if you reduce the retention of an index from 30 to 15 days, also the space on disk will be reduces, the question is: before retention reduction, had you events older than 15 days?

if yes, they will be deleted, if not obviously there wasn't any reduction.

In addition, remember that events deletion in Splunk is made at bucket level, in other words, events are stored in buckets, when the earliest event of a bucket exceed the retention period, all the bucket will be deleted, for this reason you could have events older than the retention period.

Anyway, check the disk occupation after few minutes and, if you had many events older than the retention period, the free disk space will be more than before.

Ciao.

Giuseppe