topic Re: High disk space utilization on indexer in Monitoring Splunk

High disk space utilization on indexer

Reethika — Thu, 11 Jun 2020 13:35:56 GMT

Hi.

I have disk space issue with indexer. where there is 92% utilization in opt/splunkdata dir. and most space consuming files in this directory are db files, such as "_internal_db" and some other temp folders, which also contain dbs. I'm not sure which of them to clear. Almost all files in directory are db.

could please suggest want kind of data can deleted to free some space without loosing important data.

Thanks in advance.

Re: High disk space utilization on indexer

richgalloway — Thu, 11 Jun 2020 14:23:22 GMT

Everything in /opt/splunkdata is important data. Don't touch any of it.
Either add storage to the indexer or reduce the amount of data you retain in your indexes.

Re: High disk space utilization on indexer

Reethika — Fri, 12 Jun 2020 15:35:17 GMT

Thankyou @richgalloway .

/opt/splunkdata have "temp" directory, which consumes most data. cleaning this directory is suggested?

Re: High disk space utilization on indexer

gcusello — Fri, 12 Jun 2020 16:05:43 GMT

Hi @Reethika ,

enlarge the storage (as suggested by @richgalloway ) is always the best solution.

If you cannot do this, you could also reduce the disk occupation of _internal data reducing the retention on this Index: instead of one month set e.g. 15 days:

open indexes.conf in $SPLUNK_HOME/etc/system/local, if you haven't it, create it and copy the _internal stanza from the default folder.
modify the parameter FrozenTimePeriodInSecs = 1296000,
restart Splunk.

In this way the disk occupation of this index will be reduced.

Ciao.

Giuseppe

Re: High disk space utilization on indexer

Reethika — Sat, 13 Jun 2020 04:10:44 GMT

Thanks @gcusello .

As suggested, data retention period is reduced for internal index.

But the utilization is same.

New FrozenTimePeriodInSecs parameter is applicable only future to be indexed data. And old index data would be same.

Please can you clear this out.

Re: High disk space utilization on indexer

gcusello — Sat, 13 Jun 2020 10:27:17 GMT

hi @Reethika ,

retention is appliad on the full index, so if you reduce the retention of an index from 30 to 15 days, also the space on disk will be reduces, the question is: before retention reduction, had you events older than 15 days?

if yes, they will be deleted, if not obviously there wasn't any reduction.

In addition, remember that events deletion in Splunk is made at bucket level, in other words, events are stored in buckets, when the earliest event of a bucket exceed the retention period, all the bucket will be deleted, for this reason you could have events older than the retention period.

Anyway, check the disk occupation after few minutes and, if you had many events older than the retention period, the free disk space will be more than before.

Ciao.

Giuseppe

Re: High disk space utilization on indexer

richgalloway — Mon, 15 Jun 2020 12:45:21 GMT

I don't recall ever seeing a 'temp' directory in $SPLUNK_DB. What's in it?

Re: High disk space utilization on indexer

Reethika — Wed, 17 Jun 2020 12:26:54 GMT

Hi @richgalloway @gcusello ,

So In my case, I have reduced retention period from 1 year to 3 months for an index. And after restarting splunk, its still the same. and after a day the utilization have increased.

In my scenario, /opt/splunkdata/temp/ filepath,

db
datamodel
summary

are present in /temp.

Thanks.

Re: High disk space utilization on indexer

gcusello — Wed, 17 Jun 2020 12:35:13 GMT

Hi @Reethika ,

temp seems to be an index, do you see it in the indexes.conf or in web interface?

If it's an index, see if you can reduce retention on this index.

If it isn't an index, see which data go in it, maybe there's a script or other.

Ciao.

Giuseppe

Re: High disk space utilization on indexer

Reethika — Wed, 01 Jul 2020 15:21:36 GMT

@gcusello, @richgalloway

"temp "its an index, can't find it on web interface though.

cat /opt/splunk/etc/apps/Axxxxxxxxxxxxxxxxxxxx/default/indexes.conf
[_internal]
maxTotalDataSizeMB = 70000
homePath.maxDataSizeMB = 10000
homePath = $SPLUNK_DB/_internaldb/db
coldPath.maxDataSizeMB = 60000
coldPath = $SPLUNK_DB/_internaldb/colddb
thawedPath = $SPLUNK_DB/_internaldb/thaweddb
frozenTimePeriodInSecs = 7776000

These are parameters used, and restarted. but didn't work.

earlier frozenTimePeriodInSecs was about an year.

coldPath.maxDataSizeMB > frozenTimePeriodInSecs ?

maxDataSizeMB rules over frozenTimePeriodInSecs ?

Reducing coldPath.maxDataSizeMB can help?

Thanks.

Re: High disk space utilization on indexer

richgalloway — Wed, 01 Jul 2020 16:40:20 GMT

The temp index may be defined in a different indexes.conf file. Try this command to find it.

splunk btool --debug indexes list temp

Or run this search from the GUI

| rest /services/data/indexes | dedup title | table title