Monitoring Splunk

High disk space utilization on indexer

Reethika
Path Finder

Hi.

 I have disk space issue with indexer. where there is 92% utilization in opt/splunkdata dir.  and most space consuming files in this directory are db files, such as "_internal_db" and some other temp folders, which also contain dbs. I'm not sure which of them to clear. Almost all files in directory are db. 

could please suggest want kind of data can deleted to free some space without loosing important data. 

Thanks in advance. 

Labels (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust
Everything in /opt/splunkdata is important data. Don't touch any of it.
Either add storage to the indexer or reduce the amount of data you retain in your indexes.
---
If this reply helps you, Karma would be appreciated.

Reethika
Path Finder

Thankyou @richgalloway .

/opt/splunkdata have "temp" directory, which consumes most data. cleaning this directory is suggested?

0 Karma

richgalloway
SplunkTrust
SplunkTrust
I don't recall ever seeing a 'temp' directory in $SPLUNK_DB. What's in it?
---
If this reply helps you, Karma would be appreciated.
0 Karma

Reethika
Path Finder

Hi @richgalloway @gcusello ,

So In my case, I have reduced retention period from 1 year to 3 months for an index. And after restarting splunk, its still the same. and after a day the utilization have increased.

In my scenario, /opt/splunkdata/temp/                      filepath,

  • db
  • datamodel
  • summary

 are present in /temp.

Thanks.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Reethika ,

temp seems to be an index, do you see it in the indexes.conf or in web interface?

If it's an index, see if you can reduce retention on this index.

If it isn't an index, see which data go in it, maybe there's a script or other.

Ciao.

Giuseppe

0 Karma

Reethika
Path Finder

@gcusello@richgalloway 

"temp "its an index, can't find it on web interface though. 

cat /opt/splunk/etc/apps/Axxxxxxxxxxxxxxxxxxxx/default/indexes.conf
[_internal]
maxTotalDataSizeMB = 70000
homePath.maxDataSizeMB = 10000
homePath = $SPLUNK_DB/_internaldb/db
coldPath.maxDataSizeMB = 60000
coldPath = $SPLUNK_DB/_internaldb/colddb
thawedPath = $SPLUNK_DB/_internaldb/thaweddb
frozenTimePeriodInSecs = 7776000

 

These are parameters used, and restarted. but didn't work.

earlier frozenTimePeriodInSecs was about an year.

coldPath.maxDataSizeMBfrozenTimePeriodInSecs

maxDataSizeMB rules over frozenTimePeriodInSecs ? 

Reducing coldPath.maxDataSizeMB can help?

Thanks. 

 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

The temp index may be defined in a different indexes.conf file.  Try this command to find it.

splunk btool --debug indexes list temp

Or run this search from the GUI

| rest /services/data/indexes | dedup title | table title

 

---
If this reply helps you, Karma would be appreciated.

gcusello
SplunkTrust
SplunkTrust

Hi @Reethika ,

enlarge the storage (as suggested by @richgalloway ) is always the best solution.

If you cannot do this, you could also reduce the disk occupation of _internal data reducing the retention on this Index: instead of one month set e.g. 15 days:

  • open indexes.conf in $SPLUNK_HOME/etc/system/local, if you haven't it, create it and copy the _internal stanza from the default folder.
  • modify the parameter FrozenTimePeriodInSecs  = 1296000,
  • restart Splunk.

In this way the disk occupation of this index will be reduced.

Ciao.

Giuseppe

0 Karma

Reethika
Path Finder

Thanks @gcusello .

As suggested, data retention period is reduced for internal index.

But the utilization is same.

New  FrozenTimePeriodInSecs  parameter is applicable only  future to be indexed data. And old index data would be same.

Please can you clear this out. 

 

0 Karma

gcusello
SplunkTrust
SplunkTrust

hi @Reethika ,

retention is appliad on the full index, so if you reduce the retention of an index from 30 to 15 days, also the space on disk will be reduces, the question is: before retention reduction, had you events older than 15 days?

if yes, they will be deleted, if not obviously there wasn't any reduction.

In addition, remember that events deletion in Splunk is made at bucket level, in other words, events are stored in buckets, when the earliest event of a bucket exceed the retention period, all the bucket will be deleted, for this reason you could have events older than the retention period.

Anyway, check the disk occupation after few minutes and, if you had many events older than the retention period, the free disk space will be more than before.

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...