How can I limit the size of tsidx for VMWare or NetApp on 6.x search head?

[rphillips_splk]

Description: Running VMWare App on Splunk 6.x and tsidx files have filled up my search head to 100% disk capacity. How can I limit the size of tsidx files on my search head? Summary indexing is working as expected- and located on my indexers as per the docs.

[weeb · Accepted solution]

For context, this answer refers to VMWare App 3.1.3 on Splunk 6.x.

In the Splunk App for VMware tsidx files are stored in namespaces on the search head. You can manage the size of the tsidx namespace files using a retention policy. A retention policy specifies a limit for the size of the tsidx namespaces created and the length of time that namespaces are retained in your environment.

Background info on the recommendations below:

http://docs.splunk.com/Documentation/NetApp/2.0.1/DeployNetapp/Considerationswhenusingtsidxnamespace...

http://docs.splunk.com/Documentation/VMW/3.1.1/Configuration/Managenamespacefilesize

http://docs.splunk.com/Documentation/NetApp/2.0.2/DeployNetapp/Considerationswhenusingtsidxnamespace...

Adjust a maximum size limit or maximum retention time by copying /etc/apps/SA-Utils/default/tsidx_retention.conf to the local directory. The default retention time is 2,184 days, and the maximum size of an index defaults to 500G:

[default]
maxTotalDataSizeMB = 500000
retentionTimePeriodInSecs = 188697600

Copy and move to ../local/tsidx_retention.conf on the search head to enable changes.

The following changes adjust max size of the index to 50G, and changes the retention time to 7 days:

[default]
maxTotalDataSizeMB = 50000
retentionTimePeriodInSecs = 604800

The maximum size of all buckets for an index is maxTotalDataSizeMB.

Also, there is a scripted input that runs once a day at 3:00AM that cleans up the tsidx files. The input is defined in /etc/apps/SA-Utils/default/inputs.conf. The default is probably fine, but scheduling can be adjusted or the script can be run manually if desired.

[script://./bin/tsidx_clean_up.py]
disabled = false
passAuth = splunk-system-user
# Once per day at 3 AM
interval = * 3 * * *
index = _internal
sourcetype = tsidx:clean_up

[stevepraz]

Chiming in with a bit of information that stumped me for awhile. I couldn't understand why my tsidx_retention settings were being ignored. It turns out the cleanup script mentioned above was never running. Looks like this was due to my Search Head being a Windows server. In /etc/apps/SA-Utils/default there is also a inputs.conf.windows that has the correct configuration for the clean up script to run in Windows. You need to copy that into your inputs.conf and restart.

Then, after the scheduled time, you should see the job actually run. This should also create /var/log/splunk/tsidx_clean_up with details of which files are cleaned up when it runs. If you don't see the log file, you probably have the issue described above and the job has never run.