tsidxstats? What is this?

[pvols1979]

Our search heads are filling up with tsidx files in the /var/run/splunk/dispatch/tsidxstats directory, but I am not able to find any documentation that explains what these files are. We suspect that they are search artifacts/results, but could this be summarized data?

I know that the location of these files can be changed in the indexes.conf, but I am unsure what they are and large they can be. We have seen as much as 600GB on one search head. I can resize the space we have alloted for our search head, but I have no idea how big it needs to be.

[rroberts]

TSIDX namespace-

I contributed to a namespace by running index=_* | fields action | tscollect namespace=myaction.
This created a myaction folder with a time series file under $SPLUNK_HOME/var/lib/splunk/tsidxstats.
My dashboard runs searches against this namespace by running | tstats count from myaction groupby action.

Report Acceleration-

The report acceleration summary is actually a tsidx file created with and rolls with the buckets. ie $SPLUNK_HOME/var/lib/splunk/defaultdb/hot_v1_1

[mcronkrite]

Ah the reason is because you are using a wild card in your index=_* so it's saving the tsidx locally in var/lib so you probably want to change this around and use the Splunk_Internal Messages Data Model for your dashboard queries. That has a lot of the _internal info you maybe looking for.

[tkiss]

Hi,

Any idea how to rotate or manage these files? They're filling up seach heads. Do they expire and can it be set somewhere?

T

[bandit]

Looks like mine are coming from the NetApp Ontap app and are being stored on the search head with the app.

2.1G /local/splunk/var/lib/splunk/tsidxstats/netapp_perf_aggr
53G /local/splunk/var/lib/splunk/tsidxstats/netapp_perf_disk
14G /local/splunk/var/lib/splunk/tsidxstats/netapp_perf_lun
9.7G /local/splunk/var/lib/splunk/tsidxstats/netapp_perf_volume

[lguinn2]

Check out the Manage Report Acceleration documentation (http://docs.splunk.com/Documentation/Splunk/5.0.2/Knowledge/Manageacceleratedsearchsummaries) and this page on Setting the Summary Time Range
(http://docs.splunk.com/Documentation/Splunk/5.0.2/Knowledge/Manageacceleratedsearchsummaries#Set_rep...)

When someone sets up a summary, they also set a time range (7 days, 30 days, etc) for which the acceleration summary will be kept. You can reduce this range to reduce the size of the summary - but you need to do it for each report that uses the summary.

[LukeMurphey]

Are you running Enterprise Security? I ask because Enterprise Security has a system built-in to limit the size of the files based on a retention policy. You can modify the retention policy to reduce the size.

[LukeMurphey]

TSIDX is similar to summary indexing that allows dramatically improved performance. It is used in several applications such as Enterprise Security (version 2.4 and later). This feature was first available in Splunk 5.0.

[okrabbe_splunk]

Are you running enetprise security, PCI, or one of the newer releases of our apps?

Certain apps are now using TSIDX stats to offer better search acceleration than is possible using either summary indexing or report acceleration.

One thing you might want to do is search through your schedule searches and look for tscollect. This is what is populating that directory.
http://docs.splunk.com/Documentation/Splunk/5.0.2/SearchReference/Tscollect

[lguinn2]

I believe that these are the files for Report Acceleration Summaries. You should be able to manage them by going to Manager » Report Acceleration Summaries. If there are some that have never been used, you can just delete them.
This will turn off Report Acceleration in the associated reports, so the acceleration summaries will not be re-created.

Here is some of the documentation on Report Acceleration

[sowings]

No, report acceleration data lives within the $SPLUNK_DB hierarchy, alongside the indexes themselves.

[lguinn2]

I suggest that you test by creating some acceleration summaries for youself. At this time, acceleration summaries live on the search head (sadly) as txidx files. You are right that this is not the only way to get tsidx files though.

[okrabbe_splunk]

If these are on the SH I don't think they are report acceleration summaries.

See Luke Murphey's answer..