topic Re: tsidxstats? What is this? in Knowledge Management

tsidxstats? What is this?

pvols1979 — Tue, 28 May 2013 17:09:50 GMT

Our search heads are filling up with tsidx files in the /var/run/splunk/dispatch/tsidxstats directory, but I am not able to find any documentation that explains what these files are. We suspect that they are search artifacts/results, but could this be summarized data?

I know that the location of these files can be changed in the indexes.conf, but I am unsure what they are and large they can be. We have seen as much as 600GB on one search head. I can resize the space we have alloted for our search head, but I have no idea how big it needs to be.

Re: tsidxstats? What is this?

lguinn2 — Tue, 28 May 2013 19:43:19 GMT

I believe that these are the files for Report Acceleration Summaries. You should be able to manage them by going to Manager » Report Acceleration Summaries. If there are some that have never been used, you can just delete them.
This will turn off Report Acceleration in the associated reports, so the acceleration summaries will not be re-created.

Here is some of the documentation on Report Acceleration

Re: tsidxstats? What is this?

okrabbe_splunk — Tue, 28 May 2013 20:10:17 GMT

Are you running enetprise security, PCI, or one of the newer releases of our apps?

Certain apps are now using TSIDX stats to offer better search acceleration than is possible using either summary indexing or report acceleration.

One thing you might want to do is search through your schedule searches and look for tscollect. This is what is populating that directory.
http://docs.splunk.com/Documentation/Splunk/5.0.2/SearchReference/Tscollect

Re: tsidxstats? What is this?

LukeMurphey — Tue, 28 May 2013 21:04:00 GMT

TSIDX is similar to summary indexing that allows dramatically improved performance. It is used in several applications such as Enterprise Security (version 2.4 and later). This feature was first available in Splunk 5.0.

Re: tsidxstats? What is this?

tkiss — Fri, 31 May 2013 09:38:22 GMT

Hi,

Any idea how to rotate or manage these files? They're filling up seach heads. Do they expire and can it be set somewhere?

Re: tsidxstats? What is this?

LukeMurphey — Fri, 31 May 2013 16:13:42 GMT

Are you running Enterprise Security? I ask because Enterprise Security has a system built-in to limit the size of the files based on a retention policy. You can modify the retention policy to reduce the size.

Re: tsidxstats? What is this?

lguinn2 — Sat, 01 Jun 2013 02:39:06 GMT

Check out the Manage Report Acceleration documentation (http://docs.splunk.com/Documentation/Splunk/5.0.2/Knowledge/Manageacceleratedsearchsummaries) and this page on Setting the Summary Time Range
(http://docs.splunk.com/Documentation/Splunk/5.0.2/Knowledge/Manageacceleratedsearchsummaries#Set_report_acceleration_summary_time_ranges)

When someone sets up a summary, they also set a time range (7 days, 30 days, etc) for which the acceleration summary will be kept. You can reduce this range to reduce the size of the summary - but you need to do it for each report that uses the summary.

Re: tsidxstats? What is this?

okrabbe_splunk — Sun, 02 Jun 2013 22:05:35 GMT

If these are on the SH I don't think they are report acceleration summaries.

See Luke Murphey's answer..

Re: tsidxstats? What is this?

lguinn2 — Sat, 08 Jun 2013 18:55:09 GMT

I suggest that you test by creating some acceleration summaries for youself. At this time, acceleration summaries live on the search head (sadly) as txidx files. You are right that this is not the only way to get tsidx files though.

Re: tsidxstats? What is this?

rroberts — Mon, 28 Sep 2020 14:47:48 GMT

TSIDX namespace-

I contributed to a namespace by running index=_* | fields action | tscollect namespace=myaction.
This created a myaction folder with a time series file under $SPLUNK_HOME/var/lib/splunk/tsidxstats.
My dashboard runs searches against this namespace by running | tstats count from myaction groupby action.

Report Acceleration-

The report acceleration summary is actually a tsidx file created with and rolls with the buckets. ie $SPLUNK_HOME/var/lib/splunk/defaultdb/hot_v1_1

Re: tsidxstats? What is this?

sowings — Tue, 17 Sep 2013 19:36:48 GMT

No, report acceleration data lives within the $SPLUNK_DB hierarchy, alongside the indexes themselves.

Re: tsidxstats? What is this?

mcronkrite — Mon, 28 Sep 2020 18:45:31 GMT

Ah the reason is because you are using a wild card in your index=_* so it's saving the tsidx locally in var/lib so you probably want to change this around and use the Splunk_Internal Messages Data Model for your dashboard queries. That has a lot of the _internal info you maybe looking for.

Re: tsidxstats? What is this?

bandit — Tue, 29 Sep 2020 07:20:23 GMT

Looks like mine are coming from the NetApp Ontap app and are being stored on the search head with the app.

2.1G /local/splunk/var/lib/splunk/tsidxstats/netapp_perf_aggr
53G /local/splunk/var/lib/splunk/tsidxstats/netapp_perf_disk
14G /local/splunk/var/lib/splunk/tsidxstats/netapp_perf_lun
9.7G /local/splunk/var/lib/splunk/tsidxstats/netapp_perf_volume