Knowledge Management

tsidxstats? What is this?

pvols1979
Explorer

Our search heads are filling up with tsidx files in the /var/run/splunk/dispatch/tsidxstats directory, but I am not able to find any documentation that explains what these files are. We suspect that they are search artifacts/results, but could this be summarized data?

I know that the location of these files can be changed in the indexes.conf, but I am unsure what they are and large they can be. We have seen as much as 600GB on one search head. I can resize the space we have alloted for our search head, but I have no idea how big it needs to be.

rroberts
Splunk Employee
Splunk Employee

TSIDX namespace-

I contributed to a namespace by running index=_* | fields action | tscollect namespace=myaction.
This created a myaction folder with a time series file under $SPLUNK_HOME/var/lib/splunk/tsidxstats.
My dashboard runs searches against this namespace by running | tstats count from myaction groupby action.

Report Acceleration-

The report acceleration summary is actually a tsidx file created with and rolls with the buckets. ie $SPLUNK_HOME/var/lib/splunk/defaultdb/hot_v1_1

mcronkrite
Splunk Employee
Splunk Employee

Ah the reason is because you are using a wild card in your index=_* so it's saving the tsidx locally in var/lib so you probably want to change this around and use the Splunk_Internal Messages Data Model for your dashboard queries. That has a lot of the _internal info you maybe looking for.

0 Karma

tkiss
Path Finder

Hi,

Any idea how to rotate or manage these files? They're filling up seach heads. Do they expire and can it be set somewhere?

T

0 Karma

bandit
Motivator

Looks like mine are coming from the NetApp Ontap app and are being stored on the search head with the app.

2.1G /local/splunk/var/lib/splunk/tsidxstats/netapp_perf_aggr
53G /local/splunk/var/lib/splunk/tsidxstats/netapp_perf_disk
14G /local/splunk/var/lib/splunk/tsidxstats/netapp_perf_lun
9.7G /local/splunk/var/lib/splunk/tsidxstats/netapp_perf_volume

0 Karma

lguinn2
Legend

Check out the Manage Report Acceleration documentation (http://docs.splunk.com/Documentation/Splunk/5.0.2/Knowledge/Manageacceleratedsearchsummaries) and this page on Setting the Summary Time Range
(http://docs.splunk.com/Documentation/Splunk/5.0.2/Knowledge/Manageacceleratedsearchsummaries#Set_rep...)

When someone sets up a summary, they also set a time range (7 days, 30 days, etc) for which the acceleration summary will be kept. You can reduce this range to reduce the size of the summary - but you need to do it for each report that uses the summary.

LukeMurphey
Champion

Are you running Enterprise Security? I ask because Enterprise Security has a system built-in to limit the size of the files based on a retention policy. You can modify the retention policy to reduce the size.

0 Karma

LukeMurphey
Champion

TSIDX is similar to summary indexing that allows dramatically improved performance. It is used in several applications such as Enterprise Security (version 2.4 and later). This feature was first available in Splunk 5.0.

okrabbe_splunk
Splunk Employee
Splunk Employee

Are you running enetprise security, PCI, or one of the newer releases of our apps?

Certain apps are now using TSIDX stats to offer better search acceleration than is possible using either summary indexing or report acceleration.

One thing you might want to do is search through your schedule searches and look for tscollect. This is what is populating that directory.
http://docs.splunk.com/Documentation/Splunk/5.0.2/SearchReference/Tscollect

0 Karma

lguinn2
Legend

I believe that these are the files for Report Acceleration Summaries. You should be able to manage them by going to Manager » Report Acceleration Summaries. If there are some that have never been used, you can just delete them.
This will turn off Report Acceleration in the associated reports, so the acceleration summaries will not be re-created.

Here is some of the documentation on Report Acceleration

sowings
Splunk Employee
Splunk Employee

No, report acceleration data lives within the $SPLUNK_DB hierarchy, alongside the indexes themselves.

lguinn2
Legend

I suggest that you test by creating some acceleration summaries for youself. At this time, acceleration summaries live on the search head (sadly) as txidx files. You are right that this is not the only way to get tsidx files though.

0 Karma

okrabbe_splunk
Splunk Employee
Splunk Employee

If these are on the SH I don't think they are report acceleration summaries.

See Luke Murphey's answer..

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...