Splunk Search

Running tscollect on a search head pool

sf_user_199
Path Finder

We have a dashboard that I would like to use tstats to generate the data, and run a search ever 2 minutes using tscollect.

The problem I have discovered is that since each search head in the pool saves tsidx files to it's own local tsidxstats directory, you only get a partial view of the data.

Any suggestions?

1 Solution

sf_user_199
Path Finder

We'll see if this blows up in my face, but:

  1. I stopped all search heads in the pool
  2. Created a new directory on the shared /nas that all the search heads use
  3. Copied the tsidx file that was the 'most full', as I had done some backfilling manually, to the new directory on the shared storage
  4. Deleted the tsidx directories on the search heads local storage, and replaced with a symlink to the new directory on shared storage
  5. Turned the search heads back on.

This has been running well for a few days, and the tsidx data is being updated by all search heads. Since the search that populates the data is only run by one search head, there have been no collisions so far.

View solution in original post

0 Karma

sf_user_199
Path Finder

We'll see if this blows up in my face, but:

  1. I stopped all search heads in the pool
  2. Created a new directory on the shared /nas that all the search heads use
  3. Copied the tsidx file that was the 'most full', as I had done some backfilling manually, to the new directory on the shared storage
  4. Deleted the tsidx directories on the search heads local storage, and replaced with a symlink to the new directory on shared storage
  5. Turned the search heads back on.

This has been running well for a few days, and the tsidx data is being updated by all search heads. Since the search that populates the data is only run by one search head, there have been no collisions so far.

0 Karma

jdunlea_splunk
Splunk Employee
Splunk Employee

Have you seen any performance hits having it running for a few months now? How big were/are your TSIDX namespaces?

Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...