Splunk Search

tstats issue following upgrade from 6.4.4 to 6.5.2

adayton20
Contributor

I’m having an issue with the tstats command not producing any results when calling a namespace post tscollect.

For example, I have a search where I pipe the results to a namespace

… | tscollect namespace=foo keepresults=true

The tsidx file, foo, appears in /opt/splunk/var/lib/splunk/tsidxstats and I can see the data I want in the file amongst the binary jargon.

When using tstats, I get 0 results.

|tstats count FROM foo by Image

When using an accelerated datamodel with tstats, it works just fine.

I don’t have this issue on 6.4.4.

I checked the known issues and did a fair amount of searching on this, but couldn't find anything. Was hoping maybe I missed something simple… throw me a bone here?

1 Solution

adayton20
Contributor

Nevermind. I figured it out, but had to change a setting Splunk "discourages" doing. In the Monitoring Console, under server roles, I added the role of "indexer" to my search head. It threw a bunch of warnings at me, but accepted my change. Suddenly tstats works with any namespace I specify. No idea why this worked as I was under the impression the settings in MC were for reporting? If someone could shine some light on this for educational purposes I'd be grateful. Not sure if something changed from 6.4.4 to 6.5.2 in the way the search head dispatches tstats jobs. Perhaps a server must have an "indexing" capability or configuration file enabled in a distributed environment for the tstats function to search the tsidx files saved to /opt/splunk/var/lib/splunk/tsidxstats on the search head via tscollect? What is odd to me is that in my last workplace, the distributed environment I managed had a search head where the server roles were set to search head and license master only, and tstats worked without issue... then again their setup was not 6.5+ and it was kind of janky. Anyway, cheers!

View solution in original post

adayton20
Contributor

Nevermind. I figured it out, but had to change a setting Splunk "discourages" doing. In the Monitoring Console, under server roles, I added the role of "indexer" to my search head. It threw a bunch of warnings at me, but accepted my change. Suddenly tstats works with any namespace I specify. No idea why this worked as I was under the impression the settings in MC were for reporting? If someone could shine some light on this for educational purposes I'd be grateful. Not sure if something changed from 6.4.4 to 6.5.2 in the way the search head dispatches tstats jobs. Perhaps a server must have an "indexing" capability or configuration file enabled in a distributed environment for the tstats function to search the tsidx files saved to /opt/splunk/var/lib/splunk/tsidxstats on the search head via tscollect? What is odd to me is that in my last workplace, the distributed environment I managed had a search head where the server roles were set to search head and license master only, and tstats worked without issue... then again their setup was not 6.5+ and it was kind of janky. Anyway, cheers!

dwaddle
SplunkTrust
SplunkTrust

Are you doing 'other things' other than monitoring console on your monitoring console host? See http://docs.splunk.com/Documentation/Splunk/6.5.2/DMC/WheretohostDMC. One thing MC does in a distributed environment is set up a number of distributed search groups and use those to restrict where searches get run. This can lead to unintended side effects.

One of those 'unintended side effects' controls the searching of data created at the MC node, if the MC node is not listed as an 'indexer'. And, one of the reasons tscollect is deprecated is because it makes tstats files that get stored at the search head.

I'd say that by using tscollect at all, you're venturing into a possible source of issues down the road. It seems great, and then kicks you in the face when you least expect it. Datamodels & DM Acceleration are a vastly superior solution to most problems.

adayton20
Contributor

Yes, actually I was doing other things where the MC was hosted. I did read that documentation initially, but I see now how I misinterpreted it (thought "other purposes" was referring to something like ES, or where the SH was also acting as an IDX or LM, not just using it as the main SH). I had no idea there were unintended side effects, but definitely good to know. Thank you for the information, it is very helpful and most appreciated!

0 Karma

rjthibod
Champion

Do you see any results if you run the search |tstats count FROM foo?

Also, can you find the actual tsidx file for your namespace and run the walklex command against it?

<SPLUNK_ROOT>/bin/splunk cmd walklex <PATH_TO_TSIDX_FILE>

0 Karma

adayton20
Contributor

No results simply specifying the count w/ no fields.

The output from walklex:

[root@searchhead foo]# /opt/splunk/bin/splunk cmd walklex /opt/splunk/var/lib/splunk/tsidxstats/foo/1487594311-1487591248-5257261242841871460.tsidx ""
my needle:
0 2  CommandLine::ipconfig
1 1  CommandLine::ping  8.8.8.8
2 1  Image::C:\Windows\System32\PING.EXE
3 2  Image::C:\Windows\System32\ipconfig.exe
4 3  ParentCommandLine::"C:\Windows\System32\cmd.exe"
5 3  host::myhostname
6 3  user::myusername
7 2 commandline::ipconfig
8 1 commandline::ping  8.8.8.8
9 3 host::myhostname
10 2 image::c:\windows\system32\ipconfig.exe
11 1 image::c:\windows\system32\ping.exe
12 3 parentcommandline::"c:\windows\system32\cmd.exe"
13 3 user::myusername
[root@searchhead foo]#
0 Karma

rjthibod
Champion

hrm ... not seeing anything obvious. Let me poke someone else.

0 Karma
Get Updates on the Splunk Community!

This Week's Community Digest - Splunk Community Happenings [9.26.22]

Get the latest news and updates from the Splunk Community here! Upcoming User Group Events! &#x1f44f; Check ...

BSides Splunk 2022 - The Call for Papers is now Open!

TLDR; Main Site: https://bsidessplunk.com CFP Site: https://bsidessplunk.com/cfp CFP Opens: December 15th, ...

Sending Metrics to Splunk Enterprise With the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...