tstats issue following upgrade from 6.4.4 to 6.5.2

[adayton20]

I’m having an issue with the tstats command not producing any results when calling a namespace post tscollect.

For example, I have a search where I pipe the results to a namespace

… | tscollect namespace=foo keepresults=true

The tsidx file, foo, appears in /opt/splunk/var/lib/splunk/tsidxstats and I can see the data I want in the file amongst the binary jargon.

When using tstats, I get 0 results.

|tstats count FROM foo by Image

When using an accelerated datamodel with tstats, it works just fine.

I don’t have this issue on 6.4.4.

I checked the known issues and did a fair amount of searching on this, but couldn't find anything. Was hoping maybe I missed something simple… throw me a bone here?

[adayton20 · Accepted solution]

Nevermind. I figured it out, but had to change a setting Splunk "discourages" doing. In the Monitoring Console, under server roles, I added the role of "indexer" to my search head. It threw a bunch of warnings at me, but accepted my change. Suddenly tstats works with any namespace I specify. No idea why this worked as I was under the impression the settings in MC were for reporting? If someone could shine some light on this for educational purposes I'd be grateful. Not sure if something changed from 6.4.4 to 6.5.2 in the way the search head dispatches tstats jobs. Perhaps a server must have an "indexing" capability or configuration file enabled in a distributed environment for the tstats function to search the tsidx files saved to /opt/splunk/var/lib/splunk/tsidxstats on the search head via tscollect? What is odd to me is that in my last workplace, the distributed environment I managed had a search head where the server roles were set to search head and license master only, and tstats worked without issue... then again their setup was not 6.5+ and it was kind of janky. Anyway, cheers!

[dwaddle]

Are you doing 'other things' other than monitoring console on your monitoring console host? See http://docs.splunk.com/Documentation/Splunk/6.5.2/DMC/WheretohostDMC. One thing MC does in a distributed environment is set up a number of distributed search groups and use those to restrict where searches get run. This can lead to unintended side effects.

One of those 'unintended side effects' controls the searching of data created at the MC node, if the MC node is not listed as an 'indexer'. And, one of the reasons tscollect is deprecated is because it makes tstats files that get stored at the search head.

I'd say that by using tscollect at all, you're venturing into a possible source of issues down the road. It seems great, and then kicks you in the face when you least expect it. Datamodels & DM Acceleration are a vastly superior solution to most problems.

[adayton20]

Yes, actually I was doing other things where the MC was hosted. I did read that documentation initially, but I see now how I misinterpreted it (thought "other purposes" was referring to something like ES, or where the SH was also acting as an IDX or LM, not just using it as the main SH). I had no idea there were unintended side effects, but definitely good to know. Thank you for the information, it is very helpful and most appreciated!

[rjthibod]

Do you see any results if you run the search |tstats count FROM foo?

Also, can you find the actual tsidx file for your namespace and run the walklex command against it?

<SPLUNK_ROOT>/bin/splunk cmd walklex <PATH_TO_TSIDX_FILE>

[adayton20]

No results simply specifying the count w/ no fields.

The output from walklex:

[root@searchhead foo]# /opt/splunk/bin/splunk cmd walklex /opt/splunk/var/lib/splunk/tsidxstats/foo/1487594311-1487591248-5257261242841871460.tsidx ""
my needle:
0 2  CommandLine::ipconfig
1 1  CommandLine::ping  8.8.8.8
2 1  Image::C:\Windows\System32\PING.EXE
3 2  Image::C:\Windows\System32\ipconfig.exe
4 3  ParentCommandLine::"C:\Windows\System32\cmd.exe"
5 3  host::myhostname
6 3  user::myusername
7 2 commandline::ipconfig
8 1 commandline::ping  8.8.8.8
9 3 host::myhostname
10 2 image::c:\windows\system32\ipconfig.exe
11 1 image::c:\windows\system32\ping.exe
12 3 parentcommandline::"c:\windows\system32\cmd.exe"
13 3 user::myusername
[root@searchhead foo]#

[rjthibod]

hrm ... not seeing anything obvious. Let me poke someone else.