Splunk Enterprise

what makes tstats on _internal go wrong?

MonkeyK
Builder

My teammate and I have been trying to summarize our environment to automatically build a data dictionary.  Our last feature was to add a lastSeen time to use as a rudimentary data integrity check.  
Recently this has stopped working on the _internal index.  As in tstats max time on _internal is a week ago, even though a straight SPL search on index=_internal returns results for today or any other arbitrary slice of time I query over the last week.  This suggests to me that the tsidx is messed up for _internal.  

But to make matters more confusing, yesterday I was able to submit the same query and get a correct max(_time) for index=_internal.  

Does anyone have an idea of what is going on with this behavior? Better yet, what I need to do to fix it?

If it matters, this is a clustered search head environment and we also have quite a few indexers

 

usual results

 

 

 

| tstats count max(_time) as lastSeen where index=_* earliest=-20d@d latest=@m by index
| convert ctime(lastSeen)

 

 

 

index count lastSeen

_audit99999999910/22/2021 15:39:59
_internal999999910/14/2021 20:09:35
_introspection99999999910/22/2021 15:39:59
_telemetry99910/22/2021 12:05:05
Labels (2)
0 Karma
1 Solution

gjanders
SplunkTrust
SplunkTrust

It's a known issue but not currently listed on  the known issues page (I have an active support case open on it).

I did ask why it's not on the known issues page, I'll ask support again for it to be listed...

Basically for tstats on the internal index you can add include_reduced_buckets=t, and that should make your results accurate.

However I found this made the search dramatically slower so it was better to not use tstats, they haven't provided a version with a bugfix yet...

Alerts for Splunk Admins https://splunkbase.splunk.com/app/3796/
Version Control for Splunk https://splunkbase.splunk.com/app/4355/

View solution in original post

0 Karma

gjanders
SplunkTrust
SplunkTrust

Which splunk version? I've hit an issue that appears to be a known issue with tstats and the internal index in 8.2.2...

Alerts for Splunk Admins https://splunkbase.splunk.com/app/3796/
Version Control for Splunk https://splunkbase.splunk.com/app/4355/
0 Karma

MonkeyK
Builder

I'm on 8.2.2 as well.
If the problem is really just _internal, I'm not super concerned.  But it really makes me uncomfortable that there might be errors with other indexes.

0 Karma

gjanders
SplunkTrust
SplunkTrust

It's a known issue but not currently listed on  the known issues page (I have an active support case open on it).

I did ask why it's not on the known issues page, I'll ask support again for it to be listed...

Basically for tstats on the internal index you can add include_reduced_buckets=t, and that should make your results accurate.

However I found this made the search dramatically slower so it was better to not use tstats, they haven't provided a version with a bugfix yet...

Alerts for Splunk Admins https://splunkbase.splunk.com/app/3796/
Version Control for Splunk https://splunkbase.splunk.com/app/4355/

View solution in original post

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!