Odd Behavior using tstats with a datamodel

[ccWildcard]

Splunk: 8.0.3 (I know its old we're working on approvals to upgrade)

We’re receiving behavior I have never encountered before in Windows based server and I want to see if anybody else has encountered here because this may be happening on many of our systems where users claim the product isn’t working.

We have a tstats command running on a datamodel for a dashboard. When loading less than 24 hours worth of results the panel work as expected. The second we switch to a date range (March 11 – March 11 as an example) the other panels load fine but this one takes much longer to load (up from 1.1 minutes to over 5 minutes). At some point in loading the results begin shifting fields. For instance
Normal:

Time	Host	User	Status	Description	System
<time>	<host>	<user>	<status>	<description>	<system>

Then new results begin showing up:

Time	Host	User	Status	Description	System
	<tags>	<status>	<host>	<time>

This continues on and on until eventually the search fails and the following error is presented (one example): “StatsFileReader file open failed file=D:\Splunk\var\run\splunk\dispatch\_aWEtbG96ZW5k_ aWEtbG96ZW5k _US1BdWrpdA__search8_1741807955.367128\statstmp_21805.sb.lz4”

I’ve done the following to troubleshoot:

• Turned off data model acceleration
• Verified they’re running the default view and not a custom one
• Verified this happens on multiple dashboards using similar tstats search
• If I try to replicate in a | from datamodel search I do not see this happening. Seems to only happen with the tstats based search
• Click the “Open in Search” and saw the exact behavior there as well o Job inspector shows a lot of the following error: ERROR Bucket – Failed to discretize value ‘report’ of field ‘_time’. There’s 4 log files worth of these…However there’s a bunch of different values: track_event_signatures, windows, etc
• After these it says skipping prestats because input looks already in prestats format

Here is an copy of the tstats query that has been modified a little because this is from a paid app and I don't want to upset the publisher:

| tstats prestats=true summariesonly=false allow_old_summaries=false count as count FROM datamodel=Privileged WHERE (nodename=Privileged_Execution "Privileged_Execution.tag"=* "Privileged_Execution.user"="*" host="*" ) BY _time span=1s, host, "Privileged_Execution.process", "Privileged_Execution.user", "Privileged_Execution.description", "Privileged_Execution.status", "Privileged_Execution.tag"
| bucket _time span=1s
| stats dedup_splitvals=t count AS count by _time, host, Privileged_Execution.process, Privileged_Execution.user, Privileged_Execution.description, Privileged_Execution.status, Privileged_Execution.tag
| sort limit=`recent_events_tables_limit` -_time
| rename _time as Time, host as Host, "Privileged_Execution.process" as Process, "Privileged_Execution.user" as User, "Privileged_Execution.description" as Description, "Privileged_Execution.status" as Status, "Privileged_Execution.tag" as tag
| fillnull count
| fields + Time, Host, Process, User, Description, Status, tag, count
| join max=0 type=left tag [| inputlookup system_tag | rename system as System]
| fields - tag, count
| fillnull value="" System
| mvcombine System
| sort 0 - Time
| convert timeformat="%m/%d/%Y %H:%M:%S %z" ctime(Time)

[livehybrid]

Hi @ccWildcard 

I have heard of this issue before for others using Splunk 8.x on Windows - please could you confirm which version of Splunk you are running? Also, are there any unusual file permissions on that path?

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will

[ccWildcard]

Splunk 8.0.3 for sure. I'll check the file permissions on that path on Monday and get back to you.