Splunk: 8.0.3 (I know its old we're working on approvals to upgrade) We’re receiving behavior I have never encountered before in Windows based server and I want to see if anybody else has encountered here because this may be happening on many of our systems where users claim the product isn’t working. We have a tstats command running on a datamodel for a dashboard. When loading less than 24 hours worth of results the panel work as expected. The second we switch to a date range (March 11 – March 11 as an example) the other panels load fine but this one takes much longer to load (up from 1.1 minutes to over 5 minutes). At some point in loading the results begin shifting fields. For instance Normal: Time Host User Status Description System <time> <host> <user> <status> <description> <system> Then new results begin showing up: Time Host User Status Description System <tags> <status> <host> <time> This continues on and on until eventually the search fails and the following error is presented (one example): “StatsFileReader file open failed file=D:\Splunk\var\run\splunk\dispatch\_aWEtbG96ZW5k_ aWEtbG96ZW5k _US1BdWrpdA__search8_1741807955.367128\statstmp_21805.sb.lz4” I’ve done the following to troubleshoot: Turned off data model acceleration Verified they’re running the default view and not a custom one Verified this happens on multiple dashboards using similar tstats search If I try to replicate in a | from datamodel search I do not see this happening. Seems to only happen with the tstats based search Click the “Open in Search” and saw the exact behavior there as well o Job inspector shows a lot of the following error: ERROR Bucket – Failed to discretize value ‘report’ of field ‘_time’. There’s 4 log files worth of these…However there’s a bunch of different values: track_event_signatures, windows, etc After these it says skipping prestats because input looks already in prestats format Here is an copy of the tstats query that has been modified a little because this is from a paid app and I don't want to upset the publisher: | tstats prestats=true summariesonly=false allow_old_summaries=false count as count FROM datamodel=Privileged WHERE (nodename=Privileged_Execution "Privileged_Execution.tag"=* "Privileged_Execution.user"="*" host="*" ) BY _time span=1s, host, "Privileged_Execution.process", "Privileged_Execution.user", "Privileged_Execution.description", "Privileged_Execution.status", "Privileged_Execution.tag" | bucket _time span=1s | stats dedup_splitvals=t count AS count by _time, host, Privileged_Execution.process, Privileged_Execution.user, Privileged_Execution.description, Privileged_Execution.status, Privileged_Execution.tag | sort limit=`recent_events_tables_limit` -_time | rename _time as Time, host as Host, "Privileged_Execution.process" as Process, "Privileged_Execution.user" as User, "Privileged_Execution.description" as Description, "Privileged_Execution.status" as Status, "Privileged_Execution.tag" as tag | fillnull count | fields + Time, Host, Process, User, Description, Status, tag, count | join max=0 type=left tag [| inputlookup system_tag | rename system as System] | fields - tag, count | fillnull value="" System | mvcombine System | sort 0 - Time | convert timeformat="%m/%d/%Y %H:%M:%S %z" ctime(Time)
... View more
