We are about to open up a Splunk ticket for this issue, but figured we'd check with the community first.
Problem: The tstats command is not seeing all of our indexed data and queries would suggest that our Forwarders are not sending data, which isn't true. We've run multiple queries against the index confirming the expected data exists in the index and the fields are indexed. In addition, the hosts show up in the data summary for the index. We are searching within a timeline in which events do exist in the index, so it's not like we are searching for data that never existed.
We even performed a restart of the Splunk service and noted a significant number of hosts' data in the index have stopped being processed by tstats / tsidx according to the timestamp of the latest event for the hosts. It coincides with the Splunk restart but never starts processing the data again to be visible by tstats, even after several hours. Other hosts data is processed as expected, so we have some hosts with current "lastSeen" times:
| tstats count max(_time) as lastSeen where index=windows_sec earliest=-20d@d latest=@m by host
| convert ctime(lastSeen)
Command that results in missing hosts:
| tstats values(host) by index
Similar command that also results in same "missing" hosts --- Fast Mode:
index=* | stats values(host) by index
Modifying the above command from Fast to Verbose mode results in all hosts being displayed as expected.
Additional Info:
- Splunk v8.2.6 - no correlation between different Forwarder versions either.
- Splunkd.log has been analyzed line by line pre/post Splunk service restart. No leads there.
- Tsidx reduction is (and always has been) disabled for all of our indexes.
- We have seen very similar behavior for other queries where Fast Mode results in missing data but simply changing the mode to Verbose instantly populates all expected data in the results. We even have verified that all fields are identified in the initial "generating" query - no difference in Fast Mode.
This seems like a super basic issue but has completely baffled us for some time and is causing serious heartburn and lack of trust in the data being presented to users. It's almost like a caching issue of some sort but we are grasping at straws now. Any thoughts/ideas would be welcome.
Thanks.
