We are about to open up a Splunk ticket for this issue, but figured we'd check with the community first. Problem: The tstats command is not seeing all of our indexed data and queries would suggest that our Forwarders are not sending data, which isn't true. We've run multiple queries against the index confirming the expected data exists in the index and the fields are indexed. In addition, the hosts show up in the data summary for the index. We are searching within a timeline in which events do exist in the index, so it's not like we are searching for data that never existed. We even performed a restart of the Splunk service and noted a significant number of hosts' data in the index have stopped being processed by tstats / tsidx according to the timestamp of the latest event for the hosts. It coincides with the Splunk restart but never starts processing the data again to be visible by tstats, even after several hours. Other hosts data is processed as expected, so we have some hosts with current "lastSeen" times:     | tstats count max(_time) as lastSeen where index=windows_sec earliest=-20d@d latest=@m by host | convert ctime(lastSeen)     Command that results in missing hosts:     | tstats values(host) by index     Similar command that also results in same "missing" hosts --- Fast Mode:     index=* | stats values(host) by index     Modifying the above command from Fast to Verbose mode results in all hosts being displayed as expected. Additional Info: Splunk v8.2.6 - no correlation between different Forwarder versions either. Splunkd.log has been analyzed line by line pre/post Splunk service restart. No leads there. Tsidx reduction is (and always has been) disabled for all of our indexes. We have seen very similar behavior for other queries where Fast Mode results in missing data but simply changing the mode to Verbose instantly populates all expected data in the results. We even have verified that all fields are identified in the initial "generating" query - no difference in Fast Mode. This seems like a super basic issue but has completely baffled us for some time and is causing serious heartburn and lack of trust in the data being presented to users. It's almost like a caching issue of some sort but we are grasping at straws now. Any thoughts/ideas would be welcome. Thanks. ... View more

Greetings, Would anyone have any recommendations for forwarding events from physically isolated networks to a main network and thus providing a "single pane of glass"?  The networks must be physically isolated due to security requirements. Data diode connections are approved. We cannot implement firewalls, thus supporting TCP connections (and typical Splunk forwarding protocols). Here is a reference diagram of the logical architecture. We are looking at utilizing data diodes, which then require UDP connections, which of course, limits our options. We know how to get the traffic across the data diodes from a network configuration standpoint, so it's a matter of what data and how the data is structured so it can be processed correctly by the upstream Splunk Indexer that is the big question. Reference Logical ArchitectureOur biggest issue lies in the fact that UDP ingest on the upstream Splunk server only sees the Heavy Forwarders as the source vs. the original endpoints. We've searched at other forum posts with no luck on how to adapt them if they are even adaptable for this type of scenario, given we are ingesting Windows, Linux, and Syslog from each isolated network. Example: https://community.splunk.com/t5/Getting-Data-In/Keeping-Host-data-when-using-Heavy-Forwarder/m-p/232328 What happens between the Heavy Forwarders and the upstream Splunk Indexer is where we need help. Any and all creative ideas are welcome! For example: UDP packet manipulation inserting a source hostname on the Heavy Forwarders (but how can we determine the original source hostname?) Flat file export on Heavy Forwarder + ingest on upstream Splunk Indexer (but how can we leverage existing Props & Transforms without creating lots of custom Regex?) Is there a better "flat file" method that would support Windows and Linux in addition to Syslog (via SC4S)? Na, forget it, just leave the networks separate We are currently running Splunk Enterprise v8.2.3, on premise. Thank you in advance! ... View more