Simple setup, two different sites with a single clustered Indexer in each, a local Heavy Forwarder that is also the deployment server for the UF's, and a SH in each site. I've deployed the TA_docker_simple app in both sites, installed on both HF's and the intended docker servers at each site.  Works great in one site but I get no data indexed in the other.  All UF's send in the data from the .sh scripts that the app contains (I can see event counts in their metrics.log) but on the problem site HF, I'm seeing messages like this: 06-27-2022 21:00:50.057 +0000 WARN DateParserVerbose - Accepted time (Fri Apr 1 18:31:29 2022) is suspiciously far away from the previous event's time (Fri Apr 1 19:46:38 2022), but still accepted because it was extracted by the same pattern. Context: source=docker_simple_ps|host=XXXXXX|docker:ps|6581 host = XXXXXXX source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd splunk_server = XXXXXXXX Which looks like it's trying to use a string date that is in the script output but isn't the timestamp (it's the container creation timestamp). The actual timestamp is an epoch integer at the beginning of each event.  Even if it were getting imported with the invalid timestamps I would see the data with a realtime search but I see nothing coming in.  I'm not sure how to resolve this.  Both sites are using the same copy of the app on the HF (minus the inputs.conf) and on the UFs.    It works perfectly in one site but not the other.  I've used btool to verify the props and transforms on the HF's are exactly the same.  It's probably something obvious but I can't figure this one out. ... View more

Using HF to forward all events to Indexer and external syslog. When using syslog with tcp all processing basically stopped as the queues filled up (and I've adjusted queue sizes already).  I haven't found much on Internet about this but did try UDP with the thought that is should be "send and forget" as far as the HF is concerned so it shouldn't slow data ingestion down but it still does. I'm not using a props or transforms for the syslog as I want it to send all events.  After bringing the HF up, within a few minutes the queues fill up and everything grinds to a halt. If you look at it from the local MC, you can see there is no resource load on the server and you see a little ingestion occur about every few minutes are so.  The little data that gets to the indexer gets more timestamp skewed.   I'm beating my head on that proverbial rock as this was working fine with tcp for a while and now it isn't working even using UDP. Here is my syslog outputs.conf on the HF: [syslog] defaultGroup = forwarders_syslog maxQueueSize = 10MB [syslog:forwarders_syslog] server = xx.xx.xx.xx:10514 type = udp disabled = 0 priority = <34> timestampformat = %b %e %H:%M:%S useACK=false I should also mention that there is no issue on the syslog server or the indexer, they are not taxed by any metric.  The syslog server is forwarding to another syslog via the Internet and does use tcp for that but since the incoming is written to a file, I don't see how that could impact the syslog receiving data from the HF. Any advice will be appreciated.  I've opened a case with Splunk but they have been less than responsive. ... View more