I have a similar issue to this, the exact same behavior as above for a Centrify SAML authenticated Splunk instance, however it happens at 15 minutes on the dot whether you're using the session or not. Non-SAML Splunk Web sessions don't have this issue. If you try to do anything in the window as shown above, like refresh, you get an ugly error from the Centrify Connector: Of course we've engaged Centrify/Delinea but they aren't convinced it's not a problem on Splunk side. I've verified Splunk is not restarting and there is no timeout or TTL setting I can find that equates to 15 minutes. Also can't find any events concerning the disconnect in internal Splunk logs or ingested Delinea logs. Unfortunately this behavior makes our users reluctant to rely on the SAML Sessions.
... View more
Simple setup, two different sites with a single clustered Indexer in each, a local Heavy Forwarder that is also the deployment server for the UF's, and a SH in each site.
I've deployed the TA_docker_simple app in both sites, installed on both HF's and the intended docker servers at each site. Works great in one site but I get no data indexed in the other. All UF's send in the data from the .sh scripts that the app contains (I can see event counts in their metrics.log) but on the problem site HF, I'm seeing messages like this:
06-27-2022 21:00:50.057 + 0000 WARN DateParserVerbose - Accepted time ( Fri Apr 1 18:31:29 2022 ) is suspiciously far away from the previous event ' s time ( Fri Apr 1 19:46:38 2022 ), but still accepted because it was extracted by the same pattern. Context: source=docker_simple_ps | host=XXXXXX | docker:ps | 6581
host = XXXXXXX
source = /opt/splunk/var/log/splunk/splunkd.log
sourcetype = splunkd
splunk_server = XXXXXXXX
Which looks like it's trying to use a string date that is in the script output but isn't the timestamp (it's the container creation timestamp). The actual timestamp is an epoch integer at the beginning of each event. Even if it were getting imported with the invalid timestamps I would see the data with a realtime search but I see nothing coming in. I'm not sure how to resolve this. Both sites are using the same copy of the app on the HF (minus the inputs.conf) and on the UFs.
It works perfectly in one site but not the other. I've used btool to verify the props and transforms on the HF's are exactly the same. It's probably something obvious but I can't figure this one out.
... View more
Thanks for the reply, however I don't think the Syslog Connector is applicable in my case as I'm not using Syslog to get data into Splunk. I'm using Splunk to forward all incoming events (UF, HEC, etc.) to both an Indexer and a syslog relay that forwards to an external SIEM. I can try adjusting the maxQueueSize but I only modified that after this problem started.
... View more
Using HF to forward all events to Indexer and external syslog. When using syslog with tcp all processing basically stopped as the queues filled up (and I've adjusted queue sizes already). I haven't found much on Internet about this but did try UDP with the thought that is should be "send and forget" as far as the HF is concerned so it shouldn't slow data ingestion down but it still does. I'm not using a props or transforms for the syslog as I want it to send all events. After bringing the HF up, within a few minutes the queues fill up and everything grinds to a halt. If you look at it from the local MC, you can see there is no resource load on the server and you see a little ingestion occur about every few minutes are so. The little data that gets to the indexer gets more timestamp skewed. I'm beating my head on that proverbial rock as this was working fine with tcp for a while and now it isn't working even using UDP.
Here is my syslog outputs.conf on the HF:
[syslog] defaultGroup = forwarders_syslog maxQueueSize = 10MB
[syslog:forwarders_syslog] server = xx.xx.xx.xx:10514 type = udp disabled = 0 priority = <34> timestampformat = %b %e %H:%M:%S useACK=false
I should also mention that there is no issue on the syslog server or the indexer, they are not taxed by any metric. The syslog server is forwarding to another syslog via the Internet and does use tcp for that but since the incoming is written to a file, I don't see how that could impact the syslog receiving data from the HF. Any advice will be appreciated. I've opened a case with Splunk but they have been less than responsive.
... View more