Hi, We had the same problem, but we had latencies ranging from minutes up to 15 hours, depending on the traffic load on the checkpoint (this was pre-upgrade on R77.30). It seemed that at more than 41000 events per minute, we experienced a build-up of latency. Performance and resource usage of the HF, the indexers or the checkpoint management server were just fine, no obvious culprit and no errors or warnings in opseclea:log:modinput. We have NMON running on the HF, so inspecting the CPU and memory usage was easy. Splunkd didn't do much (on average 0.35), and lea_lograbber was close to nothing (0.03). Python however used 1.6-1.7 CPU cores continuously. This was the maximum that i observed, which always happened at more than 41000 events per minute. Below the 41000 epm the process used less resources. After the upgrade to R80.10 it got even worse with latency running up to 22 hours and climbing. (I think) I managed to solve this by just setting the log level to INFO, instead of DEBUG (which I assumed was necessary for "debugging" this problem...). The debugging resulted in half a million events per minute of _internal debug logging... After changing this level, and setting the starttime on each input to a time a couple hours before (thus skipping most of the 22 hours of latency), the CPU usage of python was only 0.7 CPU core.... And fw1_loggrabber and splunkd spiked to levels not seen before (both at 2 CPU cores each). Around the same moment the Metrics log reports that it indexed 1.3 million events/minute for a couple of minutes. It seems that the DEBUG log level (very) negatively impacted the maximum events that python/lea_loggrabber could retrieve/send to splunk. Some time later the resource usage came down; splunkd and lea_loggrabber run both at 0.1 core, python at 0.03... That is at around 70000 events per minute. I'll be monitoring closely what happens under more load, but for now it seems all right. Action01 ... View more

I got this error whilst trying to disable the obsolete (according to Google Chrome) RSA key exchange ciphers by removing the "RSA+AESGCM:RSA+AES" from my cipherSuite. Upon investigating this issue, it seemed to be caused by the fact that out of the box, splunk cannot use the ECDH ciphers (ecdhCurves = ), so it always does a fallback to the ciphers with RSA as key exchange. But if you remove the RSA ciphers, there are no ciphers left for splunk to use, so it fails. By specifying "ecdhCurves = prime256v1,secp384r1,secp521r1" the ECDH ciphers can be used, and solves the problem. side-note; Using this in combination with the following config enables HSTS headers and Perfect Forward Secrecy: cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256; ecdhCurves = prime256v1,secp384r1,secp521r1 sendStictTransportSecurityHeader = true sslVersions = tls1.2 The above configuration is applicable to server.conf under [sslConfig] for the mgmt and kv store port, and in web.conf under [settings] for splunkweb. (The ciperSuite is taken form the Mozilla SSL Configuration Generator using the modern profile. Older browsers may not support this) ... View more