Hello, we have a problem with persistent queue's in our infrastructure. We have TCP inputs sending SSL traffic to a heavy forwarder which acts as an intermediate forwarder. We do not parse on the hf! All we do is putting the data from TCP directly into the index queue. That mostly works perfectly fine for nearly 1 TB data per day. But sometimes the source pushes nearly 1 TB per hour which obviously overwhelms the HF, hence the persistent queue. We have the following inputs.conf:   [tcp-ssl:4444] index = xxx persistentQueueSize=378000MB sourcetype = xxx disabled=false queue = indexQueue   I expect all files in "/opt/splunk/var/run/splunk/tcpin/" for port 4444 to not exceed the allocated size of 378GB. But as can seen below, the total size of all files for port 4444 is 474GB! Way more than the allocated 378GB. Some files say corrupted probably because we hit our disk limit on the server and Splunk couldn't write to those files anymore. Did someone else experienced this behavior before? Thanks in advance and best regards, Eric ... View more

Hi, we're trying to get cloudwatchlogs via Kinesis Firehose to a heavy forwarder in a VPC. It's not working currently, we're getting the error: "Destination: https://XX.XX.XX.XX:8088 - Failed to deliver data to Splunk or to receive acknowledgment. Make sure HEC endpoint is reachable from Firehose and it is healthy." We confirmed that the HEC-Endpoint is working via curl. (ACK enabled) We opened to the CIDR-Blocks as described here: https://docs.aws.amazon.com/de_de/firehose/latest/dev/controlling-access.html#using-iam-splunk-vpc Our HF is on Splunk 7.1.1. It's SSL secured with the shipped cert. from splunk. Maybe that's the problem? (as described here: http://docs.splunk.com/Documentation/AddOns/released/Firehose/Troubleshoot#SSL-related_data_delivery_errors) Required ports are open (8088, 8089). We aren't getting any info regarding the HEC-Endpoint in the internal logs. Other HEC-Endpoints on the HF are working fine. Does anyone has some ideas? Regards and thanks in advance, Eric ... View more