Hello, I just recently restarted my splunk enterprise instance in order to add an app and once it was back up, i noticed that one of the health checks was failing. Also no new logs were showing up in the search. I looked at the monitoring console and noticed the parsing queue was full. I also checked the metrics.log and saw some of the queues were full. If I'm understanding the data pipeline hierarchy correctly, it's the parsing queue that's actually blocked and causing the other queues to be blocked. I also checked the splunkd.log and didn't really anything that seemed related. There were some SSL errors which didn't seem related. And this other error: ERROR HttpInputDataHandler - Failed processing http input, token name=kube, channel=n/a, source_IP=172.17.8.66, reply=9, events_processed=4, http_input_body_size=7256, parsing_err="Server is busy" but that seems to be a result of the full queue. I looked into my resource usage from the monitoring console and top tool and neither cpu or mem go higher than 50% utilization. I also restarted splunk multiple times but the queue always goes to 100% instantly. I did notice a warning on startup: Bad regex value: '(::)?...', of param: props.conf / [(::)?...]; why: this regex is likely to apply to all data and may break summary indexing, among other Splunk features. However, I didn't make any changes to props.conf and everything was working before I restarted the first time so I assume this is not related. Not sure what else to try. Any help would be greatly appreciated!
... View more
