Hello, I just recently restarted my splunk enterprise instance in order to add an app and once it was back up, i noticed that one of the health checks was failing. Screen Shot 2022-02-01 at 9.16.38 PM.png Also no new logs were showing up in the search. I looked at the monitoring console and noticed the parsing queue was full. Screen Shot 2022-02-01 at 9.19.58 PM.png I also checked the metrics.log and saw some of the queues were full. Screen Shot 2022-02-01 at 9.36.03 PM.png If I'm understanding the data pipeline hierarchy correctly, it's the parsing queue that's actually blocked and causing the other queues to be blocked. I also checked the splunkd.log and didn't really anything that seemed related. There were some SSL errors which didn't seem related. And this other error: ERROR HttpInputDataHandler - Failed processing http input, token name=kube, channel=n/a, source_IP=172.17.8.66, reply=9, events_processed=4, http_input_body_size=7256, parsing_err="Server is busy" but that seems to be a result of the full queue. I looked into my resource usage from the monitoring console and top tool and neither cpu or mem go higher than 50% utilization. I also restarted splunk multiple times but the queue always goes to 100% instantly. I did notice a warning on startup: Bad regex value: '(::)?...', of param: props.conf / [(::)?...]; why: this regex is likely to apply to all data and may break summary indexing, among other Splunk features. However, I didn't make any changes to props.conf and everything was working before I restarted the first time so I assume this is not related. Not sure what else to try. Any help would be greatly appreciated!
... View more
