Have encountered an environmental health issue , where large amounts of searches are being skipped. This large number of skipped searches suggests that the environment is not healthy.
We have deployed apps (myWizard 360) previously to monitor KPIs of production and non production servers that has added the number of searches, and forwarding additional data that also might be the cause of the increase the load on the server.
We have experienced skipped searches on our Search Head since last Mar 15. The latest change we did that time was setting new deployment clients for the myWizard 360 apps, but when we knew about the skipped search issue we have reverted the configuration of the new deployment clients. After the revert, we are still experiencing the skipped searches. Thorough investigation points to these apps below to have been running accelerated datamodels that are being skipped. I have first disabled them and do Splunk restart but still issue persist. Have uninstalled the apps below then restarted Splunk and still issue remains.
Lastly is we asked, AWS team to reboot the box but this change does not have significant impact to Splunk skipped searches as well.
... View more