So I had a support ticket open and this was the next thing we tried: "I was performing a health check based on the provided diag file and get a warning in regards to File descriptor cache stress. When the Splunk file descriptor cache is full, it will not be able to effectively tail all the files it should, sometimes resulting in missed or late data. In this case an important setting that can be adjusted in limits.conf, I'm speaking about the attribute max_fd under the [inputproc] stanza, you may test to set it as max_fd = 300. Please notice that to set custom configurations, you need to do it on a local directory of limits.conf and you must restart the Splunk instance to enable configuration changes. For more details, please review the following document: https://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf (https://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf)" However that didn't fix the issue and they said we need more CPU/Cores as we are currently running 4 cores (and have been for years with no issues) and that we need to have 12 now after that upgrade (and all other upgrades after). So we are going to get a couple of new CPU's and see if that helps...we will update after we upgrade the sever.
... View more