I upgraded from 4.1.6 to 4.3.2.
Yes, Pretty much I have been all over the splunk base posts and splunk docs/wiki.
I installed splunk on splunk (sos) and reviewed most of the structures of the splunk operation.
disabled SplunkDeploymentMonitor 4.3.2 until I find time later to check on it's behavior.
disabled SplunkDeploymentMonitor_4.1.x 1.0 .
Monitored the splunkd log and other splunk logs and restarted splunk many times after every tuning changes.
some of the changs:
I increased FD to 100,000 for soft and hard for user executing splunkd and also tweaked the limits.conf in splunk sub-directories for processes per cpu and percentage of searches and then cleaned up dispatch directory of 119 stale saved searches and reports/schedules and then few other smaller tweaks here and there..
I had lots of patience. I don't like to see errors /warnings in any system/application and etc logs. INFO/notices entries are okay.
Seems like all my splunk logs are now clean of errors/warnings. We'll monitor more during busy splunk usage and update here with more information.
... View more