i have inherited a splunk installation, done by a 3rd party. We are currently using Splunk Enterprise version 8.0.2, with universal forwarders on a Solaris host (11.3) and 4 solaris zones on that host.
We are experiencing very high memory consumption and CPU usage on the host and respective zones, but a restart of the splunk daemon usually resolves the memory issues. We are currently restarting the splunk daemon's every 4-5 days.
When we do restart the splunk services, they jump to the top CPU users the moment it's started.
I have read that the high CPU could be attributed to the number of files/directories being monitored, so I ran the "splunk list monitor" command on each zone being monitored and on the host, and found that certain directories were being monitored across all forwarders, even if those directories didn't exist on that zone.
I still don't know enough about splunk (am working through a pluralsight splunk fundamentals training course) to know whether the list of files/directories to be monitored is being set at a zone/machine level or globally, and where I can go to find out.
Any assistance in this regard would be greatly appreciated.