We've been noticing a high CPU use on a windows splunk forwarder that only has a simple monitor statement.
The following monitor is used:
disabled = false
index = dataindex
sourcetype = datatype
With a few date.log files to monitor and the correct output to the indexers.
Does anyone know what the cause might be for the high CPU? The _internal logs show nothing of interest.
It's always recommended not to use wildcard in the monitor stanza if you really have less number of files to be monitored.
Also, verify the below points-
1. Number of files that are getting monitored by the command
./splunk list monitor.
2. Size of the log files.
3. Proper parsing of the log files.
4. Check if any older files are being monitored and if so you can ignore those.
I hope you would find something from checking the above listed points.
If possible, always install the forwarder on the server with the files - mounting a remote share to pull data into a UF is inefficient.
Its not always possible ( I know) but UNC file shares add failure points, latency and network overhead you are better off avoiding if possible. - Probably not the direct cause of you issue, but worth considering.
What version of UF/Windows? and how big are they logs. Do they break nicely? - Have you looked at your queues on the UF?