We are having an issue where a Universal Forwarder configured to forward a half dozen custom application logs is not forwarding any of them. It is a Windows server, and we are seeing Windows security information come across, and if I point to a known log, c:/\windows/\windowsupdate.log in inputs.conf, it sends that along. However, the other files are not coming across to the indexer.
They use custom sourcetypes that have been correctly specified and I can examine and appear to be set up correctly. The types are specified in inputs.conf for each file stanza and I can see them in Splunk's props.conf.
I cannot see a reason that only our custom files would be ignored. In splunkd.log on the forward it acknowledges their stanzas for each file:
03-31-2015 12:48:41.248 -0400 INFO TailingProcessor - Parsing configuration stanza: monitor://
There are no warnings or errors in the log file.
Splunk is running as a local system account and has permissions to the files.
I do not know what could be causing this issue and I am unsure where else I can look to diagnose the issue.
couple of things you can try, after seeing that message it sounds like parsing Que issue.
you can try increase
maxQueueSize = 200MB in outputs.conf.
some times the maxQueueSize setting causes events to be loaded into a queue in memory so Before increase check your memory and performance using top what ever OS command.
Note: check out metrics.log and parsingQueue or TCPque is full etc errors