I have a UF installed(v7.3.1) on CentOS with ulimits configured for max open files etc.
the file monitor input stanza looks as below:
[monitor:///<path_to_log_file>/*.log] disabled = false host_segment = 4 index = <index-name> sourcetype = srctype ignoreOlderThan = 1h
there are logs coming in at very high speed so the rsyslog creates a new file every 15mins, Hence the ignoreolderthan 1H clause is used .
Each time i configure a monitor stanza & restart UF.
It reads the files & sends it to the indexer. But after that, it doesn't forward any data.
UF splunkd.log stated that it was taking some huge files into batch mode & that maxKBPs limit had reached.
So I changed the limit.conf to set maxKBPs to 0.
There is no other error in Splunkd.log at UF & it still seems to be showcasing the same behavior.
Any pointers on how to resolve this or what else to look for?
If you have monitoring console set, please check indexing performance on indexers. Is any indexing queue is getting full?
So it seems like changing the maxKBPs limit to unlimited(0) fixed the problem.
Looks like UF was choking the default 256Kbps bandwidth once it picked up a huge file(~400MB).
I set the limits to 0 just before posting the question. Monitored this for about ~1.5hrs. Forwarder is reading & sending data across.
@ssadh_splunk , As you mentioned rsyslog creates new file every 15 mins, can you try and increase the ignoreOlderThan parameter by 1 more hour?
For using wildcards, just make sure you are defining in correct manner.
Ref Doc: https://docs.splunk.com/Documentation/Splunk/7.3.0/Data/Specifyinputpathswithwildcards
Apart from that, configuration looks okay. The log files should get monitored continuously.
Closing this as setting maxKBPs to zero in limits on UF fixed the issue.