topic Re: Splunk Universal Forwarder Only Sends data once on Monitor Input config followed by Restart in Monitoring Splunk

Splunk Universal Forwarder Only Sends data once on Monitor Input config followed by Restart

ssadh_splunk — Wed, 28 Aug 2019 10:05:36 GMT

I have a UF installed(v7.3.1) on CentOS with ulimits configured for max open files etc.

the file monitor input stanza looks as below:

[monitor:///<path_to_log_file>/*.log]
disabled = false
host_segment = 4
index = <index-name>
sourcetype = srctype
ignoreOlderThan = 1h

there are logs coming in at very high speed so the rsyslog creates a new file every 15mins, Hence the ignoreolderthan 1H clause is used .

Each time i configure a monitor stanza & restart UF.
It reads the files & sends it to the indexer. But after that, it doesn't forward any data.

UF splunkd.log stated that it was taking some huge files into batch mode & that maxKBPs limit had reached.
So I changed the limit.conf to set maxKBPs to 0.
There is no other error in Splunkd.log at UF & it still seems to be showcasing the same behavior.

Any pointers on how to resolve this or what else to look for?

Re: Splunk Universal Forwarder Only Sends data once on Monitor Input config followed by Restart

p_gurav — Wed, 28 Aug 2019 10:37:04 GMT

If you have monitoring console set, please check indexing performance on indexers. Is any indexing queue is getting full?

Re: Splunk Universal Forwarder Only Sends data once on Monitor Input config followed by Restart

lmethwani_splun — Wed, 28 Aug 2019 10:42:10 GMT

@ssadh_splunk , As you mentioned rsyslog creates new file every 15 mins, can you try and increase the ignoreOlderThan parameter by 1 more hour?
For using wildcards, just make sure you are defining in correct manner.
Ref Doc: https://docs.splunk.com/Documentation/Splunk/7.3.0/Data/Specifyinputpathswithwildcards

Apart from that, configuration looks okay. The log files should get monitored continuously.

Re: Splunk Universal Forwarder Only Sends data once on Monitor Input config followed by Restart

ssadh_splunk — Wed, 28 Aug 2019 11:13:56 GMT

So it seems like changing the maxKBPs limit to unlimited(0) fixed the problem.

Looks like UF was choking the default 256Kbps bandwidth once it picked up a huge file(~400MB).
I set the limits to 0 just before posting the question. Monitored this for about ~1.5hrs. Forwarder is reading & sending data across.

Re: Splunk Universal Forwarder Only Sends data once on Monitor Input config followed by Restart

ssadh_splunk — Wed, 28 Aug 2019 11:50:55 GMT

Closing this as setting maxKBPs to zero in limits on UF fixed the issue.