Archive
Highlighted

Splunk Universal Forwarder Only Sends data once on Monitor Input config followed by Restart

Splunk Employee
Splunk Employee

I have a UF installed(v7.3.1) on CentOS with ulimits configured for max open files etc.

the file monitor input stanza looks as below:

[monitor:///<path_to_log_file>/*.log]
disabled = false
host_segment = 4
index = <index-name>
sourcetype = srctype
ignoreOlderThan = 1h

there are logs coming in at very high speed so the rsyslog creates a new file every 15mins, Hence the ignoreolderthan 1H clause is used .

Each time i configure a monitor stanza & restart UF.
It reads the files & sends it to the indexer. But after that, it doesn't forward any data.

UF splunkd.log stated that it was taking some huge files into batch mode & that maxKBPs limit had reached.
So I changed the limit.conf to set maxKBPs to 0.
There is no other error in Splunkd.log at UF & it still seems to be showcasing the same behavior.

Any pointers on how to resolve this or what else to look for?

0 Karma
Highlighted

Re: Splunk Universal Forwarder Only Sends data once on Monitor Input config followed by Restart

Champion

If you have monitoring console set, please check indexing performance on indexers. Is any indexing queue is getting full?

0 Karma
Highlighted

Re: Splunk Universal Forwarder Only Sends data once on Monitor Input config followed by Restart

Splunk Employee
Splunk Employee

So it seems like changing the maxKBPs limit to unlimited(0) fixed the problem.

Looks like UF was choking the default 256Kbps bandwidth once it picked up a huge file(~400MB).
I set the limits to 0 just before posting the question. Monitored this for about ~1.5hrs. Forwarder is reading & sending data across.

0 Karma
Highlighted

Re: Splunk Universal Forwarder Only Sends data once on Monitor Input config followed by Restart

Splunk Employee
Splunk Employee

@ssadh_splunk , As you mentioned rsyslog creates new file every 15 mins, can you try and increase the ignoreOlderThan parameter by 1 more hour?
For using wildcards, just make sure you are defining in correct manner.
Ref Doc: https://docs.splunk.com/Documentation/Splunk/7.3.0/Data/Specifyinputpathswithwildcards

Apart from that, configuration looks okay. The log files should get monitored continuously.

0 Karma
Highlighted

Re: Splunk Universal Forwarder Only Sends data once on Monitor Input config followed by Restart

Splunk Employee
Splunk Employee

Closing this as setting maxKBPs to zero in limits on UF fixed the issue.

View solution in original post

0 Karma