topic Is there a way to delay splunk universal forwarder from monitoring specific files? in Getting Data In

Is there a way to delay splunk universal forwarder from monitoring specific files?

swamysanjanaput — Wed, 30 Sep 2020 03:04:13 GMT

Hello,

We have an issue monitoring os_metrics logs where the log entries are generated from a Windows command wmic and written to a file under this path D:\catmstarFiles\systems\main\logs\os_metrics*.log

The issue is that events are distorted even after placing the props (see below) in our heavy forwarder and search head cluster. The same set of files are read correctly if we are coping it to test server and monitoring it from there, however, in real-time the events are not breaking correctly as expected.

So, just wanted to know if there is an attribute that can be used in inputs.conf to reduce uf file reading/monitoring time? or Is it something to be done at source end to delay writing files to this particular path?
Can anyone please advise, if it's something to be done at source end I will then reach out to the concerned team and get it discussed. Thanks in advance

[sourcetype]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
CHARSET=AUTO
BREAK_ONLY_BEFORE=\w+\s+\d+\/\d+\/\d+\s+\d+:\d+:\d+.\d+
disabled=false
TIME_PREFIX=\w+\s
TIME_FORMAT=%m/%d/%Y %H:%M:%S.%N

Re: Is there a way to delay splunk universal forwarder from monitoring specific files?

arjunpkishore5 — Mon, 25 Nov 2019 12:47:22 GMT

What exactly do you mean by events are distorted? Do you mean they are not in the same order as the source? If the timestamp format is correct and a match is available in the file, Splunk should display events in the right format. If not, Splunk will use the index time. I would upload the file through the UI and verify if the timestamp format and any other setting in the props.conf are breaking the events right and extracting the data in the desired way.

Re: Is there a way to delay splunk universal forwarder from monitoring specific files?

woodcock — Tue, 26 Nov 2019 00:40:28 GMT

The docs:
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf
Say this:

time_before_close = <integer>
* The amount of time, in seconds, that the file monitor must wait for
  modifications before closing a file after reaching an End-of-File
  (EOF) marker.
* Tells the input not to close files that have been updated in the
  past 'time_before_close' seconds.
* Default: 3

multiline_event_extra_waittime = <boolean>
* By default, the file monitor sends an event delimiter when:
  * It reaches EOF of a file it monitors and
  * The last character it reads is a newline.
* In some cases, it takes time for all lines of a multiple-line event to
  arrive.
* Set to "true" to delay sending an event delimiter until the time that the
  file monitor closes the file, as defined by the 'time_before_close' setting,
  to allow all event lines to arrive.
* Default: false

Re: Is there a way to delay splunk universal forwarder from monitoring specific files?

swamysanjanaput — Tue, 26 Nov 2019 08:28:08 GMT

Thank you so much. I just added following attributes to inputs.conf and it worked like a charm:

time_before_close = 30
multiline_event_extra_waittime = true

Re: Is there a way to delay splunk universal forwarder from monitoring specific files?

joesrepsolc — Wed, 30 Sep 2020 03:34:18 GMT

Thanks woodcock. Just put this in place for us and solved a problem that we'd been struggling with. Added these two lines to our monitor and worked like a champ (log can take up to 5min to completely populate from all the wave of scripts).

time_before_close = 300
multiline_event_extra_waittime = true

Much appreciated!