topic Re: Ingest whole file in a single event. in Getting Data In

Ingest whole file in a single event.

VatsalJagani — Fri, 18 Sep 2020 15:11:43 GMT

I'm reading a file that is being overridden by a PowerShell script. (no append in the file)

The PowerShell script is using Out-File command to write the content to the file.

Sample content:

Domain Profile Settings:
----------------------------------------------------------------------
State                                 ONPrivate Profile Settings:
----------------------------------------------------------------------
State                                 ONPublic Profile Settings:
----------------------------------------------------------------------
State                                 ON
Ok.

props.conf file

###### FirewallStatus ######
[source::...FirewallStatus.Log]
sourcetype = FirewallStatusLog
CHECK_METHOD = modtime

[FirewallStatusLog]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE= neverbreak
DATETIME_CONFIG = NONE
pulldown_type = true

inputs.conf file

[monitor://$SPLUNK_HOME\var\log\custom\FirewallStatus.log]
disabled = 0
sourcetype = FirewallStatusLog

In my case at the first time content comes in a single event but the next time when file gets updated, the content got broken into two events by Splunk.
On a side note, it does not ingest events on all the update, if there is no update in the content. (That might be okay, I may create a lookup, this is no big deal)

Re: Ingest whole file in a single event.

richgalloway — Fri, 18 Sep 2020 17:13:12 GMT

The normal process for monitor inputs is to index events as they are added to the file. Splunk keeps track of its position in the file so events are not duplicated in the index.

If you need later updates to be included in the same event, consider having Splunk wait for additional data before indexing the file. See the multiline_event_extra_waittime and time_before_close settings in props.conf.

Re: Ingest whole file in a single event.

VatsalJagani — Mon, 21 Sep 2020 11:47:17 GMT

I guess that worked. I found these settings in inputs.conf file and not in props.conf file.

Thanks.!