topic Re: Multiline Event Breaking and Current Time in Splunk Enterprise

Multiline Event Breaking and Current Time

ifeldshteyn — Mon, 08 Jun 2020 21:33:48 GMT

[ANSWERED by to4kawa]

props.conf should be

[yoursourcetype]
DATETIME_CONFIG = CURRENT
SHOULD_LINEMERGE = false
LINE_BREAKER = .*($)

I have a Catch 22 issue. I want three things to happen.

I want to monitor a log file and to combine all lines into a single event until the file is not updated for 2 seconds or more.
I want to disable all time extraction, I want all the time set to current (ie _indextime).
Being able to handle arbitrary data without any separators.

Imagine my file is empty and then within a microsecond these three lines are added to my file.

TEST1 Fri Apr  6 20:05:59 EDT 2020
TEST2 Fri Apr  3 20:04:30 EDT 2020
TEST3 Fri Apr  1 20:05:59 EDT 2020

I would like them all to be combined into a single event. And I want the timestamp to be set to current.

If I just just start monitoring the file without any props it combines the events into a single one just fine. BUT it will try to parse the timestamps and they will be all over the place. If I mod props.conf and set DATETIME_CONFIG=CURRENT it will set the time to the current one but then split the events into single lines.

So I am in a catch 22, I can have one or the other. Any ideas what inputs/props/transforms combo I can have that ignores all time stamps, and combines the events into a single one no matter what.

I honestly want something that checks if file has not been modified for 2 seconds and then combine everything new that was added into ONE event.

Thanks!

Re: Multiline Event Breaking and Current Time

to4kawa — Sat, 04 Apr 2020 02:55:42 GMT

until the file is not updated for 2 seconds or more.
what's inputs.conf?
this is not pops.conf issue.

Re: Multiline Event Breaking and Current Time

richgalloway — Sat, 04 Apr 2020 12:45:37 GMT

These settings should help. Adjust the TRUNCATE and MAX_EVENTS settings as appropriate for your data.

inputs.conf:

[monitor://my/file]
multiline_event_extra_waittime = true
time_before_close = 2
sourcetype = foo
index = bar

props.conf:

[foo]
DATETIME_CONFIG = current
SHOULD_LINEMERGE = true
TRUNCATE = 10000
MAX_EVENTS = 256

Re: Multiline Event Breaking and Current Time

ifeldshteyn — Sat, 04 Apr 2020 22:01:21 GMT

Hi Rich,

I just tried this.

INPUTS.CONF

 [monitor:///tmp/test3.log]
    disabled = false
    sourcetype = mytest
    multiline_event_extra_waittime = true
    time_before_close = 2

PROPS.CONF

[mytest]
DATETIME_CONFIG = CURRENT
SHOULD_LINEMERGE = true
TRUNCATE = 10000
MAX_EVENTS = 256

I've added

TEST8 Fri Apr  6 21:05:59 EDT 2020
TEST8 Fri Apr  3 21:04:30 EDT 2020
TEST8 Fri Apr  1 21:05:59 EDT 2020

But they came as separate events on Splunk 8.

Re: Multiline Event Breaking and Current Time

to4kawa — Sat, 04 Apr 2020 22:34:58 GMT

SHOULD_LINEMERGE = false
LINE_BREAKER = .*($)

if SHOULD_LINEMERGE = false , LINE_BREAKER works.

.*($) is intended to be the end of the event.

Re: Multiline Event Breaking and Current Time

ifeldshteyn — Sat, 04 Apr 2020 23:20:30 GMT

That's perfect, thank you! The below came together as one event AND the timestamp is auto set to current.

4/4/20 7:09:02.000 PM   

TEST11 Fri Apr  6 22:05:59 EDT 2020
TEST11 Fri Apr  3 22:04:30 EDT 2020
TEST11 Fri Apr  1 22:05:59 EDT 2020

Appreciate your time

Re: Multiline Event Breaking and Current Time

to4kawa — Sun, 05 Apr 2020 00:25:45 GMT

your welcome
.*($) is OK, I recognized it, too.