Multiline Event Breaking and Current Time

[ifeldshteyn]

[ANSWERED by to4kawa]

props.conf should be

[yoursourcetype]
DATETIME_CONFIG = CURRENT
SHOULD_LINEMERGE = false
LINE_BREAKER = .*($)

I have a Catch 22 issue. I want three things to happen.

1. I want to monitor a log file and to combine all lines into a single event until the file is not updated for 2 seconds or more.
2. I want to disable all time extraction, I want all the time set to current (ie _indextime).
3. Being able to handle arbitrary data without any separators.

Imagine my file is empty and then within a microsecond these three lines are added to my file.

TEST1 Fri Apr  6 20:05:59 EDT 2020
TEST2 Fri Apr  3 20:04:30 EDT 2020
TEST3 Fri Apr  1 20:05:59 EDT 2020

I would like them all to be combined into a single event. And I want the timestamp to be set to current.

If I just just start monitoring the file without any props it combines the events into a single one just fine. BUT it will try to parse the timestamps and they will be all over the place. If I mod props.conf and set DATETIME_CONFIG=CURRENT it will set the time to the current one but then split the events into single lines.

So I am in a catch 22, I can have one or the other. Any ideas what inputs/props/transforms combo I can have that ignores all time stamps, and combines the events into a single one no matter what.

I honestly want something that checks if file has not been modified for 2 seconds and then combine everything new that was added into ONE event.

Thanks!

[richgalloway · Accepted solution]

These settings should help. Adjust the TRUNCATE and MAX_EVENTS settings as appropriate for your data.

inputs.conf:

[monitor://my/file]
multiline_event_extra_waittime = true
time_before_close = 2
sourcetype = foo
index = bar

props.conf:

[foo]
DATETIME_CONFIG = current
SHOULD_LINEMERGE = true
TRUNCATE = 10000
MAX_EVENTS = 256

[ifeldshteyn]

Hi Rich,

I just tried this.

INPUTS.CONF

 [monitor:///tmp/test3.log]
    disabled = false
    sourcetype = mytest
    multiline_event_extra_waittime = true
    time_before_close = 2

PROPS.CONF

[mytest]
DATETIME_CONFIG = CURRENT
SHOULD_LINEMERGE = true
TRUNCATE = 10000
MAX_EVENTS = 256

I've added

TEST8 Fri Apr  6 21:05:59 EDT 2020
TEST8 Fri Apr  3 21:04:30 EDT 2020
TEST8 Fri Apr  1 21:05:59 EDT 2020

But they came as separate events on Splunk 8.

[to4kawa]

SHOULD_LINEMERGE = false
LINE_BREAKER = .*($)

if SHOULD_LINEMERGE = false , LINE_BREAKER works.

.*($) is intended to be the end of the event.

[ifeldshteyn]

That's perfect, thank you! The below came together as one event AND the timestamp is auto set to current.

4/4/20 7:09:02.000 PM   

TEST11 Fri Apr  6 22:05:59 EDT 2020
TEST11 Fri Apr  3 22:04:30 EDT 2020
TEST11 Fri Apr  1 22:05:59 EDT 2020

Appreciate your time

[to4kawa]

your welcome
.*($) is OK, I recognized it, too.

[to4kawa]

until the file is not updated for 2 seconds or more.
what's inputs.conf?
this is not pops.conf issue.