Splunk Enterprise

Multiline Event Breaking and Current Time

ifeldshteyn
Communicator

[ANSWERED by to4kawa]

props.conf should be

[yoursourcetype]
DATETIME_CONFIG = CURRENT
SHOULD_LINEMERGE = false
LINE_BREAKER = .*($)

I have a Catch 22 issue. I want three things to happen.

  1. I want to monitor a log file and to combine all lines into a single event until the file is not updated for 2 seconds or more.
  2. I want to disable all time extraction, I want all the time set to current (ie _indextime).
  3. Being able to handle arbitrary data without any separators.

Imagine my file is empty and then within a microsecond these three lines are added to my file.

TEST1 Fri Apr  6 20:05:59 EDT 2020
TEST2 Fri Apr  3 20:04:30 EDT 2020
TEST3 Fri Apr  1 20:05:59 EDT 2020

I would like them all to be combined into a single event. And I want the timestamp to be set to current.

If I just just start monitoring the file without any props it combines the events into a single one just fine. BUT it will try to parse the timestamps and they will be all over the place. If I mod props.conf and set DATETIME_CONFIG=CURRENT it will set the time to the current one but then split the events into single lines.

So I am in a catch 22, I can have one or the other. Any ideas what inputs/props/transforms combo I can have that ignores all time stamps, and combines the events into a single one no matter what.

I honestly want something that checks if file has not been modified for 2 seconds and then combine everything new that was added into ONE event.

Thanks!

Labels (1)
Tags (1)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

These settings should help. Adjust the TRUNCATE and MAX_EVENTS settings as appropriate for your data.

inputs.conf:

[monitor://my/file]
multiline_event_extra_waittime = true
time_before_close = 2
sourcetype = foo
index = bar

props.conf:

[foo]
DATETIME_CONFIG = current
SHOULD_LINEMERGE = true
TRUNCATE = 10000
MAX_EVENTS = 256
---
If this reply helps you, Karma would be appreciated.

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust

These settings should help. Adjust the TRUNCATE and MAX_EVENTS settings as appropriate for your data.

inputs.conf:

[monitor://my/file]
multiline_event_extra_waittime = true
time_before_close = 2
sourcetype = foo
index = bar

props.conf:

[foo]
DATETIME_CONFIG = current
SHOULD_LINEMERGE = true
TRUNCATE = 10000
MAX_EVENTS = 256
---
If this reply helps you, Karma would be appreciated.
0 Karma

ifeldshteyn
Communicator

Hi Rich,

I just tried this.

INPUTS.CONF

 [monitor:///tmp/test3.log]
    disabled = false
    sourcetype = mytest
    multiline_event_extra_waittime = true
    time_before_close = 2

PROPS.CONF

[mytest]
DATETIME_CONFIG = CURRENT
SHOULD_LINEMERGE = true
TRUNCATE = 10000
MAX_EVENTS = 256

I've added

TEST8 Fri Apr  6 21:05:59 EDT 2020
TEST8 Fri Apr  3 21:04:30 EDT 2020
TEST8 Fri Apr  1 21:05:59 EDT 2020

But they came as separate events on Splunk 8.

alt text

0 Karma

to4kawa
Ultra Champion
SHOULD_LINEMERGE = false
LINE_BREAKER = .*($)

if SHOULD_LINEMERGE = false , LINE_BREAKER works.

.*($) is intended to be the end of the event.

ifeldshteyn
Communicator

That's perfect, thank you! The below came together as one event AND the timestamp is auto set to current.

4/4/20 7:09:02.000 PM   

TEST11 Fri Apr  6 22:05:59 EDT 2020
TEST11 Fri Apr  3 22:04:30 EDT 2020
TEST11 Fri Apr  1 22:05:59 EDT 2020

Appreciate your time

0 Karma

to4kawa
Ultra Champion

your welcome
.*($) is OK, I recognized it, too.

0 Karma

to4kawa
Ultra Champion

until the file is not updated for 2 seconds or more.
what's inputs.conf?
this is not pops.conf issue.

0 Karma
Get Updates on the Splunk Community!

Splunk Education - Fast Start Program!

Welcome to Splunk Education! Splunk training programs are designed to enable you to get started quickly and ...

Five Subtly Different Ways of Adding Manual Instrumentation in Java

You can find the code of this example on GitHub here. Please feel free to star the repository to keep in ...

New Splunk APM Enhancements Help Troubleshoot Your MySQL and NoSQL Databases Faster

Splunk Observability has two new enhancements to make it quicker and easier to troubleshoot slow or frequently ...