How do you event break in props.conf?

[AKG1_old1]

Hello,

I am trying to break multiline events based on regex. but some events are not splitting properly.

Events should be broken before the timestamp occurrence. In the below given example of Full GC event, it should be a single event but it has been splitting in 2 different events.

props.conf

[G1_BETA]
MAX_TIMESTAMP_LOOKAHEAD = 30
BREAK_ONLY_BEFORE = ^\d\d\d\d-\d\d-\d\d
DATETIME_CONFIG = 
NO_BINARY_CHECK = true
category = Custom
pulldown_type = 1
disabled = false

raw data

2019-01-22T12:51:29.054+0100: 69921.814: [Full GC (Allocation Failure)  23G->10201M(28G), 12.6256586 secs]
   [Eden: 8192.0K(1424.0M)->0.0B(4408.0M) Survivors: 8192.0K->0.0B Heap: 23.5G(28.0G)->10201.9M(28.0G)], [Metaspace: 200039K->200039K(1230848K)]
 [Times: user=18.85 sys=0.10, real=12.62 secs] 

2019-01-22T12:51:47.419+0100: 69940.179: [GC pause (G1 Humongous Allocation) (young) (initial-mark), 0.0300747 secs]
   [Parallel Time: 24.3 ms, GC Workers: 11]
      [GC Worker Start (ms): Min: 69940181.0, Avg: 69940181.2, Max: 69940181.3, Diff: 0.2]

[harsmarvania57]

Hi @agoyal,

Please try below config in props.conf on Indexer/Heavy Forwarder whichever comes first from Universal Forwarder.

props.conf

[yourSourcetype]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N%z
TIME_PREFIX=^
MAX_TIMESTAMP_LOOKAHEAD=28

EDIT: Updated props.conf configuration, credit to @lakshman239

[AKG1_old1]

@harsmarvania57 : Thanks for reply. but i think problem is not with regex. these files are reading live and Full GC events are getting printed in 2 parts.There is a small 1-2 sec gap between printing line. So Splunk picks half part first and 2nd part later. Is there anything can be done in such situation.

[harsmarvania57]

Have a look at below parameters in inputs.conf on UF and try this config (I didn't test this parameter so not sure how it reacts)

multiline_event_extra_waittime = <boolean>
* By default, the file monitor sends an event delimiter when:
  * It reaches EOF of a file it monitors and
  * Ihe last character it reads is a newline.
* In some cases, it takes time for all lines of a multiple-line event to
  arrive.
* Set to "true" to delay sending an event delimiter until the time that the
  file monitor closes the file, as defined by the 'time_before_close' setting,
  to allow all event lines to arrive.
* Default: false.

time_before_close = <integer>
* The amount of time, in seconds, that the file monitor must wait for
  modifications before closing a file after reaching an End-of-File
  (EOF) marker.
* Tells the input not to close files that have been updated in the
  past 'time_before_close' seconds.
* Default: 3.

[AKG1_old1]

@harsmarvania57 : Thanks I have added this to my UF but seems like not working.

[harsmarvania57]

Are you sure that gap is only 1-2 seconds for 2nd part of multi line events ? If it's more than that then you need to increase time_before_close as well.

[AKG1_old1]

yeah I tried tailing the log file and it was less than 3 secs. but I am going to try with increasing wait time to 5 sec.

[lakshman239]

use TIME_PREFIX and TIME_FORMAT as well, if you use should_linemerge

[harsmarvania57]

I am breaking logs at timestamp only using LINE_BREAKER then is it really require TIME_FORMAT and TIME_PREFIX because splunk is automatically detecting TIMESTAMP correctly.

[lakshman239]

Adding them will take away the default processing and evalate quickly and is also part of the best practices, when we manage multiline event with line breaker

[p_gurav]

Try to use SHOULD_LINEMERGE = true in props.conf.

[AKG1_old1]

Thanks for reply. but i think problem is not with regex. these files are reading live and Full GC events are getting printed in 2 parts.There is a small 1-2 sec gap between printing line. So Splunk picks half part first and 2nd part later. Is there anything can be done in such situation.

[FrankVl]

Yes, add the following to your inputs.conf: multiline_event_extra_waittime = true

[AKG1_old1]

@FrankVl : Thanks I have added this in my UNIVERSAL forwarder but not working.

[monitor:///net/dell730srv/dell730srv1/apps/LuasMaster/logs.../.log]
disabled = false
host = LUAS_2019_01_01
index = mlc_live
sourcetype = G1_BETA
multiline_event_extra_waittime = true
crcSalt =
whitelist = .*gc.log$|.*gc..log$
blacklist=logs_|fixing_|tps-archives

[FrankVl]

I assume you restarted after that change? Could be that the issue then still is with the actual linebreaking config itself.

[AKG1_old1]

Yeah I have restarted forwarder after this change. I have tried with manual upload of full file and events are break down as per requirement.

I ll try to set higher value of "time_before_close = "

[eduardkiyko_]

Try to set SHOULD_LINEMERGE to "true"