topic Re: How do you event break in props.conf? in Getting Data In

How do you event break in props.conf?

AKG1_old1 — Tue, 22 Jan 2019 12:14:32 GMT

Hello,

I am trying to break multiline events based on regex. but some events are not splitting properly.

Events should be broken before the timestamp occurrence. In the below given example of Full GC event, it should be a single event but it has been splitting in 2 different events.

props.conf

[G1_BETA]
MAX_TIMESTAMP_LOOKAHEAD = 30
BREAK_ONLY_BEFORE = ^\d\d\d\d-\d\d-\d\d
DATETIME_CONFIG = 
NO_BINARY_CHECK = true
category = Custom
pulldown_type = 1
disabled = false

raw data

2019-01-22T12:51:29.054+0100: 69921.814: [Full GC (Allocation Failure)  23G->10201M(28G), 12.6256586 secs]
   [Eden: 8192.0K(1424.0M)->0.0B(4408.0M) Survivors: 8192.0K->0.0B Heap: 23.5G(28.0G)->10201.9M(28.0G)], [Metaspace: 200039K->200039K(1230848K)]
 [Times: user=18.85 sys=0.10, real=12.62 secs] 

2019-01-22T12:51:47.419+0100: 69940.179: [GC pause (G1 Humongous Allocation) (young) (initial-mark), 0.0300747 secs]
   [Parallel Time: 24.3 ms, GC Workers: 11]
      [GC Worker Start (ms): Min: 69940181.0, Avg: 69940181.2, Max: 69940181.3, Diff: 0.2]

Re: How do you event break in props.conf?

eduardkiyko_ — Tue, 22 Jan 2019 12:30:06 GMT

Try to set SHOULD_LINEMERGE to "true"

Re: How do you event break in props.conf?

p_gurav — Tue, 22 Jan 2019 12:37:45 GMT

Try to use SHOULD_LINEMERGE = true in props.conf.

Re: How do you event break in props.conf?

harsmarvania57 — Tue, 22 Jan 2019 12:38:42 GMT

Hi @agoyal,

Please try below config in props.conf on Indexer/Heavy Forwarder whichever comes first from Universal Forwarder.

props.conf

[yourSourcetype]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N%z
TIME_PREFIX=^
MAX_TIMESTAMP_LOOKAHEAD=28

EDIT: Updated props.conf configuration, credit to @lakshman239

Re: How do you event break in props.conf?

lakshman239 — Tue, 29 Sep 2020 22:51:50 GMT

use TIME_PREFIX and TIME_FORMAT as well, if you use should_linemerge

Re: How do you event break in props.conf?

AKG1_old1 — Tue, 22 Jan 2019 12:59:30 GMT

@harsmarvania57 : Thanks for reply. but i think problem is not with regex. these files are reading live and Full GC events are getting printed in 2 parts.There is a small 1-2 sec gap between printing line. So Splunk picks half part first and 2nd part later. Is there anything can be done in such situation.

Re: How do you event break in props.conf?

AKG1_old1 — Tue, 22 Jan 2019 12:59:56 GMT

Thanks for reply. but i think problem is not with regex. these files are reading live and Full GC events are getting printed in 2 parts.There is a small 1-2 sec gap between printing line. So Splunk picks half part first and 2nd part later. Is there anything can be done in such situation.

Re: How do you event break in props.conf?

harsmarvania57 — Tue, 29 Sep 2020 22:54:10 GMT

I am breaking logs at timestamp only using LINE_BREAKER then is it really require TIME_FORMAT and TIME_PREFIX because splunk is automatically detecting TIMESTAMP correctly.

Re: How do you event break in props.conf?

harsmarvania57 — Tue, 22 Jan 2019 13:16:40 GMT

Have a look at below parameters in inputs.conf on UF and try this config (I didn't test this parameter so not sure how it reacts)

multiline_event_extra_waittime = <boolean>
* By default, the file monitor sends an event delimiter when:
  * It reaches EOF of a file it monitors and
  * Ihe last character it reads is a newline.
* In some cases, it takes time for all lines of a multiple-line event to
  arrive.
* Set to "true" to delay sending an event delimiter until the time that the
  file monitor closes the file, as defined by the 'time_before_close' setting,
  to allow all event lines to arrive.
* Default: false.

time_before_close = <integer>
* The amount of time, in seconds, that the file monitor must wait for
  modifications before closing a file after reaching an End-of-File
  (EOF) marker.
* Tells the input not to close files that have been updated in the
  past 'time_before_close' seconds.
* Default: 3.

Re: How do you event break in props.conf?

lakshman239 — Tue, 22 Jan 2019 13:28:07 GMT

Adding them will take away the default processing and evalate quickly and is also part of the best practices, when we manage multiline event with line breaker

Re: How do you event break in props.conf?

FrankVl — Tue, 22 Jan 2019 13:56:43 GMT

Yes, add the following to your inputs.conf: multiline_event_extra_waittime = true

Re: How do you event break in props.conf?

AKG1_old1 — Tue, 29 Sep 2020 22:55:03 GMT

@FrankVl : Thanks I have added this in my UNIVERSAL forwarder but not working.

[monitor:///net/dell730srv/dell730srv1/apps/LuasMaster/logs.../.log]
disabled = false
host = LUAS_2019_01_01
index = mlc_live
sourcetype = G1_BETA
multiline_event_extra_waittime = true
crcSalt =
whitelist = .*gc.log$|.*gc..log$
blacklist=logs_|fixing_|tps-archives

Re: How do you event break in props.conf?

AKG1_old1 — Wed, 23 Jan 2019 14:16:36 GMT

@harsmarvania57 : Thanks I have added this to my UF but seems like not working.

Re: How do you event break in props.conf?

FrankVl — Wed, 23 Jan 2019 14:18:16 GMT

I assume you restarted after that change? Could be that the issue then still is with the actual linebreaking config itself.

Re: How do you event break in props.conf?

harsmarvania57 — Wed, 23 Jan 2019 14:24:03 GMT

Are you sure that gap is only 1-2 seconds for 2nd part of multi line events ? If it's more than that then you need to increase time_before_close as well.

Re: How do you event break in props.conf?

AKG1_old1 — Tue, 29 Sep 2020 22:55:06 GMT

Yeah I have restarted forwarder after this change. I have tried with manual upload of full file and events are break down as per requirement.

I ll try to set higher value of "time_before_close = "

Re: How do you event break in props.conf?

AKG1_old1 — Wed, 23 Jan 2019 14:30:40 GMT

yeah I tried tailing the log file and it was less than 3 secs. but I am going to try with increasing wait time to 5 sec.