Getting Data In

How do you event break in props.conf?

Builder

Hello,

I am trying to break multiline events based on regex. but some events are not splitting properly.

Events should be broken before the timestamp occurrence. In the below given example of Full GC event, it should be a single event but it has been splitting in 2 different events.

props.conf

[G1_BETA]
MAX_TIMESTAMP_LOOKAHEAD = 30
BREAK_ONLY_BEFORE = ^\d\d\d\d-\d\d-\d\d
DATETIME_CONFIG = 
NO_BINARY_CHECK = true
category = Custom
pulldown_type = 1
disabled = false

raw data

2019-01-22T12:51:29.054+0100: 69921.814: [Full GC (Allocation Failure)  23G->10201M(28G), 12.6256586 secs]
   [Eden: 8192.0K(1424.0M)->0.0B(4408.0M) Survivors: 8192.0K->0.0B Heap: 23.5G(28.0G)->10201.9M(28.0G)], [Metaspace: 200039K->200039K(1230848K)]
 [Times: user=18.85 sys=0.10, real=12.62 secs] 

2019-01-22T12:51:47.419+0100: 69940.179: [GC pause (G1 Humongous Allocation) (young) (initial-mark), 0.0300747 secs]
   [Parallel Time: 24.3 ms, GC Workers: 11]
      [GC Worker Start (ms): Min: 69940181.0, Avg: 69940181.2, Max: 69940181.3, Diff: 0.2]

alt text

0 Karma

SplunkTrust
SplunkTrust

Hi @agoyal,

Please try below config in props.conf on Indexer/Heavy Forwarder whichever comes first from Universal Forwarder.

props.conf

[yourSourcetype]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N%z
TIME_PREFIX=^
MAX_TIMESTAMP_LOOKAHEAD=28

EDIT: Updated props.conf configuration, credit to @lakshman239

0 Karma

Builder

@harsmarvania57 : Thanks for reply. but i think problem is not with regex. these files are reading live and Full GC events are getting printed in 2 parts.There is a small 1-2 sec gap between printing line. So Splunk picks half part first and 2nd part later. Is there anything can be done in such situation.

0 Karma

SplunkTrust
SplunkTrust

Have a look at below parameters in inputs.conf on UF and try this config (I didn't test this parameter so not sure how it reacts)

multiline_event_extra_waittime = <boolean>
* By default, the file monitor sends an event delimiter when:
  * It reaches EOF of a file it monitors and
  * Ihe last character it reads is a newline.
* In some cases, it takes time for all lines of a multiple-line event to
  arrive.
* Set to "true" to delay sending an event delimiter until the time that the
  file monitor closes the file, as defined by the 'time_before_close' setting,
  to allow all event lines to arrive.
* Default: false.

time_before_close = <integer>
* The amount of time, in seconds, that the file monitor must wait for
  modifications before closing a file after reaching an End-of-File
  (EOF) marker.
* Tells the input not to close files that have been updated in the
  past 'time_before_close' seconds.
* Default: 3.
0 Karma

Builder

@harsmarvania57 : Thanks I have added this to my UF but seems like not working.

0 Karma

SplunkTrust
SplunkTrust

Are you sure that gap is only 1-2 seconds for 2nd part of multi line events ? If it's more than that then you need to increase time_before_close as well.

0 Karma

Builder

yeah I tried tailing the log file and it was less than 3 secs. but I am going to try with increasing wait time to 5 sec.

0 Karma

SplunkTrust
SplunkTrust

use TIMEPREFIX and TIMEFORMAT as well, if you use should_linemerge

0 Karma

SplunkTrust
SplunkTrust

I am breaking logs at timestamp only using LINEBREAKER then is it really require `TIMEFORMATandTIME_PREFIX` because splunk is automatically detecting TIMESTAMP correctly.

0 Karma

SplunkTrust
SplunkTrust

Adding them will take away the default processing and evalate quickly and is also part of the best practices, when we manage multiline event with line breaker

Champion

Try to use SHOULD_LINEMERGE = true in props.conf.

0 Karma

Builder

Thanks for reply. but i think problem is not with regex. these files are reading live and Full GC events are getting printed in 2 parts.There is a small 1-2 sec gap between printing line. So Splunk picks half part first and 2nd part later. Is there anything can be done in such situation.

0 Karma

Ultra Champion

Yes, add the following to your inputs.conf: multiline_event_extra_waittime = true

0 Karma

Builder

@FrankVl : Thanks I have added this in my UNIVERSAL forwarder but not working.

[monitor:///net/dell730srv/dell730srv1/apps/LuasMaster/logs.../.log]
disabled = false
host = LUAS20190101
index = mlc
live
sourcetype = G1BETA
multiline
eventextrawaittime = true
crcSalt =
whitelist = .
gc.log$|.gc..log$
blacklist=logs|fixing|tps-archives

0 Karma

Ultra Champion

I assume you restarted after that change? Could be that the issue then still is with the actual linebreaking config itself.

0 Karma

Builder

Yeah I have restarted forwarder after this change. I have tried with manual upload of full file and events are break down as per requirement.

I ll try to set higher value of "timebeforeclose = "

0 Karma

New Member

Try to set SHOULD_LINEMERGE to "true"

0 Karma