topic Re: Linebreaking not working as expected in Getting Data In

Linebreaking not working as expected

dpatnam — Mon, 28 Sep 2020 12:14:20 GMT

I have a logfile whose events are not being broken up in Splunk. Here are the two separate events that are being shown together in Splunk console.

16:45:12,772 INFO> intro_response.pl:549 main:: - Batch AAAIE120809004119P03 successfully transferred to staging server.
16:45:12,774 INFO> intro_response.pl:568 main:: - account=act,program=932,admin=opsprg12,pgmssn=932-574,wfstate='BATCH_PUBLISHED',subject=Math,grade=11,error_code='',msg='Batch published to ePEN',batchnum=AAAIE120809004119P03,batch_count=5

Here's the configuration I have in props.conf for this logfile -

TIME_FORMAT = %H:%i:%s
SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE = \d+:\d+:\d+\,\d+
MAX_EVENTS = 2000

This configuration was working fine earlier but it stopped working for some reason this week. Any help on this would be greatly appreciated.

Re: Linebreaking not working as expected

kbecker — Mon, 28 Sep 2020 12:14:25 GMT

Give this a shot...

TIME_PREFIX=^
TIME_FORMAT=%H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD=8
BREAK_ONLY_BEFORE=^/d{2}/:/d{2}/:/d{2}

Re: Linebreaking not working as expected

dpatnam — Fri, 10 Aug 2012 04:56:17 GMT

Thank you kbecker for your response. Just tried your suggestion. No luck though. I am still seeing separate events getting bundled up.

Re: Linebreaking not working as expected

lguinn2 — Fri, 10 Aug 2012 07:21:03 GMT

Since you have SHOULD_LINEMERGE=false, that implies that Splunk is not seeing your line break character properly. The BREAK_ONLY_BEFORE is not used when SHOULD_LINEMERGE=false. And MAX_EVENTS should be removed - MAX_EVENTS is the maximum number of lines per event - when you set SHOULD_LINEMERGE=false, that is irrelevant because an event can have only one line.

From Configure event linebreaking:

"Splunk determines event boundaries in two steps:

Line breaking, which uses the LINE_BREAKER attribute's regex value to split the incoming stream of bytes into separate lines. By default, the LINE_BREAKER is any sequence of newlines and carriage returns (that is, ([\r\n]+))."

So this is the default:

LINE_BREAKER=[\r\n]+

Is it possible that your line is actually separated by different characters in the log? Or, try this explicitly

TIME_FORMAT = %H:%i:%s
SHOULD_LINEMERGE = false
LINE_BREAKER=[\r\n]+

Re: Linebreaking not working as expected

kristian_kolb — Mon, 13 Aug 2012 15:10:29 GMT

Sorry, but what is %i in TIME_FORMAT? Don't you think that %M would be more correct? Or even:

TIME_FORMAT=%H:%M:%S,%3N

which would let you capture the milliseconds as well.