topic Re: line breaking issue, please help with props.com in Getting Data In

line breaking issue, please help with props.com

sim_tcr — Thu, 13 Oct 2016 10:32:41 GMT

Hello,

In our log, every new event starts with below pattern,

Sunday 2016-10-09 12:02:46,047 [tomcat-http--9]

Currently line breaking is not happening correctlly.
the props.conf on the indexer look like below,

[ecapi_log4j_2]
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
MAX_TIMESTAMP_LOOKAHEAD = 26
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?:\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3})
TRUNCATE = 100000

Can some one help to fix it please?

Thanks,
Simon Mandy

Re: line breaking issue, please help with props.com

diogofgm — Thu, 13 Oct 2016 10:45:21 GMT

Can you try this?

[ecapi_log4j_2]
TIME_PREFIX = \s
 TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
 MAX_TIMESTAMP_LOOKAHEAD = 23
 SHOULD_LINEMERGE = false
 LINE_BREAKER = ([\r\n]+)([\w]+\s)(\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3})
 TRUNCATE = 100000

Re: line breaking issue, please help with props.com

sim_tcr — Thu, 13 Oct 2016 10:54:41 GMT

That did not work. still multiple events getting clubbed together as one event in splunk.
These events are so large.
our splunkd.log has below many times,

10-13-2016 06:43:14.649 -0400 WARN  LineBreakingProcessor - Truncating line because limit of 100000 bytes has been exceeded with a line length >= 131072 - data_source="<source masked>", data_host="<host masked>", data_sourcetype="ecapi_log4j_2"

Re: line breaking issue, please help with props.com

gcusello — Thu, 13 Oct 2016 11:10:00 GMT

I don't know if "Sunday 2016-10-09 12:02:46,047 [tomcat-http--9]" is an example of a full event or the beginning of it.

Every way, you could import events using the following Time format
%A %Y-%m-%d %H:%M:%S,%3N
Splunk recognizes timestamp as event start and truncate before it.

If your example is only the beginning of an event you have to modify only SHOULD_LINEMERGE.

Bye.
Giuseppe

Re: line breaking issue, please help with props.com

jkat54 — Thu, 13 Oct 2016 11:19:09 GMT

Change truncate = 100000 to truncate=250000

Re: line breaking issue, please help with props.com

sim_tcr — Thu, 13 Oct 2016 12:59:19 GMT

"Sunday 2016-10-09 12:02:46,047 [tomcat-http--9]" is the beginning of the event
I tried below and did not work

[ecapi_log4j_2]
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
MAX_TIMESTAMP_LOOKAHEAD = 26
**SHOULD_LINEMERGE = true**
LINE_BREAKER = ([\r\n]+)(?:\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3})
TRUNCATE = 200000

Re: line breaking issue, please help with props.com

gcusello — Tue, 29 Sep 2020 11:25:02 GMT

Did you tried to modify TIME_FORMAT in %A %Y-%m-%d %H:%M:%S,%3N and don't use LINE_BREAKER and TRUNCATE?

I suggest to you to save an example of your logs in a file and try to ingest it using the web interface.
In this way you can adjust your props and viewing the results at the same time.

Bye.
Giuseppe

Re: line breaking issue, please help with props.com

twinspop — Thu, 13 Oct 2016 13:24:46 GMT

[ecapi_log4j_2]
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
MAX_TIMESTAMP_LOOKAHEAD = 40
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)[SMTWF][a-z]+\s+\d{4}-\d{2}-\d{2}
TRUNCATE = 100000

Re: line breaking issue, please help with props.com

lukejadamec — Thu, 13 Oct 2016 14:35:45 GMT

Have you tried the config on the forwarder? The data may already be parsed by the time it hits the indexer.

Re: line breaking issue, please help with props.com

sim_tcr — Fri, 14 Oct 2016 12:47:42 GMT

that did not work either.

Re: line breaking issue, please help with props.com

jkat54 — Fri, 14 Oct 2016 13:27:10 GMT

Change truncate = 100000 to a higher number like truncate=250000