Getting Data In

Why is LINE_BREAKER not always separating?

meburbo
New Member

I have a log that starts each event by a new line starting with a timestamp followed by a space and pipe, like the following:

2016-04-01T02:55:24.030 |

I have tried setting up props.conf with a new sourcetype, first trying BREAK_ONLY_BEFORE, then LINE_BREAKER. They both seem to work most of the time, but I am still finding chunks of several events lumped into one. I inspected the clumping events in the logs manually, expecting missing line end characters or some other red flag, but haven't found any real reason why the line break isn't working.

BREAK_ONLY_BEFORE:
[newsource]
BREAK_ONLY_BEFORE=([0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]*\s+|)
SHOULD_LINEMERGE=true
MAX_EVENTS=1000000
TRUNCATE=0
(Note: I have also tried this without the regex wrapped as a group)

LINE_BREAKER:
[newsource]
LINE_BREAKER=([\r\n]+)([0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]*\s+|)
SHOULD_LINEMERGE=false
TRUNCATE=0
(I have also tried this without the timestamp portion group-wrapped)

Any ideas why I'm getting these random clusters of events which should be separated?
Thanks!

(Using Splunk universal forwarder 6.2.3)

0 Karma

somesoni2
Revered Legend

Give this atry

[yoursourcetype]
LINE_BREAKER = ([\r\n]+)\d+-\d+-\d+T\d+:\d+:\d+\.\d+\s+\|
SHOULD_LINEMERGE=true
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%N
MAX_TIMESTAMP_LOOKAHEAD = 23
0 Karma

meburbo
New Member

That seems to have the same effect

0 Karma

meburbo
New Member

Turns out both were correct. I assumed that queried events in the UI meant that indexing was complete. Looks like the events were not fully processed when I was checking on them. Apologies for the time waste.

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Best Practices: Splunk auto adjust pipeline queue

When you enable autoAdjustQueue in Splunk, maxSize should be understood as the queue size Splunk starts with ...

Introducing the 2026 - 2027 SplunkTrust cohort!

The goal of the SplunkTrust™ membership has historically been to acknowledge and recognize those who go above ...

Splunk Auto Ingestion Parallel Pipeline Scaling

Why this feature matters Many Splunk environments experience ingestion pressure long before the host is fully ...