Getting Data In

How to refine our sourcetype configuration for proper line breaking of events that contain multiple date values?

burwell
SplunkTrust
SplunkTrust

We are having problems parsing lines with timestamps at the beginning of the line but then there are other fields that are also dates.

We are using Splunk 6.4.2 by the way and MOST of the time the lines are parsed correctly but not every time.

Here's a sample event line

2016-10-28 00:11:28 Foo table info: table = MyTable, baselineDate = 2016-10-27 23:00:00, baselineOldDate = 2016-10-26 23:00:00, baselineSize = 596503557, baselineOldSize = 596446556, frequency = 1 day, 0:00:00, previousDate = 2016-10-27 19:00:00, penultimateDate = 2016-10-26 19:00:00,

And here is the sourcetype definition on the indexers

SHOULD_LINEMERGE        = True
TIME_PREFIX             = ^
TIME_FORMAT             = %Y-%m-%d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 20
LINE_BREAKER            =  ([\r\n]+)
TRUNCATE                =  10000
BREAK_ONLY_BEFORE       = ^\d\d\d\d-\d+-\d+\s+\d\d:\d\d:\d\d
NO_BINARY_CHECK         = True
TZ                      = UTC

The source is:

2016-10-28 00:11:28 Foo table info: table = MyTable, baselineDate = 2016-10-27 23:00:00, baselineOldDate = 2016-10-26 23:00:00, baselineSize = 596503557, baselineOldSize = 596446556, frequency = 1 day, 0:00:00, previousDate = **2016-10-27 19:00:00**, penultimateDate = 2016-10-26 19:00:00,

What happens is that I occasionally get two events instead of one for that line

2016-10-28 00:11:28 Foo table info: table = MyTable, baselineDate = 2016-10-27 23:00:00, baselineOldDate = 2016-10-26 23:00:00, baselineSize = 596503557, baselineOldSize = 596446556, frequency = 1 day, 0:00:00, previousDate = 2016-10-27 19

0, penultimateDate = 2016-10-21 00:00:00,

So what is happening is that most comes through but starting at previousDate = 2016-10-27 19:00:00 this is split and we only get up to "19" on one line and then the next event starts with a zero. So Splunk is splitting the line.

Any refinements on my sourcetype?

0 Karma

twinspop
Influencer

You have both BREAK_ONLY_BEFORE and LINE_BREAKER defined. Ditch BREAK_ONLY_BEFORE and change SHOULD_LINEMERGE to false.

0 Karma
Get Updates on the Splunk Community!

Updated Team Landing Page in Splunk Observability

We’re making some changes to the team landing page in Splunk Observability, based on your feedback. The ...

New! Splunk Observability Search Enhancements for Splunk APM Services/Traces and ...

Regardless of where you are in Splunk Observability, you can search for relevant APM targets including service ...

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...