BREAK_ONLY_BEFORE failing for date extraction

[muguniya]

Hi Team,

We have configured props.conf file in indexer to break events before date in specific format (yyyy-mm-dd hh:mm:ss,ms), but its not working.

props.conf settings:
[sourcetype]
BREAK_ONLY_BEFORE = \d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d,\d\d\d
SHOULD_LINEMERGE = true

Sample Events in Log4j File:

Cookie Value##Wed Apr 23 21:02:31 EDT 2014

2014-04-23 10:11:44,000 DEBUG 143.171.102.228 WebContainer : 15- Getting value of source system for first time user.

Cookie Value##Wed Apr 23 21:01:00 EDT 2014

Since ##Cookie Value## contains feature date time stamp we dont want to break events based on Cookie Value. Please let us know how to break events only the event starts with yyyy-mm-dd hh:mm:ss,ms.

Thanks

[linu1988]

Please try this. The problem you are having is there are two timestamps where splunk is able to get the time for the event and trying to break into events

BREAK_ONLY_BEFORE_DATE=true
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=true
TIME_FORMAT=%Y-%m-%d %H:%M:%S,3%q

Thanks

[muguniya]

Yes, your are right, its not taking Cookie TimeStamp. We have analyzed the event and planning to add date time before ##Cookie....Thank you so much for the solution provided.

[muguniya]

Yes, your are right, its not taking Cookie TimeStamp. We are analyzed the event and plannign to add date time before ##Cookie....

Thank you so much for the solution provided.

[linu1988]

is it taking the cookie timestamp? It shouldn't because it's told not to read that time format!!!

[muguniya]

Above suggestion works when we received events as follows

2014-04-23 10:11:44,000 DEBUG 143.171.102.228 WebContainer : 15- Getting value of source system for first time user.
Cookie Value##Wed Apr 23 21:01:00 EDT 2014

it fails when we receive only below event, we dont want this event populated with future time stamp in splunk
Cookie Value##Wed Apr 23 23:01:00 EDT 2014.

Thanks

[somesoni2 · Accepted solution]

Try this in you props.conf (under your sourcetype)

[YourSourcetype]
BREAK_ONLY_BEFORE = ^\d\d\d\d-\d\d-\d\d\s\d\d:\d\d:\d\d,\d\d\d
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = true
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3Q

----------------------Update------

This seems to work with my sample data (including events with just below, takes time from event itself)
Cookie Value##Wed Apr 23 23:01:00 EDT 2014.

[YourSourcetype]
BREAK_ONLY_BEFORE = ^\d\d\d\d-\d\d-\d\d\s\d\d:\d\d:\d\d,\d\d\d
MAX_TIMESTAMP_LOOKAHEAD = 150
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = true
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3Q
pulldown_type = 1

[linu1988]

Do you event like this in splunk?

2014-04-23 10:11:44,000 DEBUG 143.171.102.228 WebContainer : 15- Getting value of source system for first time user.
Cookie Value##Wed Apr 23 21:01:00 EDT 2014

?

[muguniya]

BREAK_ONLY_BEFORE = ^\d\d\d\d-\d\d-\d\d\s\d\d:\d\d:\d\d,\d\d\d

back slash is being discarded so updated the string again

[muguniya]

We have used BREAK_ONLY_BEFORE as shown below.

BREAK_ONLY_BEFORE = ^\d\d\d\d-\d\d-\d\d\s\d\d:\d\d:\d\d,\d\d\d

[muguniya]

Hi,

No luck, still event is getting break when we receive event ##Cookie Value##Wed Apr 23 20:41:00 EDT.

props.conf settings with above suggestion:
[sourcetype]
BREAK_ONLY_BEFORE = ^\d\d\d\d-\d\d-\d\d\s\d\d:\d\d:\d\d,\d\d\d
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = true
maxDist = 75
pulldown_type = 1

Is there any way we can look for date in specific format and break the event when matched?

Thanks

[muguniya]

We have tried above said settings, its not working when we received an event like ##Cookie Value##Wed Apr 23 21:01:00 EDT 2014. Splunk treats this as a sperate event.

Please let us know for more information.

[somesoni2]

Try this
BREAK_ONLY_BEFORE = ^\d\d\d\d-\d\d-\d\d\s\d\d:\d\d:\d\d,\d\d\d