topic Re: BREAK_ONLY_BEFORE failing for date extraction in Splunk Search

BREAK_ONLY_BEFORE failing for date extraction

muguniya — Mon, 28 Sep 2020 16:26:50 GMT

Hi Team,

We have configured props.conf file in indexer to break events before date in specific format (yyyy-mm-dd hh:mm:ss,ms), but its not working.

props.conf settings:
[sourcetype]
BREAK_ONLY_BEFORE = \d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d,\d\d\d
SHOULD_LINEMERGE = true

Sample Events in Log4j File:

Cookie Value##Wed Apr 23 21:02:31 EDT 2014

2014-04-23 10:11:44,000 DEBUG 143.171.102.228 WebContainer : 15- Getting value of source system for first time user.

Cookie Value##Wed Apr 23 21:01:00 EDT 2014

Since ##Cookie Value## contains feature date time stamp we dont want to break events based on Cookie Value. Please let us know how to break events only the event starts with yyyy-mm-dd hh:mm:ss,ms.

Thanks

Re: BREAK_ONLY_BEFORE failing for date extraction

somesoni2 — Mon, 28 Sep 2020 16:26:53 GMT

Try this
BREAK_ONLY_BEFORE = ^\d\d\d\d-\d\d-\d\d\s\d\d:\d\d:\d\d,\d\d\d

Re: BREAK_ONLY_BEFORE failing for date extraction

muguniya — Mon, 28 Sep 2020 16:26:58 GMT

Hi,

No luck, still event is getting break when we receive event ##Cookie Value##Wed Apr 23 20:41:00 EDT.

props.conf settings with above suggestion:
[sourcetype]
BREAK_ONLY_BEFORE = ^\d\d\d\d-\d\d-\d\d\s\d\d:\d\d:\d\d,\d\d\d
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = true
maxDist = 75
pulldown_type = 1

Is there any way we can look for date in specific format and break the event when matched?

Thanks

Re: BREAK_ONLY_BEFORE failing for date extraction

somesoni2 — Wed, 23 Apr 2014 17:39:19 GMT

Try this in you props.conf (under your sourcetype)

[YourSourcetype]
BREAK_ONLY_BEFORE = ^\d\d\d\d-\d\d-\d\d\s\d\d:\d\d:\d\d,\d\d\d
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = true
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3Q

----------------------Update------

This seems to work with my sample data (including events with just below, takes time from event itself)
Cookie Value##Wed Apr 23 23:01:00 EDT 2014.

[YourSourcetype]
BREAK_ONLY_BEFORE = ^\d\d\d\d-\d\d-\d\d\s\d\d:\d\d:\d\d,\d\d\d
MAX_TIMESTAMP_LOOKAHEAD = 150
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = true
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3Q
pulldown_type = 1

Re: BREAK_ONLY_BEFORE failing for date extraction

muguniya — Mon, 28 Sep 2020 16:27:01 GMT

We have used BREAK_ONLY_BEFORE as shown below.

BREAK_ONLY_BEFORE = ^\d\d\d\d-\d\d-\d\d\s\d\d:\d\d:\d\d,\d\d\d

Re: BREAK_ONLY_BEFORE failing for date extraction

muguniya — Wed, 23 Apr 2014 17:39:20 GMT

BREAK_ONLY_BEFORE = ^\d\d\d\d-\d\d-\d\d\s\d\d:\d\d:\d\d,\d\d\d

back slash is being discarded so updated the string again

Re: BREAK_ONLY_BEFORE failing for date extraction

linu1988 — Wed, 23 Apr 2014 18:14:53 GMT

Do you event like this in splunk?

2014-04-23 10:11:44,000 DEBUG 143.171.102.228 WebContainer : 15- Getting value of source system for first time user.
Cookie Value##Wed Apr 23 21:01:00 EDT 2014

Re: BREAK_ONLY_BEFORE failing for date extraction

muguniya — Wed, 23 Apr 2014 18:25:07 GMT

We have tried above said settings, its not working when we received an event like ##Cookie Value##Wed Apr 23 21:01:00 EDT 2014. Splunk treats this as a sperate event.

Please let us know for more information.

Re: BREAK_ONLY_BEFORE failing for date extraction

linu1988 — Wed, 23 Apr 2014 18:28:33 GMT

Please try this. The problem you are having is there are two timestamps where splunk is able to get the time for the event and trying to break into events

BREAK_ONLY_BEFORE_DATE=true
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=true
TIME_FORMAT=%Y-%m-%d %H:%M:%S,3%q

Thanks

Re: BREAK_ONLY_BEFORE failing for date extraction

muguniya — Wed, 23 Apr 2014 20:38:41 GMT

Above suggestion works when we received events as follows

2014-04-23 10:11:44,000 DEBUG 143.171.102.228 WebContainer : 15- Getting value of source system for first time user.
Cookie Value##Wed Apr 23 21:01:00 EDT 2014

it fails when we receive only below event, we dont want this event populated with future time stamp in splunk
Cookie Value##Wed Apr 23 23:01:00 EDT 2014.

Thanks

Re: BREAK_ONLY_BEFORE failing for date extraction

linu1988 — Thu, 24 Apr 2014 18:53:56 GMT

is it taking the cookie timestamp? It shouldn't because it's told not to read that time format!!!

Re: BREAK_ONLY_BEFORE failing for date extraction

muguniya — Thu, 24 Apr 2014 19:41:27 GMT

Yes, your are right, its not taking Cookie TimeStamp. We are analyzed the event and plannign to add date time before ##Cookie....

Thank you so much for the solution provided.

Re: BREAK_ONLY_BEFORE failing for date extraction

muguniya — Thu, 24 Apr 2014 19:42:14 GMT

Yes, your are right, its not taking Cookie TimeStamp. We have analyzed the event and planning to add date time before ##Cookie....Thank you so much for the solution provided.