Splunk Search

BREAK_ONLY_BEFORE failing for date extraction

muguniya
Explorer

Hi Team,

We have configured props.conf file in indexer to break events before date in specific format (yyyy-mm-dd hh:mm:ss,ms), but its not working.

props.conf settings:
[sourcetype]
BREAK_ONLY_BEFORE = \d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d,\d\d\d
SHOULD_LINEMERGE = true

Sample Events in Log4j File:

Cookie Value##Wed Apr 23 21:02:31 EDT 2014

2014-04-23 10:11:44,000 DEBUG 143.171.102.228 WebContainer : 15- Getting value of source system for first time user.

Cookie Value##Wed Apr 23 21:01:00 EDT 2014

Since ##Cookie Value## contains feature date time stamp we dont want to break events based on Cookie Value. Please let us know how to break events only the event starts with yyyy-mm-dd hh:mm:ss,ms.

Thanks

Tags (2)
0 Karma
1 Solution

somesoni2
SplunkTrust
SplunkTrust

Try this in you props.conf (under your sourcetype)

[YourSourcetype]
BREAK_ONLY_BEFORE = ^\d\d\d\d-\d\d-\d\d\s\d\d:\d\d:\d\d,\d\d\d
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = true
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3Q

----------------------Update------

This seems to work with my sample data (including events with just below, takes time from event itself)
Cookie Value##Wed Apr 23 23:01:00 EDT 2014.

[YourSourcetype]
BREAK_ONLY_BEFORE = ^\d\d\d\d-\d\d-\d\d\s\d\d:\d\d:\d\d,\d\d\d
MAX_TIMESTAMP_LOOKAHEAD = 150
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = true
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3Q
pulldown_type = 1

View solution in original post

linu1988
Champion

Please try this. The problem you are having is there are two timestamps where splunk is able to get the time for the event and trying to break into events

BREAK_ONLY_BEFORE_DATE=true
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=true
TIME_FORMAT=%Y-%m-%d %H:%M:%S,3%q

Thanks

0 Karma

muguniya
Explorer

Yes, your are right, its not taking Cookie TimeStamp. We have analyzed the event and planning to add date time before ##Cookie....Thank you so much for the solution provided.

0 Karma

muguniya
Explorer

Yes, your are right, its not taking Cookie TimeStamp. We are analyzed the event and plannign to add date time before ##Cookie....

Thank you so much for the solution provided.

0 Karma

linu1988
Champion

is it taking the cookie timestamp? It shouldn't because it's told not to read that time format!!!

0 Karma

muguniya
Explorer

Above suggestion works when we received events as follows

2014-04-23 10:11:44,000 DEBUG 143.171.102.228 WebContainer : 15- Getting value of source system for first time user.
Cookie Value##Wed Apr 23 21:01:00 EDT 2014

it fails when we receive only below event, we dont want this event populated with future time stamp in splunk
Cookie Value##Wed Apr 23 23:01:00 EDT 2014.

Thanks

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Try this in you props.conf (under your sourcetype)

[YourSourcetype]
BREAK_ONLY_BEFORE = ^\d\d\d\d-\d\d-\d\d\s\d\d:\d\d:\d\d,\d\d\d
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = true
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3Q

----------------------Update------

This seems to work with my sample data (including events with just below, takes time from event itself)
Cookie Value##Wed Apr 23 23:01:00 EDT 2014.

[YourSourcetype]
BREAK_ONLY_BEFORE = ^\d\d\d\d-\d\d-\d\d\s\d\d:\d\d:\d\d,\d\d\d
MAX_TIMESTAMP_LOOKAHEAD = 150
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = true
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3Q
pulldown_type = 1

linu1988
Champion

Do you event like this in splunk?

2014-04-23 10:11:44,000 DEBUG 143.171.102.228 WebContainer : 15- Getting value of source system for first time user.
Cookie Value##Wed Apr 23 21:01:00 EDT 2014

?

0 Karma

muguniya
Explorer

BREAK_ONLY_BEFORE = ^\d\d\d\d-\d\d-\d\d\s\d\d:\d\d:\d\d,\d\d\d

back slash is being discarded so updated the string again

0 Karma

muguniya
Explorer

We have used BREAK_ONLY_BEFORE as shown below.

BREAK_ONLY_BEFORE = ^\d\d\d\d-\d\d-\d\d\s\d\d:\d\d:\d\d,\d\d\d

0 Karma

muguniya
Explorer

Hi,

No luck, still event is getting break when we receive event ##Cookie Value##Wed Apr 23 20:41:00 EDT.

props.conf settings with above suggestion:
[sourcetype]
BREAK_ONLY_BEFORE = ^\d\d\d\d-\d\d-\d\d\s\d\d:\d\d:\d\d,\d\d\d
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = true
maxDist = 75
pulldown_type = 1

Is there any way we can look for date in specific format and break the event when matched?

Thanks

0 Karma

muguniya
Explorer

We have tried above said settings, its not working when we received an event like ##Cookie Value##Wed Apr 23 21:01:00 EDT 2014. Splunk treats this as a sperate event.

Please let us know for more information.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Try this
BREAK_ONLY_BEFORE = ^\d\d\d\d-\d\d-\d\d\s\d\d:\d\d:\d\d,\d\d\d

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...