Getting Data In

How to configure props.conf using BREAK_ONLY_BEFORE?

dturner83
Path Finder

I have the following data as a text file. Each event should run from the Date field until the next date field.

I'm using a universal forwarder to send this data to a heavy forwarder and then on to the indexer. The events either all break or none break depending if I have anything or nothing in the props.conf on the heavy forwarder but I never get event breaking before Date.

Can someone help out here? It appears I need some help with the BREAK_ONLY_BEFORE option.

The following are my props.conf files for the heavy forwarder and indexer/search head.

[sampleoutput]
# your settings
BREAK_ONLY_BEFORE=^\s*Date                                           
NO_BINARY_CHECK=1
SHOULD_LINEMERGE = TRUE

Each item below is on a new line, but should be in the same event, until the Date field shows up again.

Date = 6/24/2014
Ad = item add #1
Description line 1 = Something Good
Description line 2 = Somethinggood2.
Display URL = example.com/somethinggood
Destination URL = http=//example.com
Campaign = Campaign1
Campaign type = Search Only
Campaign subtype = All features
Ad group = addgroup1
Status = disapproved
Clicks = 0
Impressions = 0
CTR = 0.00%
Avg. CPC = 0
Cost = 0
Avg. position = 0
Converted clicks = 0
Cost / converted click = 0
Click conversion rate = 0.00%

Date = 6/24/2014
Ad = item add #2
Description line 1 = Something good
Description line 2 = Something good 2
Display URL = example.com/somethingood
Destination URL = http=//example.com
Campaign = campaign2
Campaign type = Search Only
Campaign subtype = All features
Ad group = addgroup2
Status = disapproved
Clicks = 0
Impressions = 0
CTR = 0.00%
Avg. CPC = 0
Cost = 0
Avg. position = 0
Converted clicks = 0
Cost / converted click = 0
Click conversion rate = 0.00%

martin_mueller
SplunkTrust
SplunkTrust

This is a slightly different approach, but you should be able to use this:

LINE_BREAKER = ([\r\n]+)\s*Date
SHOULD_LINEMERGE = false

Should be much faster as well 😄

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Yeah, any indextime change to configuration files requires a restart, no matter if it's a HF or Indexer.

0 Karma

dturner83
Path Finder

This took care of it. I put this in the props.conf on the heavy forwarder and it didn't change, but then I restarted splunk on the heavy forwarder and it worked like a champ. That now makes me wonder if my other changes would have worked too but I'll take your faster approach 🙂

0 Karma

dturner83
Path Finder

Yes the sourcetype matches, the regex for ^\s*Date is something I've tried as well as ^Date and Date itself.

0 Karma

MuS
SplunkTrust
SplunkTrust

does the sourcetype match and does this regex match? you're using a regex that will match 0 or none spaces at the beginning of the string followed by Date. Did you try to use only BREAK_ONLY_BEFORE=Date ?

0 Karma
Get Updates on the Splunk Community!

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...