Getting Data In

How to configure props.conf using BREAK_ONLY_BEFORE?

dturner83
Path Finder

I have the following data as a text file. Each event should run from the Date field until the next date field.

I'm using a universal forwarder to send this data to a heavy forwarder and then on to the indexer. The events either all break or none break depending if I have anything or nothing in the props.conf on the heavy forwarder but I never get event breaking before Date.

Can someone help out here? It appears I need some help with the BREAK_ONLY_BEFORE option.

The following are my props.conf files for the heavy forwarder and indexer/search head.

[sampleoutput]
# your settings
BREAK_ONLY_BEFORE=^\s*Date                                           
NO_BINARY_CHECK=1
SHOULD_LINEMERGE = TRUE

Each item below is on a new line, but should be in the same event, until the Date field shows up again.

Date = 6/24/2014
Ad = item add #1
Description line 1 = Something Good
Description line 2 = Somethinggood2.
Display URL = example.com/somethinggood
Destination URL = http=//example.com
Campaign = Campaign1
Campaign type = Search Only
Campaign subtype = All features
Ad group = addgroup1
Status = disapproved
Clicks = 0
Impressions = 0
CTR = 0.00%
Avg. CPC = 0
Cost = 0
Avg. position = 0
Converted clicks = 0
Cost / converted click = 0
Click conversion rate = 0.00%

Date = 6/24/2014
Ad = item add #2
Description line 1 = Something good
Description line 2 = Something good 2
Display URL = example.com/somethingood
Destination URL = http=//example.com
Campaign = campaign2
Campaign type = Search Only
Campaign subtype = All features
Ad group = addgroup2
Status = disapproved
Clicks = 0
Impressions = 0
CTR = 0.00%
Avg. CPC = 0
Cost = 0
Avg. position = 0
Converted clicks = 0
Cost / converted click = 0
Click conversion rate = 0.00%

martin_mueller
SplunkTrust
SplunkTrust

This is a slightly different approach, but you should be able to use this:

LINE_BREAKER = ([\r\n]+)\s*Date
SHOULD_LINEMERGE = false

Should be much faster as well 😄

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Yeah, any indextime change to configuration files requires a restart, no matter if it's a HF or Indexer.

0 Karma

dturner83
Path Finder

This took care of it. I put this in the props.conf on the heavy forwarder and it didn't change, but then I restarted splunk on the heavy forwarder and it worked like a champ. That now makes me wonder if my other changes would have worked too but I'll take your faster approach 🙂

0 Karma

dturner83
Path Finder

Yes the sourcetype matches, the regex for ^\s*Date is something I've tried as well as ^Date and Date itself.

0 Karma

MuS
SplunkTrust
SplunkTrust

does the sourcetype match and does this regex match? you're using a regex that will match 0 or none spaces at the beginning of the string followed by Date. Did you try to use only BREAK_ONLY_BEFORE=Date ?

0 Karma
Get Updates on the Splunk Community!

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...

What's New in Splunk Cloud Platform 9.0.2208?!

Howdy!  We are happy to share the newest updates in Splunk Cloud Platform 9.0.2208! Analysts can benefit ...

Admin Console: A Single, Unified Interface for All Your Cloud Admin Needs

WATCH NOWJoin us to learn how the admin console can save you time and give you more control over the Splunk® ...