How to create props for the search so that events will break properly in Splunk?

[Vani_26]

This is my sample data:

I need props for this so that events will break properly in Splunk.

Can any one help me to know how the line breaker, time format, time prefix ect to be wriiten and any other are required in props.conf 

quotation-events~~IM~. ABC~CA~Wed Jan 02 23:24:56 EST   2023~A~0.12~0...~2345.78~SM~quotation-events

D0C5A044~~AB~DFR~Mon Jan 01 12:52:14 EST   2022~B~107.45~106.90~123.09~T~2345A1

quotation-events~~IS~;S. ABC~CA~Tue Jan 02 23:24:56 EST   2023~A~0.12~0...~2345.78~SM~quotation-events

V0C5A044~~AB~DFR~Mon Jan 01 12:52:14 EST   2022~B~107.45~106.90~123.09~T~2345A1

quotation-events~~IM~. ADC~BA~Sat Jan 01 13:24:56 EST   2023~A~0.12~0...~2345.78~SM~quotation-events

B0C5A044~~AB~DFR~Mon Jan 01 12:52:14 EST   2022~B~107.45~106.90~123.09~T~2345A1

quotation-events~~IM~. CCC~HA~Sun Jan 01 20:24:56 EST   2023~A~0.12~0...~2345.78~SM~quotation-events

G0C5A044~~AB~DFR~Mon Jan 01 12:52:14 EST   2022~B~107.45~106.90~123.09~T~2345A1

[richgalloway]

What have you tried so far?

Where should the events break?

Assuming there are four events, each beginning with "quotation-events" then these settings should do

[mysourcetype]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)quotation-events
TIME_PREFIX = ~
MAX_TIMESTAMP_LOOKAHEAD = 128
TIME_FORMAT = %a %b %d %H:%M:%S %Z  %Y

[Vani_26]

hi @richgalloway ,

Actually here we have 8 events:
So here we have 2 different events , 1st events is starting with quotation-events and 2nd events starts with  D0C5A044~~.
so i want props for this 2 kinds of events. 
i tried this props but this is not happening correctly,
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n)]+w{8}~~|quotation-events~~
NO_BINARY_CHECK=true
CHARSET=UTF-8
MAX_TIMESTAMP_LOOKAHEAD=75
disabled=false
TIME_FORMAT=%a %b %d %H:%M:%S %Z
TIME_PREFIX=(?:[^~]+~)~(?:[^~]+~){3}
TRUNCATE=99999
ANNOTATE_PUNCT=false

can u please help in this

quotation-events~~IM~. ABC~CA~Wed Jan 02 23:24:56 EST   2023~A~0.12~0...~2345.78~SM~quotation-events

D0C5A044~~AB~DFR~Mon Jan 01 12:52:14 EST   2022~B~107.45~106.90~123.09~T~2345A1

quotation-events~~IS~;S. ABC~CA~Tue Jan 02 23:24:56 EST   2023~A~0.12~0...~2345.78~SM~quotation-events

V0C5A044~~AB~DFR~Mon Jan 01 12:52:14 EST   2022~B~107.45~106.90~123.09~T~2345A1

quotation-events~~IM~. ADC~BA~Sat Jan 01 13:24:56 EST   2023~A~0.12~0...~2345.78~SM~quotation-events

B0C5A044~~AB~DFR~Mon Jan 01 12:52:14 EST   2022~B~107.45~106.90~123.09~T~2345A1

quotation-events~~IM~. CCC~HA~Sun Jan 01 20:24:56 EST   2023~A~0.12~0...~2345.78~SM~quotation-events

G0C5A044~~AB~DFR~Mon Jan 01 12:52:14 EST   2022~B~107.45~106.90~123.09~T~2345A1

[richgalloway]

Perhaps these settings will help.

[mysourcetype]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)[\w-]+~~
TIME_PREFIX = ~
MAX_TIMESTAMP_LOOKAHEAD = 128
TIME_FORMAT = %a %b %d %H:%M:%S %Z  %Y

The TIME_PREFIX setting does not need to be precise.  Once Splunk finds the prefix it will continue scanning up to MAX_TIMESTAMP_LOOKAHEAD characters until it finds text that matches TIME_FORMAT.

[mahesh27]

Hi @richgalloway
i tried the below props but its not working as expected, it is taking up space, because of that for quotation events are not breaking properly all are getting into one event.
please find the attached screenshot

[richgalloway]

The screenshot appears to show successful breaks in the right places.  What am I missing?

FTR, Splunk regular expressions use PCRE syntax rather than JavaScript.  That may change the results a test site shows.

[mahesh27]

hi @richgalloway 
as it is showing correctly but in splunk the quotation events are getting clubbed together not breaking properly.
what could be the reason behind it.

[richgalloway]

It would be more helpful to see the "clubbed-together" events.  Also, please share the following btool output from the indexer/HF.

splunk btool props list mysourcetype

Replace "mysourcetype" with the name of the sourcetype we're talking about.

[Vani_26]

Hi @richgalloway 
1. Please see the output when i run this command in indexer splunk btool props list mysourcetype

[mysourcetype]
ADD_EXTRA_TIME_FIELDS=True
ANNOTATE_PUNCT=True
TIME_FORMAT=%a %b %I:%M:%S %Z %Y
TIME_PREFIX=\w+\~\~\w+\~\w+\~\w+\~
TRANSFORMS=
detect_trailing_nulls=false
disabled=0
maxDist=100
priority=
sourcetype=
termFrequencyWeightedDist=false
AUTO_KV_JSON=true
BREAK_ONLY_BEFORE=
BREAK_ONLY_BEFORE_DATE=True
CHARSET=UTF-8
DATETIME_CONFIG= /etc/timedata.xml
DEPTH_LIMIT=1000
DETERMINE_TIMESTAMP_DATE_WITH_SYSTEM_TIME=false
EVAL-app=app
HEADER_MODE=
LB_CHUNK_BREAKER_TRUNCATE=2000000
LEARN_SOURCETYPE=true
LEARN_MODEL=true
LINE_BREAKER=([\r\n]+)[\w- ]+~~
LINE_BREAKER_LOOKBEHIND=100
MATCH_LIMIT=100000
MAX_DAYS_AGO=2000
MAX_DAYS_HENCE=2
MAX_EVENTS=256
MAX_TIMESTAMP_LOOKAHEAD=50
MUST_BREAK_AFTER=
MUST_NOT_BREAK_AFTER=
MUST_NOT_BREAK_BEFORE=
NO_BINARY_CHECK=true
SEGMENTATION=indexing-title
SEGMENTATION-all=full
SEGMENTATION-inner=inner
SEGMENTATION-outer=outer
SEGMENTATION-raw=none
SEGMENTATION-standard=standard
SHOULD_LINEMERGE=false

2. props.conf in CM

[mysourcetype]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)[\w- ]+~~
NO_BINARY_CHECK=true
CHARSET=UTF-8
MAX_TIMESTAMP_LOOKAHEAD=50
disabled=false
TRUNCATE=99999

3. As shown below the events are breaking kike this:

Event starting
D0C5A044~~AB~DFR~Mon Jan 01 12:52:14 EST 2022~B~107.45~106.90~123.09~T~2345A1
Event ending

Event starting
D0C5A044~~AB~DFR~Mon Jan 01 12:52:14 EST 2022~B~107.45~106.90~123.09~T~2345A1
Event ending

Event starting
D0C5A044~~AB~DFR~Mon Jan 01 12:52:14 EST 2022~B~107.45~106.90~123.09~T~2345A1
Event ending

Event starting
D0C5A044~~AB~DFR~Mon Jan 01 12:52:14 EST 2022~B~107.45~106.90~123.09~T~2345A1
Event ending

Event starting
D0C5A044~~AB~DFR~Mon Jan 01 12:52:14 EST 2022~B~107.45~106.90~123.09~T~2345A1
Event ending

Event starting
quotation-events~~IM~. ABC~CA~Wed Jan 02 23:24:56 EST 2023~A~0.12~0...~2345.78~SM~quotation-events
quotation-events~~IS~;S. ABC~CA~Tue Jan 02 23:24:56 EST 2023~A~0.12~0...~2345.78~SM~quotation-events
quotation-events~~IM~. ADC~BA~Sat Jan 01 13:24:56 EST 2023~A~0.12~0...~2345.78~SM~quotation-events
quotation-events~~IM~. CCC~HA~Sun Jan 01 20:24:56 EST 2023~A~0.12~0...~2345.78~SM~quotation-events
Event ending

Event starting
B0C5A044~~AB~DFR~Mon Jan 01 12:52:14 EST 2022~B~107.45~106.90~123.09~T~2345A1
Event ending

Event starting
quotation-events~~IM~. ABC~CA~Wed Jan 02 23:24:56 EST 2023~A~0.12~0...~2345.78~SM~quotation-events
quotation-events~~IS~;S. ABC~CA~Tue Jan 02 23:24:56 EST 2023~A~0.12~0...~2345.78~SM~quotation-events
quotation-events~~IM~. ADC~BA~Sat Jan 01 13:24:56 EST 2023~A~0.12~0...~2345.78~SM~quotation-events
quotation-events~~IM~. CCC~HA~Sun Jan 01 20:24:56 EST 2023~A~0.12~0...~2345.78~SM~quotation-events
Event ending

Event starting
V0C5A044~~AB~DFR~Mon Jan 01 12:52:14 EST 2022~B~107.45~106.90~123.09~T~2345A11
Event ending



here each and every quotation-event is not breaking into new line, it is coming as one event

[richgalloway · Accepted solution]

Thank you.  Between the btool output and the great show of good and bad events, I may have found the problem.  There appears to be a space after \w- in the LINE_BREAKER setting.  Because the hyphen is not the first or last character in the set it is treated as a range separator rather than a literal character.  Since '-' is not a word character, "quotation-events" does not match the regex.  Remove the space (or copy-paste from below) and line breaking should work.

[mysourcetype]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)[\w-]+~~
TIME_PREFIX = ~
MAX_TIMESTAMP_LOOKAHEAD = 128
TIME_FORMAT = %a %b %d %H:%M:%S %Z  %Y

[Vani_26]

hi @richgalloway 
my soucetype was duplicated, that's y i could not get the expected results.
now issue resolved.
thank you for your help on this.

[Vani_26]

Hi @richgalloway ,
still the same issue, events are not breaking properly.

[Vani_26]

hi all,
can any one help me on this props please