Cannot merge events MUST NOT BREAK BEFORE not sticking.

[alexsambacanada]

Hello!

Our application creates a log file a day. In the log file, every line is divided into a separate event. I am trying to have Splunk merge all the lines into one event. Simple right? Not in my case apparently.

At the end of the log is this text: Batch tasks have been completed. To finish press any key.

Example:

"Upload of C:\OESP_DATA\Feeds\Daily\MOF\request\ESPIncReq_P_3119_20160826_T014444.xml.ent succeeded
Finished building request for MOF.........
Finished putting files........
Batch tasks have been completed. To finish press any key."

So I have added this stanza to my props.conf on the indexer:

[wrkflowsched_log]
SHOULD_LINEMERGE = True
MUST_NOT_BREAK_BEFORE = Batch tasks have been completed

I have also tried this regex for the MUST_NOT_BREAK_BEFORE statement:
MUST_NOT_BREAK_BEFORE = /Batch tasks have been completed. To finish press any key/
(Which seems to match up at regexr.com)

wrkflowsched_log is the sourcetype

I then proceed to restart the indexer and write a new file in the targeted log directory on the source. I write some text, save it and then write another line and save it. Repeatedly every line shows as a new event even though I have not written the Batch tasks have been completed statement.

I have verified there is no props.conf in the local folder on the source.

Any thoughts?

Thanks!

AlexW

[twinspop]

Interesting. Is the file written a line at a time like that, or all at once in a batch-like fashion? If a line at a time, I might change to break on new line by default, then use transaction to put them back together into one event: ... | transaction maxevents=5000 source Adjust maxevents as required.

[alexsambacanada]

Hey Twinspop, this was it. I abondoned the initial approve and grouped the events together with the transaction command using startswith and endswith. Working great so far. Thanks!

[twinspop]

Awesome. Glad I could help!

[twinspop]

I'm not sure MNBB is what you want. I'd use LINE_BREAKER instead anyway:

SHOULD_LINEMEREGE = false
LINE_BREAKER = (To finish press any key.")

Note the part of the regex in parens will be consumed and NOT indexed. Adjust as appropriate.

[alexsambacanada]

Hi twinspop thanks so much for your suggestion. Unfortunately there is no change:

My new indexer props.conf (and subsequent indexer restart after):

[wrkflowsched_log]
SHOULD_LINEMEREGE = false
LINE_BREAKER = (To finish press any key.)

(i had removed the quotation in your regex as it is not part of the text)

I then created a new log file adding a line, saved it. It appears as an event. I added another line and saved it. It appears as a second event. Same issue.

Interestingly enough, I also created a larger log file with this text:

TEST line 1
TEST line 2
TEST line 3
TEST line 4
TEST line 5
To finish press any key.
Test line 6

This is how the singular event is shown in the Splunk indexer with your props stanza:

TEST line 1
TEST line 2
TEST line 3
TEST line 4
TEST line 5

Test line 6

So it just removed the line. This isn't my goal BUT it gives me hope that the props.conf is actually being read. I also attempted adding your regex to the MNBB statement (thinking the initial problem was my regex) but it changed nothing. Same problem. 1 line = 1 event.