Getting Data In

Cannot merge events MUST NOT BREAK BEFORE not sticking.

Hello!

Our application creates a log file a day. In the log file, every line is divided into a separate event. I am trying to have Splunk merge all the lines into one event. Simple right? Not in my case apparently.

At the end of the log is this text: Batch tasks have been completed. To finish press any key.

Example:

"Upload of C:\OESP_DATA\Feeds\Daily\MOF\request\ESPIncReq_P_3119_20160826_T014444.xml.ent succeeded
Finished building request for MOF.........
Finished putting files........
Batch tasks have been completed. To finish press any key."

So I have added this stanza to my props.conf on the indexer:

[wrkflowsched_log]
SHOULD_LINEMERGE = True
MUST_NOT_BREAK_BEFORE = Batch tasks have been completed

I have also tried this regex for the MUST_NOT_BREAK_BEFORE statement:
MUST_NOT_BREAK_BEFORE = /Batch tasks have been completed. To finish press any key/
(Which seems to match up at regexr.com)

wrkflowsched_log is the sourcetype

I then proceed to restart the indexer and write a new file in the targeted log directory on the source. I write some text, save it and then write another line and save it. Repeatedly every line shows as a new event even though I have not written the Batch tasks have been completed statement.

I have verified there is no props.conf in the local folder on the source.

Any thoughts?

Thanks!

AlexW

0 Karma

Influencer

Interesting. Is the file written a line at a time like that, or all at once in a batch-like fashion? If a line at a time, I might change to break on new line by default, then use transaction to put them back together into one event: ... | transaction maxevents=5000 source Adjust maxevents as required.

Hey Twinspop, this was it. I abondoned the initial approve and grouped the events together with the transaction command using startswith and endswith. Working great so far. Thanks!

0 Karma

Influencer

Awesome. Glad I could help!

0 Karma

Influencer

I'm not sure MNBB is what you want. I'd use LINE_BREAKER instead anyway:

SHOULD_LINEMEREGE = false
LINE_BREAKER = (To finish press any key.")

Note the part of the regex in parens will be consumed and NOT indexed. Adjust as appropriate.

0 Karma

Hi twinspop thanks so much for your suggestion. Unfortunately there is no change:

My new indexer props.conf (and subsequent indexer restart after):

[wrkflowsched_log]
SHOULD_LINEMEREGE = false
LINE_BREAKER = (To finish press any key.)

(i had removed the quotation in your regex as it is not part of the text)

I then created a new log file adding a line, saved it. It appears as an event. I added another line and saved it. It appears as a second event. Same issue.

Interestingly enough, I also created a larger log file with this text:

TEST line 1
TEST line 2
TEST line 3
TEST line 4
TEST line 5
To finish press any key.
Test line 6

This is how the singular event is shown in the Splunk indexer with your props stanza:

TEST line 1
TEST line 2
TEST line 3
TEST line 4
TEST line 5

Test line 6

So it just removed the line. This isn't my goal BUT it gives me hope that the props.conf is actually being read. I also attempted adding your regex to the MNBB statement (thinking the initial problem was my regex) but it changed nothing. Same problem. 1 line = 1 event.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!