line break help with incoming logging data

[joesrepsol]

I've tried several attempts at fixing this incoming line data. Looks to be pretty straight forward, but splunk is ingesting these lines as a single event, not breaking them up into individual events. Date looks pretty clean, but timestamp might not be being pickup as a separator? Need some props.conf/REGEX help to get this to split correctly.

Thanks in advance!!!

Joe

sourcetype=release_mcbatch <--UNIQUE

Sample Logging (should be 10 events):
13:15:24.817 INFO com.dstoutput.dp.util.DPConnection - DataBase Connection created.
13:15:24.933 INFO c.d.d.p.RelMgrDocUpdateProcessor - JOB :: no more pending relMgrDocUpdate records in RecordProcessQueue table.
13:15:24.933 INFO c.d.d.p.RelMgrDocUpdateProcessor - JOB :: release manager document update processor successfully completed.
13:15:24.935 INFO c.d.dp.processor.DPProcessor - DataBase Connection closed.
13:30:15.730 INFO c.d.d.p.RelMgrDocUpdateProcessor - JOB :: release manager document update processor started.
13:30:15.764 INFO c.d.d.p.RelMgrDocUpdateProcessor - fetching all pending relMgrDocUpdate records from RecordProcessQueue table
13:30:16.496 INFO com.dstoutput.dp.util.DPConnection - DataBase Connection created.
13:30:16.801 INFO c.d.d.p.RelMgrDocUpdateProcessor - JOB :: no more pending relMgrDocUpdate records in RecordProcessQueue table.
13:30:16.801 INFO c.d.d.p.RelMgrDocUpdateProcessor - JOB :: release manager document update processor successfully completed.
13:30:16.803 INFO c.d.dp.processor.DPProcessor - DataBase Connection closed.

[woodcock]

The problem that you are having may be more with your methodology than with your settings. The answers given should all work fine but you MUST:
1: Deploy the settings to ALL of your Indexers (or Heavy Forwarders, if they get the data first).
2: Restart all splunk instances on the servers where the settings files where deployed.
3: Verify by checking ONLY events that were indexed AFTER the restarts (old events will stay "bad").

[cpetterborg · Accepted solution]

@somesoni2 has provided a great answer, and he finished before I did, but I figured I'd add this to the answers anyway.

Assuming that you only want one line per event, the following in your props.conf file should work:

[release_mcbatch]
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
TIME_FORMAT=%T.%N
TIME_PREFIX=^
MAX_TIMESTAMP_LOOKAHEAD=12

But since that may not be the case, you will want to define a line breaking characteristic for your events. Though many people like to use LINE_BREAKER, I'm fine with using BREAK_ONLY_BEFORE, which is most commonly used when you go through Splunk's Add Data link. I would put in the props.conf:

[release_mcbatch]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
BREAK_ONLY_BEFORE=^\d\d:\d\d:\d\d\.\d\d\d\s
TIME_FORMAT=%T.%N
TIME_PREFIX=^
MAX_TIMESTAMP_LOOKAHEAD=12

Remember that this sort of thing goes in a props.conf file on your indexers or heavy forwarders, not the universal forwarders.

[joesrepsol]

Appreciate all the help everyone! Trying to learn more about these props.conf parameters... always messes me up. Thanks again!

[joesrepsol]

I used the revised answer from cpetterborg, which works exactly the way I wanted it to. The 1st answer may have worked as well, so I just picked 1 and now I'm back up and running.

 [release_mcbatch]
 SHOULD_LINEMERGE=true
 NO_BINARY_CHECK=true
 BREAK_ONLY_BEFORE=^\d\d:\d\d:\d\d\.\d\d\d\s
 TIME_FORMAT=%T.%N
 TIME_PREFIX=^
 MAX_TIMESTAMP_LOOKAHEAD=12

[cpetterborg]

If one of these answers has worked for you, please accept the answer so that others will know that the question has been answered satisfactorily. You can also "up vote" answers that you also have found helpful, since only one answer can be "accepted."

[somesoni2]

Did you setup event parsing configuration for your sourcetype in props.conf on your Indexer/Heavy forwarders?? If not, you'd need to do that with configuration like this (would need to restart splunkd)

props.conf

[release_mcbatch]
SHOULD_LINEMERGE=false
LINE_BREAKER = ([\r\n]+)(?=\d{2}\:\d{2}\:\d{2}\.\d{3})
TIME_PREFIX = ^
TIME_FORMAT = %H:%M:%S.%3N
MAX_TIMESTAMP_LOOKAHEAD = 12

[groland]

I don't know why you are not the top answer, this is perfect and more performant than the BREAK_ONLY_BEFORE.
I like the usage of the positive lookahead on the regex, that is exactly what I was looking for.