How to not include a pattern when line merging?

[ricotries]

I have a log from an application that isn't structured in any standard format and I am struggling with dropping certain lines at index time due to the line merging configuration.

This is a pseudo sample of the data:

----- <application version> -----
(<timestamp>) <data> 
(<timestamp>) <data>

----- <application version> -----
(<timestamp>) <data> 
(<timestamp>) <data>

----- <application version> -----
(<timestamp>) <data> 
(<timestamp>) <data>
<data>
<data>
<data>
(<timestamp>) <data>

As you can see, for some events the message is broken down into multiple lines, so the best way to break events would be by the timestamp, so this is the props.conf I wrote for this source type:

[my_new_sourcetype]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE_DATE = true
TRANSFORMS-drop_header = new_sourcetype_drop_header

 And the associated transforms.conf:

[new_sourcetype_drop_header]
REGEX = ^-{5}.+-{5}$
DEST_KEY = queue
FORMAT = nullQueue

The issue becomes that when the data is indexed, any event that would have been the <application version> header by itself is dropped, but then there are events with a linecount of 2 that look like:

(<timestamp>) <data>
----- <application version> -----

How do I force it so that the <application version> header is always made into its own event so that it can be dropped by the transforms configuration?

[richgalloway · Accepted solution]

Rather than breaking only before a date, try breaking before a date OR a header.

[my_new_sourcetype]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(<<timestamp regex>>|-{5})
TRANSFORMS-drop_header = new_sourcetype_drop_header
TIME_FORMAT = <<timestamp regex>>
TIME_PREFIX = \(

Be sure to replace <<timestamp regex>> with a regular expression that matches your timestamp strings.