topic Re: How to event break on multiple dashes? in Getting Data In

How to event break on multiple dashes?

ryancmiller — Fri, 30 Aug 2019 16:37:50 GMT

With multi-line logs, I am trying to linebreak on an obvious linebreaker of dashes (----------------------------------------------------------). (Note in the below examples it appears to be coming across as a whole line, but it should be like above).

Example log:

ProviderId : 453af5ee-6772-55ce-39b3-0f9307a96b84
EventId : 300
Keywords : 4
Level : Informational
Message : Application information
Opcode : Info
Task : 65234
Version : 0
Payload : Generic information
EventName : InfoInfo
ProcessId : 6528
ThreadId : 12524

Timestamp : 2019-08-30 12:32:50 PM

I've tried various regex expressions, one such as ^(\s+)-+(\s+)$ to break on the line, but the results don't seem to work. Also Splunk seems to interpret the Timestamp as the beginning of the log but it is actually the last part of the log before the linebreak.

In general Splunk will display the events as (note the Timestamp is first, but it should be last):

Example results:

Timestamp : 2019-08-30 12:32:50 PM

ProviderId : 453af5ee-6772-55ce-39b3-0f9307a96b84
EventId : 300
Keywords : 4
...

Re: How to event break on multiple dashes?

mayurr98 — Fri, 30 Aug 2019 17:34:56 GMT

try setting BREAK_ONLY_BEFORE = ProviderId in your props.conf

Re: How to event break on multiple dashes?

ryancmiller — Wed, 30 Sep 2020 02:01:43 GMT

That seems like a good idea actually, but for some reason it is tending to bunch up a lot of events together, or still putting the timestamp at the top. In some cases it does take out the Timestamp and includes it only as part of the Event Time itself (which is fine). I modified the props.conf within \etc\system\local and restarted the service.

The props.conf config looks like this:

[sourceTypeName]
BREAK_ONLY_BEFORE = ProviderId

Splunk Event examples:

... 1 line omitted ...
ProviderId : 453af5ee-6772-55ce-39b3-0f9307a96b84
... 13 lines omitted ...
ProviderId : 453af5ee-6772-55ce-39b3-0f9307a96b84
... 13 lines omitted ...
ProviderId : 453af5ee-6772-55ce-39b3-0f9307a96b84
... 13 lines omitted ...
ProviderId : 453af5ee-6772-55ce-39b3-0f9307a96b84
... 13 lines omitted ...
ProviderId : 453af5ee-6772-55ce-39b3-0f9307a96b84

Re: How to event break on multiple dashes?

mayurr98 — Wed, 30 Sep 2020 02:01:46 GMT

you have only BREAK_ONLY_BEFORE in props.conf for that stanza?

could you share the entire configuration for that sourcetype?

Re: How to event break on multiple dashes?

ryancmiller — Wed, 30 Sep 2020 02:01:51 GMT

Sure, here (from the Splunk UI). They are basically the default settings I believe:

Name Value

CHARSET AUTO
DATETIME_CONFIG [blank]
LINE_BREAKER ([\r\n]+)
NO_BINARY_CHECK true
SHOULD_LINEMERGE true
category Custom
disabled false
pulldown_type true

Re: How to event break on multiple dashes?

ryancmiller — Fri, 30 Aug 2019 18:17:24 GMT

And yes, that is the only line for the stanza. Created the props.conf file specifically for it.

Re: How to event break on multiple dashes?

mayurr98 — Wed, 30 Sep 2020 02:01:54 GMT

why there is LINE_BREAKER? when you have break_only_before
comment all that and try this new :

    [your_sourcetype]
    CHARSET =
    SHOULD_LINEMERGE = true
    NO_BINARY_CHECK = true
    BREAK_ONLY_BEFORE = ProviderId
    TIME_FORMAT = %Y-%m-%d %I:%M:%S %p
    TIME_PREFIX = Timestamp\s:\s

and change it from the backend

Re: How to event break on multiple dashes?

ryancmiller — Tue, 03 Sep 2019 15:28:55 GMT

The above was taken directly from the Splunk UI, which autogenerated that LINE_BREAKER.

In props.conf, it is as you've just described.

Could there be any conflicts?

Re: How to event break on multiple dashes?

ryancmiller — Tue, 03 Sep 2019 15:34:06 GMT

One more point to note, after the dashed line is a blank line. The line could break on the blank line instead of the dashes.

Re: How to event break on multiple dashes?

tscroggins — Wed, 30 Sep 2020 01:58:46 GMT

Try adding MAX_TIMESTAMP_LOOKAHEAD to your props stanza:

MAX_TIMESTAMP_LOOKAHEAD = 320

if the content length varies, use a value appropriate for the variance:

MAX_TIMESTAMP_LOOKAHEAD = 512

I use multiples of 64 on x86-64 "just in case" Splunk allocates this as a separate buffer. Different architectures have different cache line sizes.

Re: How to event break on multiple dashes?

ryancmiller — Tue, 03 Sep 2019 18:09:42 GMT

Could I be doing something wrong with the configuration itself? For example, if I try renaming the sourcetype, the new sourcetype doesn't take affect. Is there an easy way to check out what attributes are being applied?

Re: How to event break on multiple dashes?

ryancmiller — Tue, 10 Sep 2019 13:04:58 GMT

Hi Mayurr, did not want to leave this question open ended, but I am still working on a solution. From working with our teams it seems that these particular settings are being overridden by the Indexer, which I do not have direct access to.