With multi-line logs, I am trying to linebreak on an obvious linebreaker of dashes (----------------------------------------------------------). (Note in the below examples it appears to be coming across as a whole line, but it should be like above).
Example log:
ProviderId : 453af5ee-6772-55ce-39b3-0f9307a96b84
EventId : 300
Keywords : 4
Level : Informational
Message : Application information
Opcode : Info
Task : 65234
Version : 0
Payload : Generic information
EventName : InfoInfo
ProcessId : 6528
ThreadId : 12524
I've tried various regex expressions, one such as ^(\s+)-+(\s+)$ to break on the line, but the results don't seem to work. Also Splunk seems to interpret the Timestamp as the beginning of the log but it is actually the last part of the log before the linebreak.
In general Splunk will display the events as (note the Timestamp is first, but it should be last):
Example results:
ProviderId : 453af5ee-6772-55ce-39b3-0f9307a96b84
EventId : 300
Keywords : 4
...
try setting BREAK_ONLY_BEFORE = ProviderId
in your props.conf
That seems like a good idea actually, but for some reason it is tending to bunch up a lot of events together, or still putting the timestamp at the top. In some cases it does take out the Timestamp and includes it only as part of the Event Time itself (which is fine). I modified the props.conf within \etc\system\local and restarted the service.
The props.conf config looks like this:
[sourceTypeName]
BREAK_ONLY_BEFORE = ProviderId
Splunk Event examples:
... 1 line omitted ...
ProviderId : 453af5ee-6772-55ce-39b3-0f9307a96b84
... 13 lines omitted ...
ProviderId : 453af5ee-6772-55ce-39b3-0f9307a96b84
... 13 lines omitted ...
ProviderId : 453af5ee-6772-55ce-39b3-0f9307a96b84
... 13 lines omitted ...
ProviderId : 453af5ee-6772-55ce-39b3-0f9307a96b84
... 13 lines omitted ...
ProviderId : 453af5ee-6772-55ce-39b3-0f9307a96b84
you have only BREAK_ONLY_BEFORE in props.conf for that stanza?
could you share the entire configuration for that sourcetype?
And yes, that is the only line for the stanza. Created the props.conf file specifically for it.
why there is LINE_BREAKER
? when you have break_only_before
comment all that and try this new :
[your_sourcetype]
CHARSET =
SHOULD_LINEMERGE = true
NO_BINARY_CHECK = true
BREAK_ONLY_BEFORE = ProviderId
TIME_FORMAT = %Y-%m-%d %I:%M:%S %p
TIME_PREFIX = Timestamp\s:\s
and change it from the backend
Hi Mayurr, did not want to leave this question open ended, but I am still working on a solution. From working with our teams it seems that these particular settings are being overridden by the Indexer, which I do not have direct access to.
Could I be doing something wrong with the configuration itself? For example, if I try renaming the sourcetype, the new sourcetype doesn't take affect. Is there an easy way to check out what attributes are being applied?
The above was taken directly from the Splunk UI, which autogenerated that LINE_BREAKER.
In props.conf, it is as you've just described.
Could there be any conflicts?
One more point to note, after the dashed line is a blank line. The line could break on the blank line instead of the dashes.
Try adding MAX_TIMESTAMP_LOOKAHEAD to your props stanza:
MAX_TIMESTAMP_LOOKAHEAD = 320
if the content length varies, use a value appropriate for the variance:
MAX_TIMESTAMP_LOOKAHEAD = 512
I use multiples of 64 on x86-64 "just in case" Splunk allocates this as a separate buffer. Different architectures have different cache line sizes.
Sure, here (from the Splunk UI). They are basically the default settings I believe:
Name Value
CHARSET AUTO
DATETIME_CONFIG [blank]
LINE_BREAKER ([\r\n]+)
NO_BINARY_CHECK true
SHOULD_LINEMERGE true
category Custom
disabled false
pulldown_type true