Getting Data In
Highlighted

How do you custom line break multi-line logs in props.conf?

Explorer

Hi,

My log file is like this:

[#|2019-01-31 11:04:34,712 | ERROR | some data 
Logging important message |#]

In my props.conf(SplunkUniversalForwarder\etc\system\default), I have tried the following options:

[source::...\\data\\log\\*.log]
sourcetype = appcustom

[appcustom]
Option 1

BREAK_ONLY_BEFORE =|#]
SHOULD_LINEMERGE =true

Option 2

SHOULD_LINEMERGE =false
LINE_BREAKER=([\r\n]+)(|#])([\r\n]+)

Option 3

MUST_NOT_BREAK_AFTER=^[#
MUST_BREAK_AFTER=#]
SHOULD_LINEMERGE= true

If my log file contains all the lines from "[#|" to "|#]" PRIOR TO Splunk seeing the file for the first time, then the lines are successfully indexed as multiline events as follows:

First event:

[#|2019-01-31 11:04:34,712 | ERROR | some data 
Logging important message |#]
However If my application is still in the process of writing to the log file, and Splunk indexes the file, then event-breaking is totally messed up. For example, my app would flush the following lines to the log at first:
[#|2019-01-31 11:04:34,712 | ERROR | some 
and then couple of seconds later these additional lines are appended:
data Logging important message |#]

In this case I would get these events instead:

First event:

[#|2019-01-31 11:04:34,712 | ERROR | some 

Second event:

data Logging important message |#]

The first and second events in this case should really have been combined as a single event, but during index time, Splunk "closed" the first event prematurely because my app had stopped writing to the log temporarily.

I want to see only one multiline event regardless of when or what my app writes to the log file. Is this possible via configuration changes in Splunk?

Regards

Senthil

Highlighted

Re: How do you custom line break multi-line logs in props.conf?

Builder

@sduraisamy : firstly, do not make any changes to /etc/system/default, try to use /etc/system/local.
do you have the props on your indexers as well..??

0 Karma
Highlighted

Re: How do you custom line break multi-line logs in props.conf?

Explorer

Thanks @prakash007. No. I am changing the forwarder configuration only. Should i add the configuration changes in /etc/system/local/props.conf?

0 Karma
Highlighted

Re: How do you custom line break multi-line logs in props.conf?

Builder

@sduraisamy : best practice is to NOT touch any file under /etc/system/default, you can either make changes to /etc/system/local or create a custom-app if needed...
Above configs do not work on forwarders, you should configure them on indexers...
this should work on your indexers...

   props.conf
    [appcustom]
    SHOULD_LINEMERGE = true
    NO_BINARY_CHECK = true
    TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
    TIME_PREFIX = ^\[\#\|
    MAX_TIMESTAMP_LOOKAHEAD = 23

-if you think splunk is closing the file while the log is still updating, try something like this in your inputs.conf on your forwarder...

    time_before_close = <integer>
    * The amount of time, in seconds, that the file monitor must wait for
      modifications before closing a file after reaching an End-of-File
      (EOF) marker.
    * Tells the input not to close files that have been updated in the
      past 'time_before_close' seconds.
    * Default: 3.

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf

0 Karma
Highlighted

Re: How do you custom line break multi-line logs in props.conf?

Explorer

Thanks @prakash007. I will try this out on Indexers.

0 Karma
Highlighted

Re: How do you custom line break multi-line logs in props.conf?

SplunkTrust
SplunkTrust

Hi,

You should add this configuration to the props.conf on the indexers (or the heavy forwarders if you are sending your data through a a heavy forwarder)

[appcustom]
SHOULD_LINEMERGE=false
LINE_BREAKER= ([\r\n]+)\[#

I hope this helps

Highlighted

Re: How do you custom line break multi-line logs in props.conf?

Explorer

Thanks @chrisyoungerjds. I will try this out on Indexers.

0 Karma
Highlighted

Re: How do you custom line break multi-line logs in props.conf?

Esteemed Legend

Always use Option 2 whenever possible. It is way more efficient than the others. So try this:

[appcustom]
SHOULD_LINEMERGE=false
LINE_BREAKER= ([\r\n]+)\[#\|\d{4}