How do you custom line break multi-line logs in props.conf?

[sduraisamy]

Hi,

My log file is like this:

[#|2019-01-31 11:04:34,712 | ERROR | some data 
Logging important message |#]

In my props.conf(SplunkUniversalForwarder\etc\system\default), I have tried the following options:

[source::...\\data\\log\\*.log]
sourcetype = appcustom

[appcustom]
Option 1

BREAK_ONLY_BEFORE =|#]
SHOULD_LINEMERGE =true

Option 2

SHOULD_LINEMERGE =false
LINE_BREAKER=([\r\n]+)(|#])([\r\n]+)

Option 3

MUST_NOT_BREAK_AFTER=^[#
MUST_BREAK_AFTER=#]
SHOULD_LINEMERGE= true

If my log file contains all the lines from "[#|" to "|#]" PRIOR TO Splunk seeing the file for the first time, then the lines are successfully indexed as multiline events as follows:

First event:

[#|2019-01-31 11:04:34,712 | ERROR | some data 
Logging important message |#]
However If my application is still in the process of writing to the log file, and Splunk indexes the file, then event-breaking is totally messed up. For example, my app would flush the following lines to the log at first:
[#|2019-01-31 11:04:34,712 | ERROR | some 
and then couple of seconds later these additional lines are appended:
data Logging important message |#]

In this case I would get these events instead:

First event:

[#|2019-01-31 11:04:34,712 | ERROR | some 

Second event:

data Logging important message |#]

The first and second events in this case should really have been combined as a single event, but during index time, Splunk "closed" the first event prematurely because my app had stopped writing to the log temporarily.

I want to see only one multiline event regardless of when or what my app writes to the log file. Is this possible via configuration changes in Splunk?

Regards

Senthil

[woodcock]

Always use Option 2 whenever possible. It is way more efficient than the others. So try this:

[appcustom]
SHOULD_LINEMERGE=false
LINE_BREAKER= ([\r\n]+)\[#\|\d{4}

[chrisyounger]

Hi,

You should add this configuration to the props.conf on the indexers (or the heavy forwarders if you are sending your data through a a heavy forwarder)

[appcustom]
SHOULD_LINEMERGE=false
LINE_BREAKER= ([\r\n]+)\[#

I hope this helps

[sduraisamy]

Thanks @chrisyoungerjds. I will try this out on Indexers.

[prakash007]

@sduraisamy : firstly, do not make any changes to /etc/system/default, try to use /etc/system/local.
do you have the props on your indexers as well..??

[sduraisamy]

Thanks @prakash007. No. I am changing the forwarder configuration only. Should i add the configuration changes in /etc/system/local/props.conf?

[prakash007]

@sduraisamy : best practice is to NOT touch any file under /etc/system/default, you can either make changes to /etc/system/local or create a custom-app if needed...
Above configs do not work on forwarders, you should configure them on indexers...
this should work on your indexers...

   props.conf
    [appcustom]
    SHOULD_LINEMERGE = true
    NO_BINARY_CHECK = true
    TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
    TIME_PREFIX = ^\[\#\|
    MAX_TIMESTAMP_LOOKAHEAD = 23

-if you think splunk is closing the file while the log is still updating, try something like this in your inputs.conf on your forwarder...

    time_before_close = <integer>
    * The amount of time, in seconds, that the file monitor must wait for
      modifications before closing a file after reaching an End-of-File
      (EOF) marker.
    * Tells the input not to close files that have been updated in the
      past 'time_before_close' seconds.
    * Default: 3.

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf

[sduraisamy]

Thanks @prakash007. I will try this out on Indexers.