My log file is like this:
[#|2019-01-31 11:04:34,712 | ERROR | some data Logging important message |#]
In my props.conf(SplunkUniversalForwarder\etc\system\default), I have tried the following options:
[source::...\\data\\log\\*.log] sourcetype = appcustom
BREAK_ONLY_BEFORE =|#] SHOULD_LINEMERGE =true
SHOULD_LINEMERGE =false LINE_BREAKER=([\r\n]+)(|#])([\r\n]+)
MUST_NOT_BREAK_AFTER=^[# MUST_BREAK_AFTER=#] SHOULD_LINEMERGE= true
If my log file contains all the lines from "[#|" to "|#]" PRIOR TO Splunk seeing the file for the first time, then the lines are successfully indexed as multiline events as follows:
[#|2019-01-31 11:04:34,712 | ERROR | some data Logging important message |#] However If my application is still in the process of writing to the log file, and Splunk indexes the file, then event-breaking is totally messed up. For example, my app would flush the following lines to the log at first: [#|2019-01-31 11:04:34,712 | ERROR | some and then couple of seconds later these additional lines are appended: data Logging important message |#]
In this case I would get these events instead:
[#|2019-01-31 11:04:34,712 | ERROR | some
data Logging important message |#]
The first and second events in this case should really have been combined as a single event, but during index time, Splunk "closed" the first event prematurely because my app had stopped writing to the log temporarily.
I want to see only one multiline event regardless of when or what my app writes to the log file. Is this possible via configuration changes in Splunk?
You should add this configuration to the props.conf on the indexers (or the heavy forwarders if you are sending your data through a a heavy forwarder)
[appcustom] SHOULD_LINEMERGE=false LINE_BREAKER= ([\r\n]+)\[#
I hope this helps
@sduraisamy : best practice is to NOT touch any file under /etc/system/default, you can either make changes to /etc/system/local or create a custom-app if needed...
Above configs do not work on forwarders, you should configure them on indexers...
this should work on your indexers...
props.conf [appcustom] SHOULD_LINEMERGE = true NO_BINARY_CHECK = true TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N TIME_PREFIX = ^\[\#\| MAX_TIMESTAMP_LOOKAHEAD = 23
-if you think splunk is closing the file while the log is still updating, try something like this in your inputs.conf on your forwarder...
time_before_close = <integer> * The amount of time, in seconds, that the file monitor must wait for modifications before closing a file after reaching an End-of-File (EOF) marker. * Tells the input not to close files that have been updated in the past 'time_before_close' seconds. * Default: 3.