Getting Data In

multiline events using line merge weird splitting issue

cppforlife
New Member

Hello, I have a big log file that is set to be sourcetype=my_log and it basically looks like this:

--- begin_request ---
blah blah
blah blah
--- end_request ---
--- begin_request ---
blah blah
--- end_request ---

and so on. With props.conf configuration below events are correctly split most of the time but sometimes they are just split somewhere in the middle. This happens to small and large events.

[my_log]
BREAK_ONLY_BEFORE_DATE = False
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = --- begin_request ---
MUST_NOT_BREAK_AFTER = --- begin_request ---
MUST_BREAK_AFTER = --- end_request ---
MAX_EVENTS = 10000
TRUNCATE = 100000

App that produces my_log uses a buffered logger so there are random and periodic delays of how log data is flushed into the log file and it is definitely possible that a single event will not be flushed to disk as a whole. That might be the cause for the strange event splits but I'm not sure.

Any ideas?

Tags (2)
0 Karma

araitz
Splunk Employee
Splunk Employee

If you are using a buffered logger where there may be delays, this is likely the problem. Splunk will flush the file with a _doneKey after a certain time:

http://www.splunk.com/base/Documentation/latest/Admin/Inputsconf

time_before_close = <integer>
* Modtime delta required before Splunk can close a file on EOF.
* Tells the system not to close files that have been updated in past <integer> seconds.
* Defaults to 3.

By the way, the below should be sufficient:

[my_log]
BREAK_ONLY_BEFORE = \-\-\-\sbegin
MAX_EVENTS = 10000
TRUNCATE = 100000
0 Karma

Ayn
Legend

No solution to the breaking problem, but have you considered using transactions to achieve the same thing? Have Splunk index the events line by line but group them together as transactions using startswith="--- begin_request ---" and endswith="--- end_request ---".

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...