Event Line Break

[JagsP]

Hello everyone,

Please check the below data :

ERROR 2024-08-09 14:19:22,707 email-slack-notification-impl-flow.BLOCKING @3372f96f] [processor: email-slack-notification-impl-flow/processors/2/route/0/processors/0; event: 5-03aca501-42b3-11ef-ad89-0a2944cc61cb] error.notification.details: {
"correlationId" : "5-03aca501-42b3-11ef-ad89-0a2944cc61cb",
"message" : "Error Details",
"tracePoint" : "FLOW",
"priority" : "ERROR",

}

ERROR 2024-08-09 14:19:31,389 email-slack-notification-impl-flow.BLOCKING @22feab4f] [processor: email-slack-notification-impl-flow/processors/2/route/0/processors/0; event: 38de9c30-49eb-11ef-8a9e-02cfc6727565] error.notification.details: {
"correlationId" : "38de9c30-49eb-11ef-8a9e-02cfc6727565",
"message" : "Error Details", 
"priority" : "ERROR",

}

The above 2 blocks of data are coming as one event but I want them to be 2 events each starting from keyword "Error".

Below is my props.config entry for same but not working:

applog_test]

DATETIME_CONFIG =

LINE_BREAKER = ([\r\n]+)

NO_BINARY_CHECK = true

category = Custom

disabled = false

pulldown_type = true

BREAK_ONLY_BEFORE = date

SHOULD_LINEMERGE = true

TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N

TIME_PREFIX=ERROR\s+

Please help how to fix this.

Thanks in advance!

[richgalloway · Accepted solution]

Try these settings

[applog_test]
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)ERROR
NO_BINARY_CHECK = true
category = Custom
disabled = false
pulldown_type = true
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
TIME_PREFIX = ERROR\s+

Don't specify BREAK_ONLY_BEFORE_DATE if you want to break at something other than a date.  Also, don't use both BREAK_ONLY_BEFORE_DATE and LINE_BREAKER in the same stanza.  When using LINE_BREAKER, set SHOULD_LINEMERGE to false.

[JagsP]

Thanks @richgalloway The solution worked .

[richgalloway]

If your problem is resolved, then please click the "Accept as Solution" button to help future readers.

[PickleRick]

+1 on that. Whenever possible, don't use SHOULD_LINEMERGE=true. It's a very expensive setting causing Splunk to try to re-merge already split events into bigger ones. While it has some use in very specific border cases as a rule of thumb you should avoid using it completely. That's what proper LINE_BREAKER is for.

[JagsP]

Thanks @richgalloway Trying it out now. will let you know if it works.

[isoutamo]

My presentation about Data Onboarding for Helsinki UG. https://data-findings.com/wp-content/uploads/2024/04/Data-OnBoarding-2024-04-03.pdf 

It contains some hints and workflow how you could test data onboarding on your own workstation.