topic Re: Merging multiline events in Getting Data In

Merging multiline events

sjloh17 — Thu, 04 Nov 2010 10:06:59 GMT

Greetings!

I am trying to merge 2 lines into 1 event but having problems. Appreciate advice on my steps taken

Sample Log - Using logger command to push this to splunk via syslog. Hence the append of timestamp (and splunk treats that as separate events)

Nov 1 12:12:12 event login command
Nov 1 12:12:12 ----- ----- -----------
Nov 1 12:12:13 USER_Create admin2 mkuser
Nov 1 12:12:13 tstadmin
Nov 1 12:12:13 USER_Create admin2 mkuser
Nov 1 12:12:13 tstadmin
Nov 1 12:12:13 USER_Remove admin2 rmuser
Nov 1 12:12:13 tstadmin
Nov 1 12:12:13 USER_Remove admin2 rmuser
Nov 1 12:12:13 tstadmin
Objective - I would like to group the following as 1 event to show the account created "tstadmin"

Nov 1 12:12:13 USER_Create admin2 mkuser
Nov 1 12:12:13 tstadmin
Added the following stanza to \etc\system\local\props.conf

[source::c:\test-multiline.txt]
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE = USER_*
Restarted Splunk

Re: Merging multiline events

bwooden — Thu, 04 Nov 2010 11:52:49 GMT

Using your current props.conf, you should be able to replace the *SHOULD_LINEMERGE* line with this:

BREAK_ONLY_BEFORE = \w{3}\s\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}\sUSER

Replacing both of the lines with something like this would be more efficient:

MAX_TIMESTAMP_LOOKAHEAD = 16
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\w{3}\s\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}\sUSER)

Re: Merging multiline events

sjloh17 — Mon, 28 Sep 2020 09:20:47 GMT

Thanks, can you help me understand why we need
BREAK_ONLY_BEFORE = \w{3}\s\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}\sUSER?

Re: Merging multiline events

sjloh17 — Tue, 16 Nov 2010 17:31:09 GMT

I've managed to make it working by updating \etc\system\local\props.conf

[multiline.txt]

SHOULD_LINEMERGE = True

BREAK_ONLY_BEFORE = USER_*

However, taking one step forward, I'm trying to use a transform to change the name of the sourcetype. Hence, my \etc\system\local\props.conf looks like the following

[multiline.txt]

TRANSFORM-changesourcetype = setsourcetype

[UserAccount]

SHOULD_LINEMERGE = True

BREAK_ONLY_BEFORE = USER_*

While my \etc\system\local\transforms.conf looks like this

[setsourcetype]

DEST_KEY = MetaData:Sourcetype

REGEX = USERACCOUNT

FORMAT = sourcetype::UserAccount

** I'm able to see the new sourcetype being created. However, the line merge is failing to work now!! I suspect its because when Splunk reads props.conf, the UserAccount sourcetype has not been created yet so it wasnt able to set the BREAK_ONLY_BEFORE field.

Can someone confirm my fears?? And how can this be overcome?

Re: Merging multiline events

bwooden — Mon, 28 Sep 2020 09:20:56 GMT

In the BREAK_ONLY_BEFORE example, that regular expression lets Splunk know that any line with a timestamp followed by USER is the beginning of a new event. I recommend using the LINE_BREAKER option though, per original response.

Re: Merging multiline events

sjloh17 — Thu, 18 Nov 2010 10:04:34 GMT

I see, thanks for the info. Moving forward, I'm trying to separate the specific syslog messages from the rest (via assigning a new sourcetype). However, it appears that setting sourcetype at index time via TRANSFORMS does not select rules in props based on the new sourcetype at index time, and it only affects search time. (See post below).

Is there any way this can be done?