Getting Data In

Merging multiline events

sjloh17
Explorer

Greetings!

I am trying to merge 2 lines into 1 event but having problems. Appreciate advice on my steps taken

  1. Sample Log - Using logger command to push this to splunk via syslog. Hence the append of timestamp (and splunk treats that as separate events)

    Nov 1 12:12:12 event login command
    Nov 1 12:12:12 ----- ----- -----------
    Nov 1 12:12:13 USER_Create admin2 mkuser
    Nov 1 12:12:13 tstadmin
    Nov 1 12:12:13 USER_Create admin2 mkuser
    Nov 1 12:12:13 tstadmin
    Nov 1 12:12:13 USER_Remove admin2 rmuser
    Nov 1 12:12:13 tstadmin
    Nov 1 12:12:13 USER_Remove admin2 rmuser
    Nov 1 12:12:13 tstadmin

  2. Objective - I would like to group the following as 1 event to show the account created "tstadmin"

    Nov 1 12:12:13 USER_Create admin2 mkuser
    Nov 1 12:12:13 tstadmin

  3. Added the following stanza to \etc\system\local\props.conf

    [source::c:\test-multiline.txt]
    SHOULD_LINEMERGE = True
    BREAK_ONLY_BEFORE = USER_*

  4. Restarted Splunk

Tags (1)

sjloh17
Explorer

I've managed to make it working by updating \etc\system\local\props.conf

[multiline.txt]

SHOULD_LINEMERGE = True

BREAK_ONLY_BEFORE = USER_*

However, taking one step forward, I'm trying to use a transform to change the name of the sourcetype. Hence, my \etc\system\local\props.conf looks like the following

[multiline.txt]

TRANSFORM-changesourcetype = setsourcetype

[UserAccount]

SHOULD_LINEMERGE = True

BREAK_ONLY_BEFORE = USER_*

While my \etc\system\local\transforms.conf looks like this

[setsourcetype]

DEST_KEY = MetaData:Sourcetype

REGEX = USERACCOUNT

FORMAT = sourcetype::UserAccount

** I'm able to see the new sourcetype being created. However, the line merge is failing to work now!! I suspect its because when Splunk reads props.conf, the UserAccount sourcetype has not been created yet so it wasnt able to set the BREAK_ONLY_BEFORE field.

Can someone confirm my fears?? And how can this be overcome?

0 Karma

bwooden
Splunk Employee
Splunk Employee

Using your current props.conf, you should be able to replace the *SHOULD_LINEMERGE* line with this:

BREAK_ONLY_BEFORE = \w{3}\s\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}\sUSER

Replacing both of the lines with something like this would be more efficient:

MAX_TIMESTAMP_LOOKAHEAD = 16
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\w{3}\s\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}\sUSER)

sjloh17
Explorer

I see, thanks for the info. Moving forward, I'm trying to separate the specific syslog messages from the rest (via assigning a new sourcetype). However, it appears that setting sourcetype at index time via TRANSFORMS does not select rules in props based on the new sourcetype at index time, and it only affects search time. (See post below).

Is there any way this can be done?

0 Karma

bwooden
Splunk Employee
Splunk Employee

In the BREAK_ONLY_BEFORE example, that regular expression lets Splunk know that any line with a timestamp followed by USER is the beginning of a new event. I recommend using the LINE_BREAKER option though, per original response.

0 Karma

sjloh17
Explorer

Thanks, can you help me understand why we need
BREAK_ONLY_BEFORE = \w{3}\s\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}\sUSER?

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...