Greetings!
I am trying to merge 2 lines into 1 event but having problems. Appreciate advice on my steps taken
Sample Log - Using logger command to push this to splunk via syslog. Hence the append of timestamp (and splunk treats that as separate events)
Nov 1 12:12:12 event login command
Nov 1 12:12:12 ----- ----- -----------
Nov 1 12:12:13 USER_Create admin2 mkuser
Nov 1 12:12:13 tstadmin
Nov 1 12:12:13 USER_Create admin2 mkuser
Nov 1 12:12:13 tstadmin
Nov 1 12:12:13 USER_Remove admin2 rmuser
Nov 1 12:12:13 tstadmin
Nov 1 12:12:13 USER_Remove admin2 rmuser
Nov 1 12:12:13 tstadmin
Objective - I would like to group the following as 1 event to show the account created "tstadmin"
Nov 1 12:12:13 USER_Create admin2 mkuser
Nov 1 12:12:13 tstadmin
Added the following stanza to \etc\system\local\props.conf
[source::c:\test-multiline.txt]
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE = USER_*
Restarted Splunk
I've managed to make it working by updating \etc\system\local\props.conf
[multiline.txt]
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE = USER_*
However, taking one step forward, I'm trying to use a transform to change the name of the sourcetype. Hence, my \etc\system\local\props.conf looks like the following
[multiline.txt]
TRANSFORM-changesourcetype = setsourcetype
[UserAccount]
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE = USER_*
While my \etc\system\local\transforms.conf looks like this
[setsourcetype]
DEST_KEY = MetaData:Sourcetype
REGEX = USERACCOUNT
FORMAT = sourcetype::UserAccount
** I'm able to see the new sourcetype being created. However, the line merge is failing to work now!! I suspect its because when Splunk reads props.conf, the UserAccount sourcetype has not been created yet so it wasnt able to set the BREAK_ONLY_BEFORE field.
Can someone confirm my fears?? And how can this be overcome?
Using your current props.conf, you should be able to replace the *SHOULD_LINEMERGE* line with this:
BREAK_ONLY_BEFORE = \w{3}\s\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}\sUSER
Replacing both of the lines with something like this would be more efficient:
MAX_TIMESTAMP_LOOKAHEAD = 16
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\w{3}\s\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}\sUSER)
I see, thanks for the info. Moving forward, I'm trying to separate the specific syslog messages from the rest (via assigning a new sourcetype). However, it appears that setting sourcetype at index time via TRANSFORMS does not select rules in props based on the new sourcetype at index time, and it only affects search time. (See post below).
Is there any way this can be done?
In the BREAK_ONLY_BEFORE example, that regular expression lets Splunk know that any line with a timestamp followed by USER is the beginning of a new event. I recommend using the LINE_BREAKER option though, per original response.
Thanks, can you help me understand why we need
BREAK_ONLY_BEFORE = \w{3}\s\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}\sUSER?