How do I split one event on my Splunk instance into multiple events?

[swinte12]

I have several indexes in my Splunk Instance. One of these instances is merging some of my log events into a single event incorrectly. I need help splitting them up. At the bottom of this question, I have posted an example. What I need help with is... what is the easiest way to split these events so that each event is separate? I believe that the best way is to specify the attributes in the props.conf file to change the line breaking and merging behavior so that all lines are merged ("SHOULD_LINEMERGE = true") and then set the attribute to Break the lines at the beginning when the date is given ("BREAK_ONLY_BEFORE = ^(?P[^ ]+)") - not sure my regex is correct.

1. is this the correct way to resolve my problem?
2. do I change this in the Splunk Enterprise Local props.conf file, or do I make this change on the Forwarder for this particular index?
3. how do I ensure it does not change the behavior of other indexes or sourcetypes?
4. is this regex correct for what I am trying to do? I am trying to break whenever the timestamp exists at the beginning like 18:48:56. It is always in the same format ##:##:##. sometimes there are stacktrace or multiline logs in one event, that is why I need them to be merged and seperated differently.

these lines are recognized as separate events

    18:48:55 Completed Teststep: successful login using email: {0} and password: {1} with arguments: [swinters@gci.com, Ch0c0late$tarfish]
    18:48:56 Invoking Teststep: get {url} with arguments: []
    18:48:56 Invoking Teststep: get {url} with arguments: []
    18:48:56 Invoking Teststep: get {url} with arguments: 
**these lines are not recognized as separate events but should break after the timestamp
18:45:59 Failure Teststep: First successful login after deploy using email: {0} and password: {1} with arguments: [swinters@gci.com, Ch0c0late$tarfish]
com.qmetry.qaf.automation.step.JavaStepReporter$JPThrowable: org.openqa.selenium.TimeoutException: Expected condition failed: Wait time out for User Menu to be present (tried for 60 second(s) with 1000 MILLISECONDS interval)
Build info: version: '3.5.3', revision: 'a88d25fe6b', time: '2017-08-29T12:42:44.417Z'
System info: host: '52dc70ddb77f', ip: '172.17.0.2', os.name: 'Linux', os.arch: 'amd64', os.version: '3.10.0-123.8.1.el7.x86_64', java.version: '1.8.0_92'
Driver info: driver.version: unknown
    at com.qmetry.qaf.automation.step.JavaStepReporter$MockJavaStep.doExecute(JavaStepReporter.java:144)
    at com.qmetry.qaf.automation.step.BaseTestStep.execute(BaseTestStep.java:149)
    at com.qmetry.qaf.automation.step.JavaStepReporter.javaTestStep(JavaStepReporter.java:88)
    at hubTesting.steps.globalSteps.LoginSteps.firstLogin(LoginSteps.java:15)
    at hubTesting.test.globalTests.RefreshHubUser.refreshUserMgmtUser(RefreshHubUser.java:79)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
    at org.testng.internal.Invoker.invokeMethod(Invoker.java:571)
    at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:707)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:979)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.openqa.selenium.TimeoutException: Expected condition failed: Wait time out for User Menu to be present (tried for 60 second(s) with 1000 MILLISECONDS interval)
Build info: version: '3.5.3', revision: 'a88d25fe6b', time: '2017-08-29T12:42:44.417Z'
System info: host: '52dc70ddb77f', ip: '172.17.0.2', os.name: 'Linux', os.arch: 'amd64', os.version: '3.10.0-123.8.1.el7.x86_64', java.version: '1.8.0_92'
Driver info: driver.version: unknown
    at org.openqa.selenium.support.ui.FluentWait.timeoutException(FluentWait.java:296)
    at org.openqa.selenium.support.ui.FluentWait.until(FluentWait.java:265)
    at com.qmetry.qaf.automation.ui.webdriver.QAFExtendedWebElement.waitForPresent(QAFExtendedWebElement.java:499)
    at com.qmetry.qaf.automation.step.CommonStep.waitForPresent(CommonStep.java:805)
    at hubTesting.steps.globalSteps.LoginSteps.firstLogin_aroundBody0(LoginSteps.java:23)
    at hubTesting.steps.globalSteps.LoginSteps$AjcClosure1.run(LoginSteps.java:1)
    at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
    at com.qmetry.qaf.automation.step.JavaStepReporter$MockJavaStep.doExecute(JavaStepReporter.java:138)
    ... 17 more
18:45:59 Completed Teststep: First successful login after deploy using email: {0} and password: {1} with arguments: [swinters@gci.com, Ch0c0late$tarfish]
18:46:00 Invoking Teststep: get {url} with arguments: []
18:46:00 Invoking Teststep: get {url} with arguments: []
18:46:00 Failure Teststep: First successful login after deploy using email: {0} and password: {1} with arguments: [swinters@gci.com, Ch0c0late$tarfish]
com.qmetry.qaf.automation.step.JavaStepReporter$JPThrowable: org.openqa.selenium.TimeoutException: Expected condition failed: Wait time out for User Menu to be present (tried for 60 second(s) with 1000 MILLISECONDS interval)

[skoelpin]

You should apply base configs in your props.conf and rerstart the Splunk service

Try this

[sourcetype]
TIME_PREFIX = ^
TIME_FORMAT = %H:%M:%S
SHOULD_LINE_MERGE = false
LINE_BREAKER = ([\r\n]+)(\s+\d+:\d+:\d+)
MAX_TIMESTAMP_LOOKAHEAD = 15
TRUNCATE = 1000

Make sure to replace sourcetype with your sourcetype. Also, is there a space in front of the hour? If so then the above configs will work

[micahkemp · Accepted solution]

You don't need SHOULD_LINEMERGE, just LINE_BREAKER. This should split lines into new events only when they start with a timestamp:

props.conf:

[<sourcetype>]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\n\r]+)\d{2}:\d{2}:\d{2}

Oh, and my regex assumes that the leading space on your log lines isn't actually present in the actual data, but an artifact of your copy/paste into answers.