Getting Data In

How to write regex to event break a multi line file into single event?

Hemnaath
Motivator

I have multi line file (_json), which I am trying to create a individual events, the multi line file contains array of id, message and timestamp.

Sample Event data: 

{ [-]
logEvents: [ [-]
{ [-]
id: 3576745055635743000077342515139507954347666517578940416
message: START RequestId: 4e1251df-11d9-55d0-918a-09bb06b96122 Version: $LATEST

timestamp: 1603867953198
}
{ [+]
}
{ [-]
id: 35767450557316368740614159310005543840071546062336098306
message: [2020-10-28T06:52:33.240Z][4e1251df-11d9-55d0-918c-09cc06b96122][INFO][wfm-test2-lmd-towSyncWorkOrderWOM][HeaderProcessor.py, 23][The filtered request headers are {"test-PartyID": "test"}]

timestamp: 1603867953241
}
{ [+]
}
{ [-]
id: 3576745057558067905821073966314329716666554135734059012
message: [2020-10-28T06:52:34.59Z][4e1251df-11d9-55d0-918c-09cc06b96122][INFO][wfm-test2-lmd-towSyncWorkOrderWOM][lambda_function.py, 37][Response received from SNOW with status code :202 and response as {"result":{"message":"Message has been received!","value":"WOR200033942808"}}]

timestamp: 1603867954060
}
{ [+]
}
{ [+]
}
]
logGroup: /aws/lambda/wfm-test2-lmd-towSyncWorkOrderWOM
logStream: 2020/10/28/[$LATEST]0e5e38b8bf8e4247a5f063e5e1fdaf51
messageType: DATA_MESSAGE
owner: 126208963777
subscriptionFilters: [ [+]
]

Can you please guide me how to break this multi line event using the line breaker.

Labels (2)

Hemnaath
Motivator

Hey I was able to break the multi line events into single events using the below stanza 

SHOULD_LINEMERGE=false
LINE_BREAKER=(\[|,\s*|\], )({"id":|"logGroup":)
disabled=false
MAX_TIMESTAMP_LOOKAHEAD=13
TIME_FORMAT=%s%3Q
TIME_PREFIX="timestamp":\s+
TZ=UTC
TRUNCATE=100000

Hemnaath
Motivator

hey tried the same but that is not working 

SHOULD_LINEMERGE = false

LINE_BREAKER=([\r\n]+)\}()\{id:

 

richgalloway
SplunkTrust
SplunkTrust

Have you tried 

LINE_BREAKER = \}()\{id:
---
If this reply helps you, Karma would be appreciated.
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Vibe-coding, AI, and Splunkcraft: Highlights from the .conf26 Builder Bar

If you stopped by the Builder Bar at .conf26, thank you! This year, we brought ...

Thanks for the Memories: .conf26 Took Learning to New Heights

Thank you, Splunk Community, for making .conf26 in Denver one for the books. From packed Splunk University ...

Best Practices: Splunk auto adjust pipeline queue

When you enable autoAdjustQueue in Splunk, maxSize should be understood as the queue size Splunk starts with ...