Getting Data In

Multi-line event break question

New Member

Currently all of the logs coming in from a call manager are being broken up per line and I am trying to merge them into a multi-line event. Logs are coming in over a shared UDP 514 port I have managed to assign a source type per event to these specific hosts but I am not yet able to configure the multiline breaks.

transforms.conf inside the app folder

DELIMS = ":"
FIELDS = "field1","field2","field3","call_id_label","call_id","field6","field7"

DELIMS = " "
FIELDS = "field1","field2","field3","field4","field5","call_order"

DELIMS = ";"
FIELDS = "field1","tag"

DELIMS = " ="
FIELDS = "field1","field2","field3","field4","field5","field6","field7","field8","field9","sip_tag","field11","field12","field13","field14","field15"

REGEX = (?:[A-z][A-z][A-z])\s\s(?:\d\s\d\d:\d\d:\d\d)\s*\[?(|||[\w\.\-]*\]?\s
FORMAT = sourcetype::telecom
DEST_KEY = MetaData:Sourcetype

props.conf inside the app folder

REPORT-call_id = REPORT-call_id
REPORT-call_order = REPORT-call_order
EXTRACT-sip_from = (?=[^F]*(?:From:|F.*From:))^(?:[^:\n]*:){5}(?P<sip_from>[^<]+)
EXTRACT-phone_to,trunk_to = (?=[^T]*(?:To:|T.*To:))^[^<\n]*<\w+:(?P<phone_to>[^@]+)[^@\n]*@(?P<trunk_to>\d+\.\d+\.\d+\.\d+)
EXTRACT-phone_from,from_trunk = (?=[^F]*(?:From:|F.*From:))^[^<\n]*<\w+:(?P<phone_from>[^@]+)[^@\n]*@(?P<from_trunk>[^>]+)
EXTRACT-tag_to = (?=[^T]*(?:To:|T.*To:))^[^;\n]*;(?P<tag_to>.+)
EXTRACT-tag_from = (?=[^F]*(?:From:|F.*From:))^[^;\n]*;(?P<tag_from>.+)
EXTRACT-sip_to = (?=[^T]*(?:To:|T.*To:))^(?:[^:\n]*:){5}(?P<sip_to>[^<]+)
EXTRACT-SIP_internal_error = (?=[^S]*(?:SIP: Internal Error|S.*SIP: Internal Error))^(?:[^:\n]*:){10}\s+\w+\s+\w+\s+(?P<SIP_internal_error>[^:]+)
EXTRACT-cause_code = (?=[^C]*(?:Cause Value=|C.*Cause Value=))^[^=\n]*=(?P<cause_code>\d+)
BREAK_ONLY_BEFORE = (?:[A-z][A-z][A-z])\s\s(?:\d\s\d\d:\d\d:\d\d)\s*\[?(?:\d\d.\d\d\d.\d\d\d.\d\d\d)[\w\.\-]*\]?\s(?:\d\d\d\d\d\d\d\d\d|\d\d\d\d\d\d\d\d):\s(?:\w\w\w\w\w\w\w\w\w\w\w-\w\w-\w\w):\s(Content-Length):\s
category = Custom
disabled = false

TRANSFORMS-changesourcetype = set_sourcetype_telecom


Jun  3 16:59:06 68938545: XXXXXXXXXXXX-VG-01: 
Jun  3 16:59:06 68938544: XXXXXXXXXXXX-VG-01: Content-Length: 0
Jun  3 16:59:06 68938543: XXXXXXXXXXXX-VG-01: CSeq: 101 OPTIONS
Jun  3 16:59:06 68938542: XXXXXXXXXXXX-VG-01: Call-ID: A0600097-858111E9-A4BFB6A6-6B5D3B24@
Jun  3 16:59:06 68938541: XXXXXXXXXXXX-VG-01: From: <sip:>;tag=A6721478-248C
Jun  3 16:59:06 68938540: XXXXXXXXXXXX-VG-01: To: <sip:>;tag=dsdacc6cdd
Jun  3 16:59:06 68938539: XXXXXXXXXXXX-VG-01: Via: SIP/2.0/TCP;branch=z9hG4bK18B3E1D39
Jun  3 16:59:06 68938538: XXXXXXXXXXXX-VG-01: SIP/2.0 200 Ok
Jun  3 16:59:06 68938537: XXXXXXXXXXXX-VG-01: Received: 
Jun  3 16:59:06 68938536: XXXXXXXXXXXX-VG-01: 4329485: Jun  3 16:58:59.943: //1726404/000000000000/SIP/Msg/ccsipDisplayMsg:
Jun  3 16:59:06 68938535: XXXXXXXXXXXX-VG-01: 

I've got the line break working locally when testing but no luck so far with the actual ingestion of the logs.

Any help would be greatly appreciated!

0 Karma


Nothing in your sample data matches the BREAK_ONLY_BEFORE regex so none of your events will break. Tell us where the event should break and we can help with the expression.

If this reply helps you, Karma would be appreciated.
0 Karma

New Member

sorry, I was asked to anonymize the sample data and that made it not match.

the break happens at line 2, I tried just using

BREAK_ONLY_BEFORE = Content-Length:

But since it wasn't working I tried the above regex to be more specific and it was matching the data before I skewed it.

I'm barely getting started with regex, would the following be a better string?

0 Karma


Yes, that regex works better, but is quite inefficient. Try \w{3}\s+\d+\s\d\d:\d\d:\d\d\s+\[?\d+.\d+.\d+.\d+[\w\.\-]*\]?\s\d+:\s\w+-\w+-\w+:\s(Content-Length):\s. I use to test regex strings.

If this reply helps you, Karma would be appreciated.
0 Karma

Revered Legend

Your TRANSFORMS for setting sourcetype as telecom executes after the data has been parsed (event breaking, timestamp parsing etc). Does that UDP port gets data of other type and those types don't need that line merge??

0 Karma

New Member

yes that udp port receives other types of data that will not be using the line merge

0 Karma
Get Updates on the Splunk Community!

Splunk Security Content for Threat Detection & Response, Q1 Roundup

Join Principal Threat Researcher, Michael Haag, as he walks through:An introduction to the Splunk Threat ...

Splunk Life | Happy Pride Month!

Happy Pride Month, Splunk Community! &#x1f308; In the United States, as well as many countries around the ...

SplunkTrust | Where Are They Now - Michael Uschmann

The Background Five years ago, Splunk published several videos showcasing members of the SplunkTrust to share ...