Multi-line event break question

[kcastillo]

Currently all of the logs coming in from a call manager are being broken up per line and I am trying to merge them into a multi-line event. Logs are coming in over a shared UDP 514 port I have managed to assign a source type per event to these specific hosts but I am not yet able to configure the multiline breaks.

transforms.conf inside the app folder

[REPORT-call_id]
DELIMS = ":"
FIELDS = "field1","field2","field3","call_id_label","call_id","field6","field7"

[REPORT-call_order]
DELIMS = " "
FIELDS = "field1","field2","field3","field4","field5","call_order"

[REPORT-tagsfield]
DELIMS = ";"
FIELDS = "field1","tag"

[REPORT-siptag]
DELIMS = " ="
FIELDS = "field1","field2","field3","field4","field5","field6","field7","field8","field9","sip_tag","field11","field12","field13","field14","field15"

[set_sourcetype_telecom]
REGEX = (?:[A-z][A-z][A-z])\s\s(?:\d\s\d\d:\d\d:\d\d)\s*\[?(10.100.200.205|10.100.205.205|10.160.205.205|10.144.205.205)[\w\.\-]*\]?\s
FORMAT = sourcetype::telecom
DEST_KEY = MetaData:Sourcetype

props.conf inside the app folder

[telecom]
REPORT-call_id = REPORT-call_id
REPORT-call_order = REPORT-call_order
EXTRACT-sip_from = (?=[^F]*(?:From:|F.*From:))^(?:[^:\n]*:){5}(?P<sip_from>[^<]+)
EXTRACT-phone_to,trunk_to = (?=[^T]*(?:To:|T.*To:))^[^<\n]*<\w+:(?P<phone_to>[^@]+)[^@\n]*@(?P<trunk_to>\d+\.\d+\.\d+\.\d+)
EXTRACT-phone_from,from_trunk = (?=[^F]*(?:From:|F.*From:))^[^<\n]*<\w+:(?P<phone_from>[^@]+)[^@\n]*@(?P<from_trunk>[^>]+)
EXTRACT-tag_to = (?=[^T]*(?:To:|T.*To:))^[^;\n]*;(?P<tag_to>.+)
EXTRACT-tag_from = (?=[^F]*(?:From:|F.*From:))^[^;\n]*;(?P<tag_from>.+)
EXTRACT-sip_to = (?=[^T]*(?:To:|T.*To:))^(?:[^:\n]*:){5}(?P<sip_to>[^<]+)
EXTRACT-SIP_internal_error = (?=[^S]*(?:SIP: Internal Error|S.*SIP: Internal Error))^(?:[^:\n]*:){10}\s+\w+\s+\w+\s+(?P<SIP_internal_error>[^:]+)
EXTRACT-cause_code = (?=[^C]*(?:Cause Value=|C.*Cause Value=))^[^=\n]*=(?P<cause_code>\d+)
BREAK_ONLY_BEFORE = (?:[A-z][A-z][A-z])\s\s(?:\d\s\d\d:\d\d:\d\d)\s*\[?(?:\d\d.\d\d\d.\d\d\d.\d\d\d)[\w\.\-]*\]?\s(?:\d\d\d\d\d\d\d\d\d|\d\d\d\d\d\d\d\d):\s(?:\w\w\w\w\w\w\w\w\w\w\w-\w\w-\w\w):\s(Content-Length):\s
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = true
category = Custom
disabled = false

[source::udp:514]
TRANSFORMS-changesourcetype = set_sourcetype_telecom

sample

Jun  3 16:59:06 10.10.10.1 68938545: XXXXXXXXXXXX-VG-01: 
Jun  3 16:59:06 10.10.10.1 68938544: XXXXXXXXXXXX-VG-01: Content-Length: 0
Jun  3 16:59:06 10.10.10.1 68938543: XXXXXXXXXXXX-VG-01: CSeq: 101 OPTIONS
Jun  3 16:59:06 10.10.10.1 68938542: XXXXXXXXXXXX-VG-01: Call-ID: A0600097-858111E9-A4BFB6A6-6B5D3B24@10.10.10.1
Jun  3 16:59:06 10.10.10.1 68938541: XXXXXXXXXXXX-VG-01: From: <sip:10.10.10.1>;tag=A6721478-248C
Jun  3 16:59:06 10.10.10.1 68938540: XXXXXXXXXXXX-VG-01: To: <sip:10.10.10.2>;tag=dsdacc6cdd
Jun  3 16:59:06 10.10.10.1 68938539: XXXXXXXXXXXX-VG-01: Via: SIP/2.0/TCP 10.10.10.1:5060;branch=z9hG4bK18B3E1D39
Jun  3 16:59:06 10.10.10.1 68938538: XXXXXXXXXXXX-VG-01: SIP/2.0 200 Ok
Jun  3 16:59:06 10.10.10.1 68938537: XXXXXXXXXXXX-VG-01: Received: 
Jun  3 16:59:06 10.10.10.1 68938536: XXXXXXXXXXXX-VG-01: 4329485: Jun  3 16:58:59.943: //1726404/000000000000/SIP/Msg/ccsipDisplayMsg:
Jun  3 16:59:06 10.10.10.1 68938535: XXXXXXXXXXXX-VG-01: 

I've got the line break working locally when testing but no luck so far with the actual ingestion of the logs.

Any help would be greatly appreciated!

[richgalloway]

Nothing in your sample data matches the BREAK_ONLY_BEFORE regex so none of your events will break. Tell us where the event should break and we can help with the expression.

[kcastillo]

sorry, I was asked to anonymize the sample data and that made it not match.

the break happens at line 2, I tried just using

BREAK_ONLY_BEFORE = Content-Length:

But since it wasn't working I tried the above regex to be more specific and it was matching the data before I skewed it.

I'm barely getting started with regex, would the following be a better string?

(?:[A-z]*)\s\s(?:\d*\s\d\d:\d\d:\d\d)\s*\[?(?:\d*.\d*.\d*.\d*)[\w\.\-]*\]?\s(?:\d*):\s(?:\w*-\w*-\w*):\s(Content-Length):\s

[richgalloway]

Yes, that regex works better, but is quite inefficient. Try \w{3}\s+\d+\s\d\d:\d\d:\d\d\s+\[?\d+.\d+.\d+.\d+[\w\.\-]*\]?\s\d+:\s\w+-\w+-\w+:\s(Content-Length):\s. I use regex101.com to test regex strings.

[somesoni2]

Your TRANSFORMS for setting sourcetype as telecom executes after the data has been parsed (event breaking, timestamp parsing etc). Does that UDP port gets data of other type and those types don't need that line merge??

[kcastillo]

yes that udp port receives other types of data that will not be using the line merge