Getting Data In

Multi-line event break question

kcastillo
New Member

Currently all of the logs coming in from a call manager are being broken up per line and I am trying to merge them into a multi-line event. Logs are coming in over a shared UDP 514 port I have managed to assign a source type per event to these specific hosts but I am not yet able to configure the multiline breaks.

transforms.conf inside the app folder

[REPORT-call_id]
DELIMS = ":"
FIELDS = "field1","field2","field3","call_id_label","call_id","field6","field7"

[REPORT-call_order]
DELIMS = " "
FIELDS = "field1","field2","field3","field4","field5","call_order"

[REPORT-tagsfield]
DELIMS = ";"
FIELDS = "field1","tag"

[REPORT-siptag]
DELIMS = " ="
FIELDS = "field1","field2","field3","field4","field5","field6","field7","field8","field9","sip_tag","field11","field12","field13","field14","field15"

[set_sourcetype_telecom]
REGEX = (?:[A-z][A-z][A-z])\s\s(?:\d\s\d\d:\d\d:\d\d)\s*\[?(10.100.200.205|10.100.205.205|10.160.205.205|10.144.205.205)[\w\.\-]*\]?\s
FORMAT = sourcetype::telecom
DEST_KEY = MetaData:Sourcetype

props.conf inside the app folder

[telecom]
REPORT-call_id = REPORT-call_id
REPORT-call_order = REPORT-call_order
EXTRACT-sip_from = (?=[^F]*(?:From:|F.*From:))^(?:[^:\n]*:){5}(?P<sip_from>[^<]+)
EXTRACT-phone_to,trunk_to = (?=[^T]*(?:To:|T.*To:))^[^<\n]*<\w+:(?P<phone_to>[^@]+)[^@\n]*@(?P<trunk_to>\d+\.\d+\.\d+\.\d+)
EXTRACT-phone_from,from_trunk = (?=[^F]*(?:From:|F.*From:))^[^<\n]*<\w+:(?P<phone_from>[^@]+)[^@\n]*@(?P<from_trunk>[^>]+)
EXTRACT-tag_to = (?=[^T]*(?:To:|T.*To:))^[^;\n]*;(?P<tag_to>.+)
EXTRACT-tag_from = (?=[^F]*(?:From:|F.*From:))^[^;\n]*;(?P<tag_from>.+)
EXTRACT-sip_to = (?=[^T]*(?:To:|T.*To:))^(?:[^:\n]*:){5}(?P<sip_to>[^<]+)
EXTRACT-SIP_internal_error = (?=[^S]*(?:SIP: Internal Error|S.*SIP: Internal Error))^(?:[^:\n]*:){10}\s+\w+\s+\w+\s+(?P<SIP_internal_error>[^:]+)
EXTRACT-cause_code = (?=[^C]*(?:Cause Value=|C.*Cause Value=))^[^=\n]*=(?P<cause_code>\d+)
BREAK_ONLY_BEFORE = (?:[A-z][A-z][A-z])\s\s(?:\d\s\d\d:\d\d:\d\d)\s*\[?(?:\d\d.\d\d\d.\d\d\d.\d\d\d)[\w\.\-]*\]?\s(?:\d\d\d\d\d\d\d\d\d|\d\d\d\d\d\d\d\d):\s(?:\w\w\w\w\w\w\w\w\w\w\w-\w\w-\w\w):\s(Content-Length):\s
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = true
category = Custom
disabled = false

[source::udp:514]
TRANSFORMS-changesourcetype = set_sourcetype_telecom

sample

Jun  3 16:59:06 10.10.10.1 68938545: XXXXXXXXXXXX-VG-01: 
Jun  3 16:59:06 10.10.10.1 68938544: XXXXXXXXXXXX-VG-01: Content-Length: 0
Jun  3 16:59:06 10.10.10.1 68938543: XXXXXXXXXXXX-VG-01: CSeq: 101 OPTIONS
Jun  3 16:59:06 10.10.10.1 68938542: XXXXXXXXXXXX-VG-01: Call-ID: A0600097-858111E9-A4BFB6A6-6B5D3B24@10.10.10.1
Jun  3 16:59:06 10.10.10.1 68938541: XXXXXXXXXXXX-VG-01: From: <sip:10.10.10.1>;tag=A6721478-248C
Jun  3 16:59:06 10.10.10.1 68938540: XXXXXXXXXXXX-VG-01: To: <sip:10.10.10.2>;tag=dsdacc6cdd
Jun  3 16:59:06 10.10.10.1 68938539: XXXXXXXXXXXX-VG-01: Via: SIP/2.0/TCP 10.10.10.1:5060;branch=z9hG4bK18B3E1D39
Jun  3 16:59:06 10.10.10.1 68938538: XXXXXXXXXXXX-VG-01: SIP/2.0 200 Ok
Jun  3 16:59:06 10.10.10.1 68938537: XXXXXXXXXXXX-VG-01: Received: 
Jun  3 16:59:06 10.10.10.1 68938536: XXXXXXXXXXXX-VG-01: 4329485: Jun  3 16:58:59.943: //1726404/000000000000/SIP/Msg/ccsipDisplayMsg:
Jun  3 16:59:06 10.10.10.1 68938535: XXXXXXXXXXXX-VG-01: 

I've got the line break working locally when testing but no luck so far with the actual ingestion of the logs.

Any help would be greatly appreciated!

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Nothing in your sample data matches the BREAK_ONLY_BEFORE regex so none of your events will break. Tell us where the event should break and we can help with the expression.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

kcastillo
New Member

sorry, I was asked to anonymize the sample data and that made it not match.

the break happens at line 2, I tried just using

BREAK_ONLY_BEFORE = Content-Length:

But since it wasn't working I tried the above regex to be more specific and it was matching the data before I skewed it.

I'm barely getting started with regex, would the following be a better string?

(?:[A-z]*)\s\s(?:\d*\s\d\d:\d\d:\d\d)\s*\[?(?:\d*.\d*.\d*.\d*)[\w\.\-]*\]?\s(?:\d*):\s(?:\w*-\w*-\w*):\s(Content-Length):\s
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Yes, that regex works better, but is quite inefficient. Try \w{3}\s+\d+\s\d\d:\d\d:\d\d\s+\[?\d+.\d+.\d+.\d+[\w\.\-]*\]?\s\d+:\s\w+-\w+-\w+:\s(Content-Length):\s. I use regex101.com to test regex strings.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

somesoni2
Revered Legend

Your TRANSFORMS for setting sourcetype as telecom executes after the data has been parsed (event breaking, timestamp parsing etc). Does that UDP port gets data of other type and those types don't need that line merge??

0 Karma

kcastillo
New Member

yes that udp port receives other types of data that will not be using the line merge

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!