Line breaker for multiline communigate logs

[robgreen]

Hello,
I have onbox Communigate logs i am trying to get imported into splunk for the PBXApp application. It logs multiline data where the "sender" is unique until it changes. for example

00:00:00.423 5 SIP [a.b.c.d]:5060 <- [a.b.c.d]:5060 inp(527): BYE sip:signode-35888-1B5CE480@a.b.c.d SIP/2.0\r\nVia: SIP/2.0/UDP a.b.c.d.

would be

time loglevel sender message

Maybe i am over thinking this or maybe splunk can't do this, but i want all logs with "sender" being the same until it changes to be one event. I have the data below (and extra line breaks to show where i want the event data split).

00:00:00.423 5 SIP [a.b.c.d]:5060 <- [a.b.c.d]:5060 inp(527): BYE sip:signode-35888-1B5CE480@a.b.c.d SIP/2.0\r\nVia: SIP/2.0/UDP a.b.c.d

00:00:00.423 2 SIPDATA-270876 inp: req [a.b.c.d]:5060 <- udp[a.b.c.d]:5060 BYE(527 bytes) sip:signode-35888-1B5CE480@a.b.c.d
00:00:00.423 5 SIPDATA-270876 inp: BYE sip:signode-35888-1B5CE480@a.b.c.d SIP/2.0
00:00:00.423 5 SIPDATA-270876 inp: Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bKilja01002gqh1gg0v6o1sd0000010.1
00:00:00.423 5 SIPDATA-270876 inp: From: <sip:+17178109151@a.b.c.d>;tag=SD6m3gc99-6006245088712893_c1ba.b.c.d0415929401.0_15699560_51184391
00:00:00.423 5 SIPDATA-270876 inp: To: <sip:unknown@a.b.c.d>;tag=000000000035888-46281E9F-1B5CE480
00:00:00.423 5 SIPDATA-270876 inp: Call-ID: 1334015937140-2aaae00d93b0-b0073040-00103f48@a.b.c.d.gwout
00:00:00.423 5 SIPDATA-270876 inp: Max-Forwards: 68
00:00:00.423 5 SIPDATA-270876 inp: CSeq: 2 BYE
00:00:00.423 5 SIPDATA-270876 inp: Content-Length: 0
00:00:00.423 5 SIPDATA-270876 inp: Route: <sip:a.b.c.d:5060;lr>
00:00:00.423 5 SIPDATA-270876 inp: Route: <sip:rev.23009616.dialog.cgatepro;lr>
00:00:00.423 5 SIPDATA-270876 inp: 
00:00:00.423 4 SIPDATA-270876 Hash=2017930222
00:00:00.423 4 SIPDATA-270876 self-route removed: <sip:a.b.c.d:5060;lr>
00:00:00.423 4 SIPDATA-270876 special route detected: sip:rev.23009616.dialog.cgatepro;lr
00:00:00.423 2 SIPDATA-270876 created SIPS-408676

00:00:00.423 4 ACCOUNT(pbx) taken from cache
00:00:00.423 4 ACCOUNT(pbx) placed in cache

00:00:00.424 2 SIPDATA-270877 out: rsp [a.b.c.d]:5060 -> udp[a.b.c.d]:5060 200-BYE(424 bytes)
00:00:00.424 5 SIPDATA-270877 out: SIP/2.0 200 OK
00:00:00.424 5 SIPDATA-270877 out: Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bKilja01002gqh1gg0v6o1sd0000010.1
00:00:00.424 5 SIPDATA-270877 out: From: <sip:+17178109151@a.b.c.d>;tag=SD6m3gc99-6006245088712893_c1ba.b.c.d0415929401.0_15699560_51184391
00:00:00.424 5 SIPDATA-270877 out: To: <sip:unknown@a.b.c.d>;tag=000000000035888-46281E9F-1B5CE480
00:00:00.424 5 SIPDATA-270877 out: Call-ID: 1334015937140-2aaae00d93b0-b0073040-00103f48@a.b.c.d.gwout
00:00:00.424 5 SIPDATA-270877 out: CSeq: 2 BYE
00:00:00.424 5 SIPDATA-270877 out: Server: CommuniGatePro/5.2.11
00:00:00.424 5 SIPDATA-270877 out: Content-Length: 0
00:00:00.424 5 SIPDATA-270877 out: 

00:00:00.424 5 SIP [a.b.c.d]:5060 -> [a.b.c.d]:5060 out(424): SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bKilja01002gqh1g

00:00:00.424 2 SIPDATA-270878 out: req [a.b.c.d]:5060 -> udp[a.b.c.d]:5060 BYE(570 bytes) sip:unknown@a.b.c.d:5060

Once the "sender" changes then there is no going back.. Said another way, the log lines are not interspersed.

Is this possible? A side question but of lesser importance is that the "date" of the log line is in the filename, is there a way to extract it from it? i.e. the above contents would be a in a file called 2012-04-09.log.

thanks,
rob

[arizvi801]

Try adding this to your props.conf:

SHOULD_LINEMERGE = 1
BREAK_ONLY_BEFORE = /d+/:/d+/:/d+/./d+/s/d/sSIP/s+/[/w/./w/./w/./w/]/:/d+

The regular expression matches:

00:00:00.423 5 SIP [a.b.c.d]:5060

/d+ means 1 or more digits, /: means the character ":" is literal, /w+ means 1 or more alphabetical characters and finally, I have used a.b.c.d literally as four alphabetical chars with periods between them, if it needs to be changed to an IP address, replace "/w+" by "/d+".

This config will allow each SIP call as 1 event with multiple lines.

NOTE: Setting SHOULD_LINEMERGE = 1 does come with a slightly greater load on Splunk, so please test the load before applying to PROD.

[gkanapathy]

No you can not do this at index time. You would have to index each line separately, and then you could use the transaction command at search time to assemble the events, something like | transaction sender maxspan=2