Hello,
I have onbox Communigate logs i am trying to get imported into splunk for the PBXApp application. It logs multiline data where the "sender" is unique until it changes. for example
00:00:00.423 5 SIP [a.b.c.d]:5060 <- [a.b.c.d]:5060 inp(527): BYE sip:signode-35888-1B5CE480@a.b.c.d SIP/2.0\r\nVia: SIP/2.0/UDP a.b.c.d.
would be
time loglevel sender message
Maybe i am over thinking this or maybe splunk can't do this, but i want all logs with "sender" being the same until it changes to be one event. I have the data below (and extra line breaks to show where i want the event data split).
00:00:00.423 5 SIP [a.b.c.d]:5060 <- [a.b.c.d]:5060 inp(527): BYE sip:signode-35888-1B5CE480@a.b.c.d SIP/2.0\r\nVia: SIP/2.0/UDP a.b.c.d
00:00:00.423 2 SIPDATA-270876 inp: req [a.b.c.d]:5060 <- udp[a.b.c.d]:5060 BYE(527 bytes) sip:signode-35888-1B5CE480@a.b.c.d
00:00:00.423 5 SIPDATA-270876 inp: BYE sip:signode-35888-1B5CE480@a.b.c.d SIP/2.0
00:00:00.423 5 SIPDATA-270876 inp: Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bKilja01002gqh1gg0v6o1sd0000010.1
00:00:00.423 5 SIPDATA-270876 inp: From: <sip:+17178109151@a.b.c.d>;tag=SD6m3gc99-6006245088712893_c1ba.b.c.d0415929401.0_15699560_51184391
00:00:00.423 5 SIPDATA-270876 inp: To: <sip:unknown@a.b.c.d>;tag=000000000035888-46281E9F-1B5CE480
00:00:00.423 5 SIPDATA-270876 inp: Call-ID: 1334015937140-2aaae00d93b0-b0073040-00103f48@a.b.c.d.gwout
00:00:00.423 5 SIPDATA-270876 inp: Max-Forwards: 68
00:00:00.423 5 SIPDATA-270876 inp: CSeq: 2 BYE
00:00:00.423 5 SIPDATA-270876 inp: Content-Length: 0
00:00:00.423 5 SIPDATA-270876 inp: Route: <sip:a.b.c.d:5060;lr>
00:00:00.423 5 SIPDATA-270876 inp: Route: <sip:rev.23009616.dialog.cgatepro;lr>
00:00:00.423 5 SIPDATA-270876 inp:
00:00:00.423 4 SIPDATA-270876 Hash=2017930222
00:00:00.423 4 SIPDATA-270876 self-route removed: <sip:a.b.c.d:5060;lr>
00:00:00.423 4 SIPDATA-270876 special route detected: sip:rev.23009616.dialog.cgatepro;lr
00:00:00.423 2 SIPDATA-270876 created SIPS-408676
00:00:00.423 4 ACCOUNT(pbx) taken from cache
00:00:00.423 4 ACCOUNT(pbx) placed in cache
00:00:00.424 2 SIPDATA-270877 out: rsp [a.b.c.d]:5060 -> udp[a.b.c.d]:5060 200-BYE(424 bytes)
00:00:00.424 5 SIPDATA-270877 out: SIP/2.0 200 OK
00:00:00.424 5 SIPDATA-270877 out: Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bKilja01002gqh1gg0v6o1sd0000010.1
00:00:00.424 5 SIPDATA-270877 out: From: <sip:+17178109151@a.b.c.d>;tag=SD6m3gc99-6006245088712893_c1ba.b.c.d0415929401.0_15699560_51184391
00:00:00.424 5 SIPDATA-270877 out: To: <sip:unknown@a.b.c.d>;tag=000000000035888-46281E9F-1B5CE480
00:00:00.424 5 SIPDATA-270877 out: Call-ID: 1334015937140-2aaae00d93b0-b0073040-00103f48@a.b.c.d.gwout
00:00:00.424 5 SIPDATA-270877 out: CSeq: 2 BYE
00:00:00.424 5 SIPDATA-270877 out: Server: CommuniGatePro/5.2.11
00:00:00.424 5 SIPDATA-270877 out: Content-Length: 0
00:00:00.424 5 SIPDATA-270877 out:
00:00:00.424 5 SIP [a.b.c.d]:5060 -> [a.b.c.d]:5060 out(424): SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bKilja01002gqh1g
00:00:00.424 2 SIPDATA-270878 out: req [a.b.c.d]:5060 -> udp[a.b.c.d]:5060 BYE(570 bytes) sip:unknown@a.b.c.d:5060
Once the "sender" changes then there is no going back.. Said another way, the log lines are not interspersed.
Is this possible? A side question but of lesser importance is that the "date" of the log line is in the filename, is there a way to extract it from it? i.e. the above contents would be a in a file called 2012-04-09.log.
thanks,
rob
Try adding this to your props.conf:
SHOULD_LINEMERGE = 1
BREAK_ONLY_BEFORE = /d+/:/d+/:/d+/./d+/s/d/sSIP/s+/[/w/./w/./w/./w/]/:/d+
The regular expression matches:
00:00:00.423 5 SIP [a.b.c.d]:5060
/d+ means 1 or more digits, /: means the character ":" is literal, /w+ means 1 or more alphabetical characters and finally, I have used a.b.c.d literally as four alphabetical chars with periods between them, if it needs to be changed to an IP address, replace "/w+" by "/d+".
This config will allow each SIP call as 1 event with multiple lines.
NOTE: Setting SHOULD_LINEMERGE = 1 does come with a slightly greater load on Splunk, so please test the load before applying to PROD.
No you can not do this at index time. You would have to index each line separately, and then you could use the transaction
command at search time to assemble the events, something like | transaction sender maxspan=2