XML Event breaking and routing to nullqueue

jsanjeb — Wed, 25 Apr 2018 08:27:38 GMT

Hi Splunkers,

I am working on removing XML headers of events coming from queue to Splunk. However the events are not breaking as expected as I am not able to figure out right regex. XML is like this,

`Tue Apr 24 07:34:13 CDT 2018 name=QUEUE_msg_received event_id=ID:414d5120504443524444303220202020f4decc5ac8811c21 msg_dest=MFTTRANSACTION.EVENTS.LOG.INPUT.QUEUE msg_body=<?xml version="1.0" encoding="UTF-8"?><transaction version="5.00" ID="414d512050424d4654443031202020202030cb5a1d29fb24" agentRole="callAgent" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="TransferLog.xsd" xmlns=""><action time="2018-04-24T12:34:12.813Z">completed</action><agent agent="SAPBE_DEV_C700S021_01" QMgr="PBMFTD01"><systemInfo architecture="IA64N" name="HP-UX" version="B.11.31"/></agent>SAPBE_DEV_C700S021_01</line><line>BFGCL0213I: agent SAPBE_DEV_C700S021_01 responded to ping in 0.403 seconds.</line><line>BFGCL0212I: Issuing ping request to agent SAPBE_DEV_C700S021_01<line>BFGAN0049I: Move operation assigned transfer ID: 414d512050424d4654443031202020202030cb5a2b29fb24</line><line>BFGAN0053I: Successfully completion of move operation: 414d512050424d4654443031202020202030cb5a2b29fb24</line><line>BFGCL0211I: </line><line>BFGCL0211I: complete:</line><line>BFGCL0210W: Transfer Complete</line></stdout></result></callResult></call></transferSet><statistics><retryCount>0</retryCount><numFileFailures>0</numFileFailures><numFileWarnings>0</numFileWarnings></statistics></transaction>`

I should break the events in this way,

Event first half,

Tue Apr 24 07:34:13 CDT 2018 name=QUEUE_msg_received event_id=ID:414d5120504443524444303220202020f4decc5ac8811c21 msg_dest=MFTTRANSACTION.EVENTS.LOG.INPUT.QUEUE msg_body=

Event second half,

<?xml version="1.0" encoding="UTF-8"?><transaction version="5.00" ID="414d512050424d4654443031202020202030cb5a1d29fb24" agentRole="callAgent" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="TransferLog.xsd" xmlns=""><action time="2018-04-24T12:34:12.813Z">completed</action><agent agent="SAPBE_DEV_C700S021_01" QMgr="PBMFTD01"><systemInfo architecture="IA64N" name="HP-UX" version="B.11.31"/></agent>SAPBE_DEV_C700S021_01</line><line>BFGCL0213I: agent SAPBE_DEV_C700S021_01 responded to ping in 0.403 seconds.</line><line>BFGCL0212I: Issuing ping request to agent SAPBE_DEV_C700S021_01<line>BFGAN0049I: Move operation assigned transfer ID: 414d512050424d4654443031202020202030cb5a2b29fb24</line><line>BFGAN0053I: Successfully completion of move operation: 414d512050424d4654443031202020202030cb5a2b29fb24</line><line>BFGCL0211I: </line><line>BFGCL0211I: complete:</line><line>BFGCL0210W: Transfer Complete</line></stdout></result></callResult></call></transferSet><statistics><retryCount>0</retryCount><numFileFailures>0</numFileFailures><numFileWarnings>0</numFileWarnings></statistics></transaction>

I should route event first half to nullqueue while sending the second half to indexer. Kindly help me with regex for null proper line breaking as well as regex for routing to nullqueue.

Thanks in advance

Re: XML Event breaking and routing to nullqueue

somesoni2 — Wed, 25 Apr 2018 16:48:23 GMT

Try this in props.conf of your indexer/heavy forwarder (the SEDCMD will remove the text you don't want in your event)

[yourSourceTypeHere]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\w+\s+\w+\s+\d{2}\:\d{2}\:\d{2})
TIME_PREFIX = time=\"
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N%z
MAX_TIMESTAMP_LOOKAHEAD = 24
SEDCMD-removeheader = s/^(.+)\smsg_body\=//g

Re: XML Event breaking and routing to nullqueue

jsanjeb — Thu, 26 Apr 2018 09:30:42 GMT

Great!! It worked. Thanks for the fix.

Re: XML Event breaking and routing to nullqueue

jsanjeb — Thu, 26 Apr 2018 09:33:09 GMT

But the header which i have mentioned in question ie event first half is added by forwarder to the data. Is there a way we can stop forwarder from adding this?

topic Re: XML Event breaking and routing to nullqueue in Deployment Architecture

XML Event breaking and routing to nullqueue

Re: XML Event breaking and routing to nullqueue

Re: XML Event breaking and routing to nullqueue

Re: XML Event breaking and routing to nullqueue