XML Event breaking and routing to nullqueue

[jsanjeb]

Hi Splunkers,

I am working on removing XML headers of events coming from queue to Splunk. However the events are not breaking as expected as I am not able to figure out right regex. XML is like this,

`Tue Apr 24 07:34:13 CDT 2018 name=QUEUE_msg_received event_id=ID:414d5120504443524444303220202020f4decc5ac8811c21 msg_dest=MFTTRANSACTION.EVENTS.LOG.INPUT.QUEUE msg_body=<?xml version="1.0" encoding="UTF-8"?><transaction version="5.00" ID="414d512050424d4654443031202020202030cb5a1d29fb24" agentRole="callAgent" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="TransferLog.xsd" xmlns=""><action time="2018-04-24T12:34:12.813Z">completed</action><agent agent="SAPBE_DEV_C700S021_01" QMgr="PBMFTD01"><systemInfo architecture="IA64N" name="HP-UX" version="B.11.31"/></agent>SAPBE_DEV_C700S021_01</line><line>BFGCL0213I: agent SAPBE_DEV_C700S021_01 responded to ping in 0.403 seconds.</line><line>BFGCL0212I: Issuing ping request to agent SAPBE_DEV_C700S021_01<line>BFGAN0049I: Move operation assigned transfer ID: 414d512050424d4654443031202020202030cb5a2b29fb24</line><line>BFGAN0053I: Successfully completion of move operation: 414d512050424d4654443031202020202030cb5a2b29fb24</line><line>BFGCL0211I: </line><line>BFGCL0211I: complete:</line><line>BFGCL0210W: Transfer Complete</line></stdout></result></callResult></call></transferSet><statistics><retryCount>0</retryCount><numFileFailures>0</numFileFailures><numFileWarnings>0</numFileWarnings></statistics></transaction>`

I should break the events in this way,

Event first half,

Tue Apr 24 07:34:13 CDT 2018 name=QUEUE_msg_received event_id=ID:414d5120504443524444303220202020f4decc5ac8811c21 msg_dest=MFTTRANSACTION.EVENTS.LOG.INPUT.QUEUE msg_body=

Event second half,

<?xml version="1.0" encoding="UTF-8"?><transaction version="5.00" ID="414d512050424d4654443031202020202030cb5a1d29fb24" agentRole="callAgent" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="TransferLog.xsd" xmlns=""><action time="2018-04-24T12:34:12.813Z">completed</action><agent agent="SAPBE_DEV_C700S021_01" QMgr="PBMFTD01"><systemInfo architecture="IA64N" name="HP-UX" version="B.11.31"/></agent>SAPBE_DEV_C700S021_01</line><line>BFGCL0213I: agent SAPBE_DEV_C700S021_01 responded to ping in 0.403 seconds.</line><line>BFGCL0212I: Issuing ping request to agent SAPBE_DEV_C700S021_01<line>BFGAN0049I: Move operation assigned transfer ID: 414d512050424d4654443031202020202030cb5a2b29fb24</line><line>BFGAN0053I: Successfully completion of move operation: 414d512050424d4654443031202020202030cb5a2b29fb24</line><line>BFGCL0211I: </line><line>BFGCL0211I: complete:</line><line>BFGCL0210W: Transfer Complete</line></stdout></result></callResult></call></transferSet><statistics><retryCount>0</retryCount><numFileFailures>0</numFileFailures><numFileWarnings>0</numFileWarnings></statistics></transaction>

I should route event first half to nullqueue while sending the second half to indexer. Kindly help me with regex for null proper line breaking as well as regex for routing to nullqueue.

Thanks in advance

[somesoni2 · Accepted solution]

Try this in props.conf of your indexer/heavy forwarder (the SEDCMD will remove the text you don't want in your event)

[yourSourceTypeHere]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\w+\s+\w+\s+\d{2}\:\d{2}\:\d{2})
TIME_PREFIX = time=\"
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N%z
MAX_TIMESTAMP_LOOKAHEAD = 24
SEDCMD-removeheader = s/^(.+)\smsg_body\=//g

[jsanjeb]

Great!! It worked. Thanks for the fix.

[jsanjeb]

But the header which i have mentioned in question ie event first half is added by forwarder to the data. Is there a way we can stop forwarder from adding this?