Deployment Architecture

XML Event breaking and routing to nullqueue

jsanjeb
Explorer

Hi Splunkers,

I am working on removing XML headers of events coming from queue to Splunk. However the events are not breaking as expected as I am not able to figure out right regex. XML is like this,

`Tue Apr 24 07:34:13 CDT 2018 name=QUEUE_msg_received event_id=ID:414d5120504443524444303220202020f4decc5ac8811c21 msg_dest=MFTTRANSACTION.EVENTS.LOG.INPUT.QUEUE msg_body=<?xml version="1.0" encoding="UTF-8"?><transaction version="5.00" ID="414d512050424d4654443031202020202030cb5a1d29fb24" agentRole="callAgent" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="TransferLog.xsd" xmlns=""><action time="2018-04-24T12:34:12.813Z">completed</action><agent agent="SAPBE_DEV_C700S021_01" QMgr="PBMFTD01"><systemInfo architecture="IA64N" name="HP-UX" version="B.11.31"/></agent>SAPBE_DEV_C700S021_01</line><line>BFGCL0213I: agent SAPBE_DEV_C700S021_01 responded to ping in 0.403 seconds.</line><line>BFGCL0212I: Issuing ping request to agent SAPBE_DEV_C700S021_01<line>BFGAN0049I: Move operation assigned transfer ID: 414d512050424d4654443031202020202030cb5a2b29fb24</line><line>BFGAN0053I: Successfully completion of move operation: 414d512050424d4654443031202020202030cb5a2b29fb24</line><line>BFGCL0211I: </line><line>BFGCL0211I: complete:</line><line>BFGCL0210W: Transfer Complete</line></stdout></result></callResult></call></transferSet><statistics><retryCount>0</retryCount><numFileFailures>0</numFileFailures><numFileWarnings>0</numFileWarnings></statistics></transaction>`

I should break the events in this way,

Event first half,

Tue Apr 24 07:34:13 CDT 2018 name=QUEUE_msg_received event_id=ID:414d5120504443524444303220202020f4decc5ac8811c21 msg_dest=MFTTRANSACTION.EVENTS.LOG.INPUT.QUEUE msg_body=

Event second half,

<?xml version="1.0" encoding="UTF-8"?><transaction version="5.00" ID="414d512050424d4654443031202020202030cb5a1d29fb24" agentRole="callAgent" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="TransferLog.xsd" xmlns=""><action time="2018-04-24T12:34:12.813Z">completed</action><agent agent="SAPBE_DEV_C700S021_01" QMgr="PBMFTD01"><systemInfo architecture="IA64N" name="HP-UX" version="B.11.31"/></agent>SAPBE_DEV_C700S021_01</line><line>BFGCL0213I: agent SAPBE_DEV_C700S021_01 responded to ping in 0.403 seconds.</line><line>BFGCL0212I: Issuing ping request to agent SAPBE_DEV_C700S021_01<line>BFGAN0049I: Move operation assigned transfer ID: 414d512050424d4654443031202020202030cb5a2b29fb24</line><line>BFGAN0053I: Successfully completion of move operation: 414d512050424d4654443031202020202030cb5a2b29fb24</line><line>BFGCL0211I: </line><line>BFGCL0211I: complete:</line><line>BFGCL0210W: Transfer Complete</line></stdout></result></callResult></call></transferSet><statistics><retryCount>0</retryCount><numFileFailures>0</numFileFailures><numFileWarnings>0</numFileWarnings></statistics></transaction>

I should route event first half to nullqueue while sending the second half to indexer. Kindly help me with regex for null proper line breaking as well as regex for routing to nullqueue.

Thanks in advance

Tags (1)
0 Karma
1 Solution

somesoni2
Revered Legend

Try this in props.conf of your indexer/heavy forwarder (the SEDCMD will remove the text you don't want in your event)

[yourSourceTypeHere]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\w+\s+\w+\s+\d{2}\:\d{2}\:\d{2})
TIME_PREFIX = time=\"
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N%z
MAX_TIMESTAMP_LOOKAHEAD = 24
SEDCMD-removeheader = s/^(.+)\smsg_body\=//g

View solution in original post

0 Karma

somesoni2
Revered Legend

Try this in props.conf of your indexer/heavy forwarder (the SEDCMD will remove the text you don't want in your event)

[yourSourceTypeHere]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\w+\s+\w+\s+\d{2}\:\d{2}\:\d{2})
TIME_PREFIX = time=\"
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N%z
MAX_TIMESTAMP_LOOKAHEAD = 24
SEDCMD-removeheader = s/^(.+)\smsg_body\=//g
0 Karma

jsanjeb
Explorer

Great!! It worked. Thanks for the fix.

0 Karma

jsanjeb
Explorer

But the header which i have mentioned in question ie event first half is added by forwarder to the data. Is there a way we can stop forwarder from adding this?

0 Karma
Get Updates on the Splunk Community!

.conf25 Community Recap

Hello Splunkers, And just like that, .conf25 is in the books! What an incredible few days — full of learning, ...

Splunk App Developers | .conf25 Recap & What’s Next

If you stopped by the Builder Bar at .conf25 this year, thank you! The retro tech beer garden vibes were ...

Congratulations to the 2025-2026 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...