Splunk Search

Handling XML Events With Embedded Carriage Returns

jwhughes58
Communicator

I'm getting events from a device and on rare occasions the event data contains an embedded carriage return. I've tried multiple ways of splitting the data but the embedded carriage return causes the event to be split incorrectly. As an example:

syslog header : <?xml> <alerts> <alert> <info>This is a single line alert</info> </alert> </alerts>
syslog header : <?xml> <alerts> <alert> <info>This is a multi-line alert
This is alert line 2
This is alert line 3
This is alert line 4</info> </alert> </alerts>
syslog header : <?xml> <alerts> <alert> <info>This is a single line alert</info> </alert> </alerts>

In the local/props.conf I've tried many ways of splitting the syslog header from the XML so I can process the XML. All the things I've tried don't work. Any ideas?

TIA,
Joe

0 Karma
1 Solution

maciep
Champion

If the goal is to get Splunk to parse the events correctly before indexing, then I think you want to use LINE_BREAKER to tell Splunk exactly how to break your events. These settings would go in props.conf on your parsing infrastructure (most likely your indexer).

This is untested, but should give you an idea.

LINE_BREAKER = ([\r\n]+)(?=syslog header)
SHOULD_LINEMERGE = false

So we tell Splunk that our line breaker is carriage return / line feed, but only if the next line starts with syslog header. And then we make sure Splunk doesn't try to merge lines, because we told it exactly how they need to be broken.

View solution in original post

0 Karma

maciep
Champion

If the goal is to get Splunk to parse the events correctly before indexing, then I think you want to use LINE_BREAKER to tell Splunk exactly how to break your events. These settings would go in props.conf on your parsing infrastructure (most likely your indexer).

This is untested, but should give you an idea.

LINE_BREAKER = ([\r\n]+)(?=syslog header)
SHOULD_LINEMERGE = false

So we tell Splunk that our line breaker is carriage return / line feed, but only if the next line starts with syslog header. And then we make sure Splunk doesn't try to merge lines, because we told it exactly how they need to be broken.

0 Karma

jwhughes58
Communicator

Your line breaker is close to what was in the props.conf that I wrote. This is the final solution

LINE_BREAKER = ([\r\n])(\w+\s+\d+\s+\d{2}:\d{2}:\d{2}\s+)

Once I had that in props.conf the data got split correctly. Thanks for the help.

0 Karma

niketnilay
Legend

@jwhughes58, can you try replace command to replace the line feed characters with spaces:

<Your Base Search>
| eval raw_without_line_feed= replace(_raw,"\n\r"," ")
| rex field=raw_without_line_feed "\<\?xml\>\s(?<xml_data>.*)\<\/alerts\>"
| eval xml_data=xml_data."<alerts>"

PS: rex and eval are just as example for simple field extraction if you should use your own based on actual data.
Once you have validated that the extraction is working fine, you can overwrite you _raw in first eval.

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...