Handling XML Events With Embedded Carriage Returns

[jwhughes58]

I'm getting events from a device and on rare occasions the event data contains an embedded carriage return. I've tried multiple ways of splitting the data but the embedded carriage return causes the event to be split incorrectly. As an example:

syslog header : <?xml> <alerts> <alert> <info>This is a single line alert</info> </alert> </alerts>
syslog header : <?xml> <alerts> <alert> <info>This is a multi-line alert
This is alert line 2
This is alert line 3
This is alert line 4</info> </alert> </alerts>
syslog header : <?xml> <alerts> <alert> <info>This is a single line alert</info> </alert> </alerts>

In the local/props.conf I've tried many ways of splitting the syslog header from the XML so I can process the XML. All the things I've tried don't work. Any ideas?

TIA,
Joe

[maciep · Accepted solution]

If the goal is to get Splunk to parse the events correctly before indexing, then I think you want to use LINE_BREAKER to tell Splunk exactly how to break your events. These settings would go in props.conf on your parsing infrastructure (most likely your indexer).

This is untested, but should give you an idea.

LINE_BREAKER = ([\r\n]+)(?=syslog header)
SHOULD_LINEMERGE = false

So we tell Splunk that our line breaker is carriage return / line feed, but only if the next line starts with syslog header. And then we make sure Splunk doesn't try to merge lines, because we told it exactly how they need to be broken.

[jwhughes58]

Your line breaker is close to what was in the props.conf that I wrote. This is the final solution

LINE_BREAKER = ([\r\n])(\w+\s+\d+\s+\d{2}:\d{2}:\d{2}\s+)

Once I had that in props.conf the data got split correctly. Thanks for the help.

[niketn]

@jwhughes58, can you try replace command to replace the line feed characters with spaces:

<Your Base Search>
| eval raw_without_line_feed= replace(_raw,"\n\r"," ")
| rex field=raw_without_line_feed "\<\?xml\>\s(?<xml_data>.*)\<\/alerts\>"
| eval xml_data=xml_data."<alerts>"

PS: rex and eval are just as example for simple field extraction if you should use your own based on actual data.
Once you have validated that the extraction is working fine, you can overwrite you _raw in first eval.