I'm working with a data source that has two different versions. In one version the information is double quoted while the other version is single quoted. This is causing me issues because the single quoted information will still have the single quotes while the double quoted won't have any quotes. This is throwing counts off since the single quoted string and the unquoted string are not the same. Without using SEDCMD since we want the actual raw source, I've been trying to work out how to do this. I've got this in a search
| eval raw=_raw
| rex field=raw mode=sed "s/\'/\"/g"
| rex field=raw "\[(?<audit_event>[^\:]+)\:(?<vendor_severity>[^\:]+).+(?<vendor_xml>\<vendor.+\<\/vendor\>)"
Now I'm trying to convert it to props and transforms. My props.conf
EXTRACT-vendor_raw = (?<raw>^.*$)
REPORT-vendor_extract_fields = vendor_replace_single_quotes, vendor_fields
KV_MODE = xml
REGEX = \[(?<audit_event>[^\:]+)\:(?<vendor_severity>[^\:]+).+(?<vendor_xml>\<vendor.+\<\/vendor\>)
SOURCE = raw
What I can't figure out is how do the replace like in the search either in props.conf or transforms.conf. Everything I've found uses the SEDCMD. Any thoughts on this?
I don't want to replace the single quotes with double quotes in _raw for legal reasons. I can use eval after the report, but I would have to change every value that is single quoted and I am not guaranteed the names. I would like to make a backup of _raw, change single quotes to double quotes, and get the name value pairs.
In transforms.conf, get rid of the empty [vendor_replace_single_quotes] stanza.
In props.conf, get rid of vendor_replace_single_quotes in auto extraction vendor_extract_fields;
then, add the EVAL:
# Automatically apply transform named "vendor_fields";
# 'vendor_xml' field may contain single or double quotes
REPORT-vendor_extract_fields = vendor_fields
# Replace any single quote in 'vendor_xml' field with double quote
EVAL-vendor_xml = replace(vendor_xml, "'", "\"")
Check to make sure the above segment is under sourcetype, source, or host in props.conf that matches your search. Your vendor_xml field should no longer contain single quotes. Nothing in _raw is changed. (The order in which these appeared in props.conf isn't important; in fact, it would be better to use Web UI to do these things and allow Splunk to order props.conf automatically although you will lose comments.)
Of course, if another field, or other extracted fields, have undesired single quote, you can do the same to them.