How to prevent Splunk from creating a new field and value in an event given that a field contains the symbol "equal"?

[jordilazo]

Hi,

I have been able to enter the following data in splunk through key value with the following format:

sourcetype="excel_page_10" mail_sender="jordi@jordilazo.com" mail_recipient="lazo@jordilazo.es" mail_date_ep="1635qqqqwe2160816.0" mail_nummails="1222asdasd.adasdqweqw" mail_level="0@qw....." mail_info="NO" mail_removal="NO" mail_area="Miami" mail_subject="RE: NMXWZFOG< >VSTI" mail_id="XXX-KKKK-NNNN-KNZI" mail_reviewcomment="Comentario:ÑC<AZR=@P""\a"

 As can be seen in the image, splunk has been able to correctly classify all the fields and value. However it has created a new field called AZR with the value @P. This is because it has detected an = inside the comment review value and created it.

What do I have to modify in the props and transform so that it detects the entire reviewcomment field as 1 single value and includes the symbol =?

[chaker]

 Hi @jordilazo 

This props.conf contains index time and search time instructions (the extraction is search time)

You could put this props.conf file on your searchhead, or create a new one with just the extract entry provided.

[jordilazo]

Hi Chaker,

Unfortunately I have tried everything but Splunk keeps automatically creating a new field for the mail_reviewcomment field.
Maybe the problem is in the input.conf?
I run the script automatically.

Here I leave my configuration:

HF:

inputs.conf

[script://script.py]
disabled = 0
index = jordi_emails
interval = 55 14 * * *
source = external
sourcetype = preproc

props.conf

[preproc]
SHOULD_LINEMERGE = false
NO_BINARY_CHECK = true
TIME_PREFIX = timestamp=
MAX_TIMESTAMP_LOOKAHEAD = 10
CHARSET = UTF-8
KV_MODE = auto
TRANSFORMS-dynamic_sourcetype = dynamic_sourcetype

[test]
EXTRACT-mail_reviewcomment = mail_reviewcomment="(?<mail_reviewcomment>.+?)"

transform.conf

[dynamic_sourcetype]
REGEX = sourcetype="(\w+)"
FORMAT = sourcetype::$1
DEST_KEY = MetaData:Sourcetype

SH:

Looks exactly the same like the props.conf in my HF (as you said).

[chaker · Accepted solution]

Try changing the sourcetype reference from this:

[test]
EXTRACT-mail_reviewcomment = mail_reviewcomment="(?<mail_reviewcomment>.+?)"

To this:

[preproc]
EXTRACT-mail_reviewcomment = mail_reviewcomment="(?<mail_reviewcomment>.+?)"

If you still can't get it working, I suggest using the field extractor tool. You can paste the regex straight in there. Below is the result I get.

https://docs.splunk.com/Documentation/Splunk/9.0.1/Knowledge/ExtractfieldsinteractivelywithIFX

[jordilazo]

This solution works but it wasn't exactly what I wanted.

In this post this person asked the same and its looks like there is no solution.
How to escape equal signs (=) in key value data? - Splunk Community

[jordilazo]

Hi Chaker,

I think it is working but the problem is that I have another event that contains double quotes" inside the value.

Here is an example: 

mail_reviewcomment="Comentario:ÑC<2KLAZR=@Q"&"\A"

So splunk is getting confused again.

How can I modify the REGEX so splunk will process the field?

[chaker]

Hi @jordilazo 

As long as mail_reviewcomment is the last field in all the events, the following regex will work.

EXTRACT-mail_reviewcomment = mail_reviewcomment="(?<mail_reviewcomment>.+?)$

If the schema changes, and this field is no longer the last field in the event, this regex will not work, and will require some changes.

[chaker]

props.conf
[INSERT_SOURCETYPE_HERE]
EXTRACT-mail_reviewcomment = mail_reviewcomment="(?<mail_reviewcomment>.+?)"

No need to modify transforms.conf in this case.

[jordilazo]

Hi Chaker,

My props looks like:

[preproc]

SHOULD_LINEMERGE = false

NO_BINARY_CHECK = true

TIME_PREFIX = timestamp=

MAX_TIMESTAMP_LOOKAHEAD = 10

CHARSET = UTF-8

KV_MODE = auto

TRANSFORMS-dynamic_sourcetype = dynamic_sourcetype

EXTRACT-test = mail_reviewcomment="(?<mail_reviewcomment>.+?)"

But is still not working. Any idea? should I introduce the events like CSV?