topic Re: How to prevent Splunk from creating a new field and value in an event given that a field contains the symbol "e in Splunk Enterprise

How to prevent Splunk from creating a new field and value in an event given that a field contains the symbol "equal"?

jordilazo — Thu, 29 Sep 2022 19:21:27 GMT

Hi,

I have been able to enter the following data in splunk through key value with the following format:

sourcetype="excel_page_10" mail_sender="jordi@jordilazo.com" mail_recipient="lazo@jordilazo.es" mail_date_ep="1635qqqqwe2160816.0" mail_nummails="1222asdasd.adasdqweqw" mail_level="0@qw....." mail_info="NO" mail_removal="NO" mail_area="Miami" mail_subject="RE: NMXWZFOG< >VSTI" mail_id="XXX-KKKK-NNNN-KNZI" mail_reviewcomment="Comentario:ÑC<AZR=@P""\a"

As can be seen in the image, splunk has been able to correctly classify all the fields and value. However it has created a new field called AZR with the value @P. This is because it has detected an = inside the comment review value and created it.

What do I have to modify in the props and transform so that it detects the entire reviewcomment field as 1 single value and includes the symbol =?

Re: How to prevent splunk from creating a new field and value in an event given that a field contains the symbol "e

chaker — Thu, 29 Sep 2022 00:53:05 GMT

props.conf
[INSERT_SOURCETYPE_HERE]
EXTRACT-mail_reviewcomment = mail_reviewcomment="(?<mail_reviewcomment>.+?)"

No need to modify transforms.conf in this case.

Re: How to prevent splunk from creating a new field and value in an event given that a field contains the symbol "e

jordilazo — Thu, 29 Sep 2022 08:45:41 GMT

Hi Chaker,

My props looks like:

[preproc]

SHOULD_LINEMERGE = false

NO_BINARY_CHECK = true

TIME_PREFIX = timestamp=

MAX_TIMESTAMP_LOOKAHEAD = 10

CHARSET = UTF-8

KV_MODE = auto

TRANSFORMS-dynamic_sourcetype = dynamic_sourcetype

EXTRACT-test = mail_reviewcomment="(?<mail_reviewcomment>.+?)"

But is still not working. Any idea? should I introduce the events like CSV?

Re: How to prevent Splunk from creating a new field and value in an event given that a field contains the symbol "e

chaker — Thu, 29 Sep 2022 09:05:58 GMT

Hi @jordilazo

This props.conf contains index time and search time instructions (the extraction is search time)

You could put this props.conf file on your searchhead, or create a new one with just the extract entry provided.

Re: How to prevent Splunk from creating a new field and value in an event given that a field contains the symbol "e

jordilazo — Thu, 29 Sep 2022 13:17:24 GMT

Hi Chaker,

Unfortunately I have tried everything but Splunk keeps automatically creating a new field for the mail_reviewcomment field.
Maybe the problem is in the input.conf?
I run the script automatically.

Here I leave my configuration:

HF:

inputs.conf [script://script.py] disabled = 0 index = jordi_emails interval = 55 14 * * * source = external sourcetype = preproc

props.conf [preproc] SHOULD_LINEMERGE = false NO_BINARY_CHECK = true TIME_PREFIX = timestamp= MAX_TIMESTAMP_LOOKAHEAD = 10 CHARSET = UTF-8 KV_MODE = auto TRANSFORMS-dynamic_sourcetype = dynamic_sourcetype [test] EXTRACT-mail_reviewcomment = mail_reviewcomment="(?<mail_reviewcomment>.+?)"

transform.conf [dynamic_sourcetype] REGEX = sourcetype="(\w+)" FORMAT = sourcetype::$1 DEST_KEY = MetaData:Sourcetype

SH:

Looks exactly the same like the props.conf in my HF (as you said).

Re: How to prevent Splunk from creating a new field and value in an event given that a field contains the symbol "e

chaker — Thu, 29 Sep 2022 14:15:22 GMT

Try changing the sourcetype reference from this:

[test] EXTRACT-mail_reviewcomment = mail_reviewcomment="(?<mail_reviewcomment>.+?)"

To this:

[preproc] EXTRACT-mail_reviewcomment = mail_reviewcomment="(?<mail_reviewcomment>.+?)"

If you still can't get it working, I suggest using the field extractor tool. You can paste the regex straight in there. Below is the result I get.

https://docs.splunk.com/Documentation/Splunk/9.0.1/Knowledge/ExtractfieldsinteractivelywithIFX

Re: How to prevent Splunk from creating a new field and value in an event given that a field contains the symbol "e

jordilazo — Thu, 29 Sep 2022 15:42:33 GMT

Hi Chaker,

I think it is working but the problem is that I have another event that contains double quotes" inside the value.

Here is an example:

mail_reviewcomment="Comentario:ÑC<2KLAZR=@Q"&"\A"

So splunk is getting confused again.

How can I modify the REGEX so splunk will process the field?

Re: How to prevent Splunk from creating a new field and value in an event given that a field contains the symbol "e

chaker — Fri, 30 Sep 2022 00:26:19 GMT

Hi @jordilazo

As long as mail_reviewcomment is the last field in all the events, the following regex will work.

EXTRACT-mail_reviewcomment = mail_reviewcomment="(?<mail_reviewcomment>.+?)$

If the schema changes, and this field is no longer the last field in the event, this regex will not work, and will require some changes.

Re: How to prevent Splunk from creating a new field and value in an event given that a field contains the symbol "e

jordilazo — Mon, 03 Oct 2022 07:23:26 GMT

This solution works but it wasn't exactly what I wanted.

In this post this person asked the same and its looks like there is no solution.
How to escape equal signs (=) in key value data? - Splunk Community