topic props.conf and transforms.conf does not work in Getting Data In

props.conf and transforms.conf does not work

SplunkCSIT — Thu, 17 Apr 2014 02:12:25 GMT

Hi,i dont the content field to be forward to indexer, i configured props.conf and transforms.conf but it does not work. Anyone can assist?

<xml>
    <Field1>123</Field1>
    <Field2>456</Field2>
    <Body>Not to be forward to indexer</Body>
    <Field3>333</Field3>
    <content>not to be forward</content>
</xml>

at props.conf,

[test]
TRANSFORMS-null = content

at transforms.conf,

[content]
REGEX = <content>(.+)</content> 
DEST_KEY = queue 
FORMAT = nullQueue

i want to have the final result at the indexer as:

<xml>
    <Field1>123</Field1>
    <Field2>456</Field2>
    <Body>Not to be forward to indexer</Body>
    <Field3>333</Field3>  
</xml>

I configured the props.conf and transforms.conf for both forwarder and indexer but it does not work. Any problem with my config files?

Re: props.conf and transforms.conf does not work

lguinn2 — Thu, 17 Apr 2014 04:33:20 GMT

Yes - your configuration files have problems. For one thing, the transform you listed, if it worked, would send the entire event to the null queue, not just the line that you have listed. That might actually work, if you treat each line of the input as a separate event (and fix the regular expression), but that is not what I would recommend. Try this instead:

props.conf

[test]
BREAK_ONLY_BEFORE =\<xml\>
KV_MODE = xml
TRANSFORMS-test1 = content

transforms.conf

[content]
SOURCE_KEY=_raw
REGEX=(.*?)\<content\>.*?\</content\>(.*)
DEST_KEY=_raw
FORMAT=$1$2

This should completely remove the <content> tag and its contents, while leaving the rest of the event unchanged.

props.conf and transforms.conf change how the data is parsed. So if you are using a Universal Forwarder, the props.conf and transforms.conf files belong on the indexer(s). Heavy forwarders will parse the data, so if you are using one, then you must put these .conf files on the heavy forwarder.

For more information on the parsing settings, look at the event processing topics in the Getting Data In manual.

Re: props.conf and transforms.conf does not work

SplunkCSIT — Thu, 17 Apr 2014 06:17:52 GMT

THks for providing the info. i had configured the .conf files on the heavy forwarder, so i should not see the tag and its contents in the data preview, when the input is configured? But i still see the tag and its contents. Pls assist. thks

Re: props.conf and transforms.conf does not work

SplunkCSIT — Thu, 17 Apr 2014 16:11:54 GMT

thks, i tested it worked if the contents in the tag are few characters but if the content in the tag is more than few hundred Kbytes info, it cannot work. Any other suggestions?

Re: props.conf and transforms.conf does not work

lguinn2 — Thu, 17 Apr 2014 17:37:43 GMT

If the events are large, add this to props.conf

TRUNCATE = 0
MAX_EVENTS = 10000

TRUNCATE is the maximum number of bytes in an event. Setting it to 0 means "no limit."

MAX_EVENTS is the maximum number of lines in an event. The default is 256. I arbitrarily set it to 10,000 in the example.

Re: props.conf and transforms.conf does not work

lguinn2 — Thu, 17 Apr 2014 17:39:03 GMT

Data Preview looks at the file BEFORE it is parsed, so you will still see the content there. When you search sourcetype=test, you should not see the content

Re: props.conf and transforms.conf does not work

SplunkCSIT — Fri, 18 Apr 2014 06:26:51 GMT

max_events = 10000000 also not enough, any other alternative? thks

Re: props.conf and transforms.conf does not work

payal4296 — Tue, 15 Sep 2020 11:11:19 GMT

Hi @lguinn2 , I am trying to remove some of the sensitive information to be indexed by Splunk.

But these configurations are not working ,even after getting the configuration reflected over btool and validating the regex over SPL.
Can you please have a look on it?

props.conf
[o365:management:activity]
TRANSFORMS-anonymize = info-anonymizer
KV_MODE = json
TRUNCATE = 10485760

transforms.conf
[info-anonymizer]
DEST_KEY = _raw
FORMAT = $1$2
REGEX = (.*\"SensitiveInformationDetections\"\:\s\{)\"DetectedValues\"\:\s\[.*\]\,\s(\"ResultsTruncated\"\:.*)

Have already Validated regex over SPL, It is working fine.

|regex _raw="(.*\"SensitiveInformationDetections\"\:\s\{)\"DetectedValues\"\:\s\[.*\]\,\s(\"ResultsTruncated\"\:.*)"

and

|rex field=_raw "(?<before>.*\"SensitiveInformationDetections\"\:\s\{)\"DetectedValues\"\:\s\[.*\]\,\s(?<after>\"ResultsTruncated\"\:.*)"
|eval _raw=before+""+after

Re: props.conf and transforms.conf does not work

payal4296 — Tue, 15 Sep 2020 11:37:16 GMT

https://community.splunk.com/t5/Getting-Data-In/Masking-sensitive-information-from-event/m-p/519664#M87857