Getting Data In

props.conf and transforms.conf does not work

Communicator

Hi,i dont the content field to be forward to indexer, i configured props.conf and transforms.conf but it does not work. Anyone can assist?

<xml>
    <Field1>123</Field1>
    <Field2>456</Field2>
    <Body>Not to be forward to indexer</Body>
    <Field3>333</Field3>
    <content>not to be forward</content>
</xml>

at props.conf,

[test]
TRANSFORMS-null = content

at transforms.conf,

[content]
REGEX = <content>(.+)</content> 
DEST_KEY = queue 
FORMAT = nullQueue

i want to have the final result at the indexer as:

<xml>
    <Field1>123</Field1>
    <Field2>456</Field2>
    <Body>Not to be forward to indexer</Body>
    <Field3>333</Field3>  
</xml>

I configured the props.conf and transforms.conf for both forwarder and indexer but it does not work. Any problem with my config files?

Tags (1)
0 Karma
1 Solution

Legend

Yes - your configuration files have problems. For one thing, the transform you listed, if it worked, would send the entire event to the null queue, not just the line that you have listed. That might actually work, if you treat each line of the input as a separate event (and fix the regular expression), but that is not what I would recommend. Try this instead:

props.conf

[test]
BREAK_ONLY_BEFORE =\<xml\>
KV_MODE = xml
TRANSFORMS-test1 = content

transforms.conf

[content]
SOURCE_KEY=_raw
REGEX=(.*?)\<content\>.*?\</content\>(.*)
DEST_KEY=_raw
FORMAT=$1$2

This should completely remove the <content> tag and its contents, while leaving the rest of the event unchanged.

props.conf and transforms.conf change how the data is parsed. So if you are using a Universal Forwarder, the props.conf and transforms.conf files belong on the indexer(s). Heavy forwarders will parse the data, so if you are using one, then you must put these .conf files on the heavy forwarder.

For more information on the parsing settings, look at the event processing topics in the Getting Data In manual.

View solution in original post

Legend

Yes - your configuration files have problems. For one thing, the transform you listed, if it worked, would send the entire event to the null queue, not just the line that you have listed. That might actually work, if you treat each line of the input as a separate event (and fix the regular expression), but that is not what I would recommend. Try this instead:

props.conf

[test]
BREAK_ONLY_BEFORE =\<xml\>
KV_MODE = xml
TRANSFORMS-test1 = content

transforms.conf

[content]
SOURCE_KEY=_raw
REGEX=(.*?)\<content\>.*?\</content\>(.*)
DEST_KEY=_raw
FORMAT=$1$2

This should completely remove the <content> tag and its contents, while leaving the rest of the event unchanged.

props.conf and transforms.conf change how the data is parsed. So if you are using a Universal Forwarder, the props.conf and transforms.conf files belong on the indexer(s). Heavy forwarders will parse the data, so if you are using one, then you must put these .conf files on the heavy forwarder.

For more information on the parsing settings, look at the event processing topics in the Getting Data In manual.

View solution in original post

Explorer

Hi @lguinn2 , I am trying to remove some of the sensitive information to be indexed by Splunk.

But these configurations are not working ,even after getting the configuration reflected over btool and validating the regex over SPL.
Can you please have a look on it?

props.conf
[o365:management:activity]
TRANSFORMS-anonymize = info-anonymizer
KV_MODE = json
TRUNCATE = 10485760

transforms.conf
[info-anonymizer]
DEST_KEY = _raw
FORMAT = $1$2
REGEX = (.*\"SensitiveInformationDetections\"\:\s\{)\"DetectedValues\"\:\s\[.*\]\,\s(\"ResultsTruncated\"\:.*)


Have already Validated regex over SPL, It is working fine.

|regex _raw="(.*\"SensitiveInformationDetections\"\:\s\{)\"DetectedValues\"\:\s\[.*\]\,\s(\"ResultsTruncated\"\:.*)"

and

|rex field=_raw "(?<before>.*\"SensitiveInformationDetections\"\:\s\{)\"DetectedValues\"\:\s\[.*\]\,\s(?<after>\"ResultsTruncated\"\:.*)"
|eval _raw=before+""+after

0 Karma

Communicator

max_events = 10000000 also not enough, any other alternative? thks

0 Karma

Legend

Data Preview looks at the file BEFORE it is parsed, so you will still see the content there. When you search sourcetype=test, you should not see the content

0 Karma

Legend

If the events are large, add this to props.conf

TRUNCATE = 0
MAX_EVENTS = 10000

TRUNCATE is the maximum number of bytes in an event. Setting it to 0 means "no limit."

MAX_EVENTS is the maximum number of lines in an event. The default is 256. I arbitrarily set it to 10,000 in the example.

0 Karma

Communicator

thks, i tested it worked if the contents in the tag are few characters but if the content in the tag is more than few hundred Kbytes info, it cannot work. Any other suggestions?

0 Karma

Communicator

THks for providing the info. i had configured the .conf files on the heavy forwarder, so i should not see the tag and its contents in the data preview, when the input is configured? But i still see the tag and its contents. Pls assist. thks

0 Karma
Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes
and swag!