Hi,i dont the content field to be forward to indexer, i configured props.conf and transforms.conf but it does not work. Anyone can assist?
<xml>
<Field1>123</Field1>
<Field2>456</Field2>
<Body>Not to be forward to indexer</Body>
<Field3>333</Field3>
<content>not to be forward</content>
</xml>
at props.conf,
[test]
TRANSFORMS-null = content
at transforms.conf,
[content]
REGEX = <content>(.+)</content>
DEST_KEY = queue
FORMAT = nullQueue
i want to have the final result at the indexer as:
<xml>
<Field1>123</Field1>
<Field2>456</Field2>
<Body>Not to be forward to indexer</Body>
<Field3>333</Field3>
</xml>
I configured the props.conf and transforms.conf for both forwarder and indexer but it does not work. Any problem with my config files?
Yes - your configuration files have problems. For one thing, the transform you listed, if it worked, would send the entire event to the null queue, not just the line that you have listed. That might actually work, if you treat each line of the input as a separate event (and fix the regular expression), but that is not what I would recommend. Try this instead:
props.conf
[test]
BREAK_ONLY_BEFORE =\<xml\>
KV_MODE = xml
TRANSFORMS-test1 = content
transforms.conf
[content]
SOURCE_KEY=_raw
REGEX=(.*?)\<content\>.*?\</content\>(.*)
DEST_KEY=_raw
FORMAT=$1$2
This should completely remove the <content>
tag and its contents, while leaving the rest of the event unchanged.
props.conf
and transforms.conf
change how the data is parsed. So if you are using a Universal Forwarder, the props.conf
and transforms.conf
files belong on the indexer(s). Heavy forwarders will parse the data, so if you are using one, then you must put these .conf
files on the heavy forwarder.
For more information on the parsing settings, look at the event processing topics in the Getting Data In manual.
Yes - your configuration files have problems. For one thing, the transform you listed, if it worked, would send the entire event to the null queue, not just the line that you have listed. That might actually work, if you treat each line of the input as a separate event (and fix the regular expression), but that is not what I would recommend. Try this instead:
props.conf
[test]
BREAK_ONLY_BEFORE =\<xml\>
KV_MODE = xml
TRANSFORMS-test1 = content
transforms.conf
[content]
SOURCE_KEY=_raw
REGEX=(.*?)\<content\>.*?\</content\>(.*)
DEST_KEY=_raw
FORMAT=$1$2
This should completely remove the <content>
tag and its contents, while leaving the rest of the event unchanged.
props.conf
and transforms.conf
change how the data is parsed. So if you are using a Universal Forwarder, the props.conf
and transforms.conf
files belong on the indexer(s). Heavy forwarders will parse the data, so if you are using one, then you must put these .conf
files on the heavy forwarder.
For more information on the parsing settings, look at the event processing topics in the Getting Data In manual.
Hi @lguinn2 , I am trying to remove some of the sensitive information to be indexed by Splunk.
But these configurations are not working ,even after getting the configuration reflected over btool and validating the regex over SPL.
Can you please have a look on it?
props.conf
[o365:management:activity]
TRANSFORMS-anonymize = info-anonymizer
KV_MODE = json
TRUNCATE = 10485760
transforms.conf
[info-anonymizer]
DEST_KEY = _raw
FORMAT = $1$2
REGEX = (.*\"SensitiveInformationDetections\"\:\s\{)\"DetectedValues\"\:\s\[.*\]\,\s(\"ResultsTruncated\"\:.*)
Have already Validated regex over SPL, It is working fine.
|regex _raw="(.*\"SensitiveInformationDetections\"\:\s\{)\"DetectedValues\"\:\s\[.*\]\,\s(\"ResultsTruncated\"\:.*)"
and
|rex field=_raw "(?<before>.*\"SensitiveInformationDetections\"\:\s\{)\"DetectedValues\"\:\s\[.*\]\,\s(?<after>\"ResultsTruncated\"\:.*)"
|eval _raw=before+""+after
max_events = 10000000 also not enough, any other alternative? thks
Data Preview looks at the file BEFORE it is parsed, so you will still see the content there. When you search sourcetype=test
, you should not see the content
If the events are large, add this to props.conf
TRUNCATE = 0
MAX_EVENTS = 10000
TRUNCATE
is the maximum number of bytes in an event. Setting it to 0 means "no limit."
MAX_EVENTS
is the maximum number of lines in an event. The default is 256. I arbitrarily set it to 10,000 in the example.
thks, i tested it worked if the contents in the
THks for providing the info. i had configured the .conf files on the heavy forwarder, so i should not see the