topic Modify Field with Regex at Index Time in Splunk Search

Modify Field with Regex at Index Time

dbuehler — Tue, 27 Oct 2020 21:22:44 GMT

Hey guys,

I have IIS logs that are logging multiple IPs to the X-Forwarded-For field as below:

114.119.136.78,+162.158.119.25

I would like to apply a regex to the X-Forwarded-For field at index time to ensure the field only contains the first IP, like:

114.119.136.78

In other words, anything after the first comma should be cut out of the field.

So far I have tried to achieve this with the following props/transforms:

#props [iis] TRANSFORMS-rm-extra-ips = rm_extra_ips #transforms [rm_extra_ips] SOURCE_KEY = field:X_Forwarded_For REGEX = ^(.+?),

How do I do this?

Thanks!

Re: Modify Field with Regex at Index Time

bowesmana — Tue, 27 Oct 2020 23:21:20 GMT

If you want to modify the field at index time you would use SEDCMD, described in props.conf docs here

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf

and also see this

https://docs.splunk.com/Documentation/Splunk/8.1.0/Data/Anonymizedata

and also some community posts, e.g.

https://community.splunk.com/t5/Getting-Data-In/How-to-configure-SEDCMD-in-props-conf/m-p/196633

https://community.splunk.com/t5/Getting-Data-In/SEDCMD-a-field/m-p/413484

Hope this helps

Re: Modify Field with Regex at Index Time

dbuehler — Wed, 28 Oct 2020 12:33:42 GMT

Thanks, I was looking into SEDCMD originally, I've used it for other purposes before.

Can SEDCMD operate on just one field? Or does it have to operate on the entire event (_raw)?

Re: Modify Field with Regex at Index Time

isoutamo — Wed, 28 Oct 2020 14:39:38 GMT

Based on props.conf's spec no, if must operate towards _raw.

SEDCMD-<class> = <sed script> * Only used at index time. * Commonly used to anonymize incoming data at index time, such as credit card or social security numbers. For more information, search the online documentation for "anonymize data." * Used to specify a sed script which Splunk software applies to the _raw field. * A sed script is a space-separated list of sed commands. Currently the following subset of sed commands is supported: * replace (s) and character substitution (y). * Syntax: * replace - s/regex/replacement/flags * regex is a perl regular expression (optionally containing capturing groups). * replacement is a string to replace the regex match. Use \n for back references, where "n" is a single digit. * flags can be either: g to replace all matches, or a number to replace a specified match. * substitute - y/string1/string2/ * substitutes the string1[i] with string2[i] * No default.

r. Ismo

Re: Modify Field with Regex at Index Time

dbuehler — Wed, 04 Nov 2020 17:59:30 GMT

For some reason my props.conf config isn't being applied to my data.

I've found that this is the regex I need:

s/,+[.0-9\:a-z]*//g

And this regex works perfectly when run manually, i.e. using a sed command against a text file with a sample event:

cat sample.txt | sed 's/,+[.0-9\:a-z]*//g'

My props.conf (placed on my two indexers in /opt/splunk/etc/apps/my-iis-app/local) is configured to apply to the 'iis' sourcetype, which is correct, and looks like:

[iis]

SEDCMD-remove-extra-ips = s/,+[.0-9\:a-z]*//g

After restarting Splunk, the events are coming in un-modified. It appears the regex isn't being applied at all, as even if I change my config to a very simple test regex, that doesn't work either, e.g.:

[iis]
SEDCMD-test = s/10/test/g

Any ideas?

Re: Modify Field with Regex at Index Time

isoutamo — Wed, 04 Nov 2020 18:46:38 GMT

Shoul you have \. Instead of . In your shed expression? First match to . second one match to every character.

Re: Modify Field with Regex at Index Time

dbuehler — Wed, 04 Nov 2020 19:10:25 GMT

That regex works just the same in manual tests, but does not work when applied as a SEDCMD in props.conf.

It seems clear my props SEDCMD is not being applied to my data at all.

If I run:

/opt/splunk/bin/splunk btool --debug props list |grep SEDCMD

I see my setting showing up in the output. Is there any other way to troubleshoot if my SEDCMD is being applied?