transforms.conf and props.conf for replacing/substituing values in data that we are indexing

[simon21]

I want to replace/substitute the string value in the raw data with new string value. I have successfully done the substitution using props.conf (SED-cmd)

But now I need to do the same with transforms.conf

Scenario:

Date,filedsA
19-Jun,Ignore
19-Jun,Ignore
19-Jun,Ignore
19-Jun,ABC
19-Jun,DEF

From the above data, I need to replace/substitute "Ignore" with "Deferred"

So far, my transform.conf looks like this:

[replacement]
REGEX = ^Ignore
FORMAT = deferred
DEST_KEY = _raw

Props.conf
[replacement1]
TRANSFORMS-replace = replacement
BREAK_ONLY_BEFORE_DATE =
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = csv
KV_MODE = none
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Structured
description = Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled = false
pulldown_type = true

[woodcock · Accepted solution]

The problem is the stanza header in props.conf should be [<YourSourcetypeHere>] but in any case, don't do it like that; use SEDCMD like this:

[<YourSourcetypeHere>]
SEDCMD-replace_ignore_with_deferred = s/Ignore/deferred/

But even more, it is poor form to modify your data this way, because it gives auditors the impression that this is the way the data really originated/always-was. It would be better to use a lookup YourLookupHere.csv that has data like this:

oldFieldA,newFieldA
Ignore,disabled

Then use it like this:

Your Spl Here ... | lookup YourLookupHere.csc fieldA OUTPUT oldFieldA AS fieldA OUTPUT newFieldA AS fieldA

[simon21]

I do not wish to use SPL. Also, tried and successfully tested using props.conf (SEDCMD). But I particularity need to use the transforms.conf and props.conf to replace/substitute the values.

[woodcock]

Did you notice that at the start of this answer I also told you what is wrong with your original attempt?

[woodcock]

Why? It is far more complicated. It sound like you need an answer for a test.

[FrankVl]

And what exactly is your question / problem? I guess this overwrites your entire raw event with just "deferred", which is not what you want?

Is that sample data your full raw events or only part of it? Does this replacement have to be done at index time? If so, why does SED not work? If not, why not simply do this with a LOOKUP or EVAL?

[simon21]

Yes, it indeed replaCes the entire event. This is my entire sample data. Replacement needs to e done at index time. SED did work, but we particularly need to make it work using transforms.conf

Need it to happen via the conf files only. Hence not looking at lookup option or the eval SPL options.

[FrankVl]

If you really need to do it like this, I guess you need to change the REGEX and FORMAT parts, such that the REGEX matches the full raw event, captures the bits that you want to keep and then in FORMAT refer to the capture groups to keep the rest of the event.

But I'm not 100% sure how this exactly works with csv indexed_extractions and somehow with this sample data you shared it does not make too much sense that this config does anything (since Ignore is not actually at the start of the event).

But in general, it would work something like this in transforms.conf:

[replacement]
REGEX = (.*?)Ignore(.*)
FORMAT = $1deferred$2
DEST_KEY = _raw

See also: https://docs.splunk.com/Documentation/Splunk/latest/Data/Anonymizedata