topic Re: transforms.conf and props.conf for replacing/substituing values in data that we are indexing in Getting Data In

transforms.conf and props.conf for replacing/substituing values in data that we are indexing

simon21 — Wed, 30 Sep 2020 01:04:49 GMT

I want to replace/substitute the string value in the raw data with new string value. I have successfully done the substitution using props.conf (SED-cmd)

But now I need to do the same with transforms.conf

Scenario:

Date,filedsA
19-Jun,Ignore
19-Jun,Ignore
19-Jun,Ignore
19-Jun,ABC
19-Jun,DEF

From the above data, I need to replace/substitute "Ignore" with "Deferred"

So far, my transform.conf looks like this:

[replacement]
REGEX = ^Ignore
FORMAT = deferred
DEST_KEY = _raw

Props.conf
[replacement1]
TRANSFORMS-replace = replacement
BREAK_ONLY_BEFORE_DATE =
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = csv
KV_MODE = none
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Structured
description = Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled = false
pulldown_type = true

Re: transforms.conf and props.conf for replacing/substituing values in data that we are indexing

FrankVl — Fri, 28 Jun 2019 12:51:25 GMT

And what exactly is your question / problem? I guess this overwrites your entire raw event with just "deferred", which is not what you want?

Is that sample data your full raw events or only part of it? Does this replacement have to be done at index time? If so, why does SED not work? If not, why not simply do this with a LOOKUP or EVAL?

Re: transforms.conf and props.conf for replacing/substituing values in data that we are indexing

woodcock — Sat, 29 Jun 2019 14:51:29 GMT

The problem is the stanza header in props.conf should be [<YourSourcetypeHere>] but in any case, don't do it like that; use SEDCMD like this:

[<YourSourcetypeHere>]
SEDCMD-replace_ignore_with_deferred = s/Ignore/deferred/

But even more, it is poor form to modify your data this way, because it gives auditors the impression that this is the way the data really originated/always-was. It would be better to use a lookup YourLookupHere.csv that has data like this:

oldFieldA,newFieldA
Ignore,disabled

Then use it like this:

Your Spl Here ... | lookup YourLookupHere.csc fieldA OUTPUT oldFieldA AS fieldA OUTPUT newFieldA AS fieldA

Re: transforms.conf and props.conf for replacing/substituing values in data that we are indexing

simon21 — Thu, 04 Jul 2019 06:51:32 GMT

I do not wish to use SPL. Also, tried and successfully tested using props.conf (SEDCMD). But I particularity need to use the transforms.conf and props.conf to replace/substitute the values.

Re: transforms.conf and props.conf for replacing/substituing values in data that we are indexing

simon21 — Thu, 04 Jul 2019 06:53:41 GMT

Yes, it indeed replaCes the entire event. This is my entire sample data. Replacement needs to e done at index time. SED did work, but we particularly need to make it work using transforms.conf

Need it to happen via the conf files only. Hence not looking at lookup option or the eval SPL options.

Re: transforms.conf and props.conf for replacing/substituing values in data that we are indexing

FrankVl — Thu, 04 Jul 2019 07:40:56 GMT

If you really need to do it like this, I guess you need to change the REGEX and FORMAT parts, such that the REGEX matches the full raw event, captures the bits that you want to keep and then in FORMAT refer to the capture groups to keep the rest of the event.

But I'm not 100% sure how this exactly works with csv indexed_extractions and somehow with this sample data you shared it does not make too much sense that this config does anything (since Ignore is not actually at the start of the event).

But in general, it would work something like this in transforms.conf:

[replacement]
REGEX = (.*?)Ignore(.*)
FORMAT = $1deferred$2
DEST_KEY = _raw

Re: transforms.conf and props.conf for replacing/substituing values in data that we are indexing

woodcock — Thu, 04 Jul 2019 14:24:58 GMT

Why? It is far more complicated. It sound like you need an answer for a test.

Re: transforms.conf and props.conf for replacing/substituing values in data that we are indexing

woodcock — Sat, 06 Jul 2019 01:25:08 GMT

Did you notice that at the start of this answer I also told you what is wrong with your original attempt?