topic Re: Transformations and props behavior in Getting Data In

Transformations and props behavior

cmlombardo — Mon, 02 Oct 2023 13:27:18 GMT

I would like to understand better how transformations work, in terms of priority and data flow.

Let's say I have 3 transformations in place, that I want to apply to a sourcetype (e.g. "test"):

[tr1]

[tr2]

[tr3]

Can somebody explain to me what the difference is (if any) between these 2 props.conf:

----1----

[test] TRANSFORMS-1=tr1 TRANSFORMS-2=tr2 TRANSFORMS-3=tr3

---2----

[test] TRANSFORMS-1=tr1,tr2,tr3

Is the resulting data transformation the same on both cases?

Thank you!

Re: Transformations and props behavior

gcusello — Mon, 02 Oct 2023 13:35:32 GMT

Hi @cmlombardo,

I'm not sure (someone else could contradict me!) but they should be the same thing.

Anyway, I prefer the second solution to be more sure about the sequence of transformation application.

because the execution sequence could be important.

Ciao.

Giuseppe

Re: Transformations and props behavior

richgalloway — Mon, 02 Oct 2023 13:43:52 GMT

As I understand it, transforms listed separately are processed in lexicographical order whereas those listed in a single TRANSFORMS setting are processed in the order given.

IOW, this

[test] TRANSFORMS-3=tr3 TRANSFORMS-1=tr1 TRANSFORMS-2=tr2

is the same as this

[test] TRANSFORMS-1=tr1 TRANSFORMS-2=tr2 TRANSFORMS-3=tr3

But this

TRANSFORMS-1=tr3,tr1,tr2

will process the transforms in the listed order.

Re: Transformations and props behavior

cmlombardo — Mon, 02 Oct 2023 13:56:03 GMT

So, if they are processed in lexicographical order, then the result should be the same once the data passes through my 2 transformation examples.

Best practice, as I understand it, is to list the transformations in the second form

TRANFORMS=tr1,tr2,tr3

so that there is no doubt on the order they are processed.

Re: Transformations and props behavior

richgalloway — Mon, 02 Oct 2023 13:59:18 GMT

It's not necessarily true that the data will be the same after each example. In some (many/most?) cases, the order of transformations could be significant and is why I recommend using the second format.

Re: Transformations and props behavior

gcusello — Mon, 02 Oct 2023 14:02:12 GMT

hi @cmlombardo ,

the order of transformation is relevant!

only for example, if you read at https://docs.splunk.com/Documentation/Splunk/9.1.1/Forwarding/Routeandfilterdatad#Filter_event_data_and_send_to_queues , if you want to keep some events and discard the rest, you have to executebefore the transofrmation on all the events (regex=.) and then the tranformation of a part of data, if you change the order, the tranformation doesn't work.

Ciao.

Giuseppe

Re: Transformations and props behavior

cmlombardo — Mon, 02 Oct 2023 14:03:58 GMT

In the example, the lexicographic order will process the transformations in the same order, as TRANSFORMS-1 comes before TRANSFORMS-2 (and so on).