Transformations and props behavior

[cmlombardo]

I would like to understand better how transformations work, in terms of priority and data flow.

Let's say I have 3 transformations in place, that I want to apply to a sourcetype (e.g. "test"):

[tr1]

[tr2]

[tr3]

Can somebody explain to me what the difference is (if any) between these 2 props.conf:

----1----

[test]
TRANSFORMS-1=tr1
TRANSFORMS-2=tr2
TRANSFORMS-3=tr3

---2----

[test]
TRANSFORMS-1=tr1,tr2,tr3

Is the resulting data transformation the same on both cases?

Thank you!

[richgalloway]

As I understand it, transforms listed separately are processed in lexicographical order whereas those listed in a single TRANSFORMS setting are processed in the order given.

IOW, this

[test]
TRANSFORMS-3=tr3
TRANSFORMS-1=tr1
TRANSFORMS-2=tr2

is the same as this

[test]
TRANSFORMS-1=tr1
TRANSFORMS-2=tr2
TRANSFORMS-3=tr3

But this

TRANSFORMS-1=tr3,tr1,tr2

will process the transforms in the listed order.

[cmlombardo]

So, if they are processed in lexicographical order, then the result should be the same once the data passes through my 2 transformation examples.

Best practice, as I understand it, is to list the transformations in the second form

TRANFORMS=tr1,tr2,tr3

so that there is no doubt on the order they are processed.

[gcusello]

hi @cmlombardo ,

the order of transformation is relevant!

only for example, if you read at https://docs.splunk.com/Documentation/Splunk/9.1.1/Forwarding/Routeandfilterdatad#Filter_event_data_... , if you want to keep some events and discard the rest, you have to executebefore the transofrmation on all the events (regex=.) and then the tranformation of a part of data, if you change the order, the tranformation doesn't work.

Ciao.

Giuseppe

[richgalloway]

It's not necessarily true that the data will be the same after each example.  In some (many/most?) cases, the order of transformations could be significant and is why I recommend using the second format.

[cmlombardo]

In the example, the lexicographic order will process the transformations in the same order, as TRANSFORMS-1 comes before TRANSFORMS-2 (and so on).

[gcusello]

Hi @cmlombardo,

I'm not sure (someone else could contradict me!) but they should be the same thing.

Anyway, I prefer the second solution to be more sure about the sequence of transformation application.

because the execution sequence could be important.

Ciao.

Giuseppe