transforms.conf – supporting alternatives in REGEX and numbering the alternative-groups in the FORMAT lines

[SonnyB]

In the transforms.conf file, how do I support the alternatives on the REGEX line with the
corresponding FORMAT line group’s numbering? See the example attempt below:
I can have either 4 groups or 6 groups on the REGEX line
(that’s why there is a pipe "|" there;
InComingList SOURCE_KEY could have either 4 groups or 6 groups).
Corresponding FORMAT line then need to be able to: either support 4 groups’ matches
or 6 groups’ matches. How to do that?
How to number the groups - increase the numbers or restart-the-numbering again
for the part after the pipe?

[Unfortunately transforms.conf specification does not support
conditional-logic presently].

Any quick insights will be deeply appreciated!

---

[testTransformPara]

REGEX=((\w+?)_(\w+)=(\d+):(\d+)) | ((\w+?)_(\w+)=(\d+):(\d+)@(\w+)=(\d+))

FORMAT = (FIELD1::$1 FIELD2::$2 FIELD3::$3 FIELD4::$4) | (FIELD1::$1 FIELD2::$2 FIELD3::$3 FIELD4::$4 FIELD5::$5 FIELD4::$6)

SOURCE_KEY = InComingList

MV_ADD = true

---

[Ayn]

Why not just have two separate transforms? Like you say transforms.conf does not support this kind of conditional extraction and I think it's likely it never will either since you can solve the problem by using separate transforms.

[dwaddle]

Also, "better practices" (I would say "best" but that is subjective) suggest that you build context-specific apps for your various inputs and their processing. If you have a single props.conf with 'thousands of lines' in it, then you may want to consider breaking those down into different apps and managing your revisions in smaller units.

[Ayn]

Refer to dwaddle's comments - if you have that complicated of a process for changing props.conf but for some reason not transforms.conf, you have other problems to solve than this.

[SonnyB]

Ayn: Can't get into more specifics here, but, imagine a scenario, where you already have thousands of lines of sourcetypes defined in props.conf and if any of the existing paras are changed, that then mandates a regression testing cycle for all the related clients. If the props is not touched in this fashion, then this dilemma is circumvented. transforms.conf's existing fields are matched perfectly and remain unperturbed. Some optional new fields are added, that are then tested out for the new client-apps.

[dwaddle]

I agree with Ayn here. The proper solution is multiple transforms.conf stanzas with multiple references to them within props.conf. If your change management / code freeze processes are such that you can change one Splunk config file but not a related Splunk config file, perhaps your processes are not aligned with reality in a live Splunk deployment.

[Ayn]

...that relate to props.conf but not to transforms.conf? This sounds very much like a constructed problem.

[SonnyB]

some frozen-code issues etc.

[Ayn]

Could you give me a reason why you wouldn't be able to change/add props settings?

[SonnyB]

Ayn: If there is a constraint (say, for some reason), that the corresponding props.conf can not be changed. So, the new transforms' para creation is not possible. In that case, we have to design the alternatives of the REGEX and FORMAT in the existing paragraph expressions only. What is the formal guideline from Splunk on this, if it exists. Could not find any direct examples/discussion of this topic yet.

[bmacias84]

Can you give some data samples. There may be a better approach.