topic Re: transforms.conf – supporting alternatives in REGEX and numbering the alternative-groups in the FORMAT lines in Splunk Search

transforms.conf – supporting alternatives in REGEX and numbering the alternative-groups in the FORMAT lines

SonnyB — Wed, 01 May 2013 17:10:34 GMT

In the transforms.conf file, how do I support the alternatives on the REGEX line with the
corresponding FORMAT line group’s numbering? See the example attempt below:
I can have either 4 groups or 6 groups on the REGEX line
(that’s why there is a pipe "|" there;
InComingList SOURCE_KEY could have either 4 groups or 6 groups).
Corresponding FORMAT line then need to be able to: either support 4 groups’ matches
or 6 groups’ matches. How to do that?
How to number the groups - increase the numbers or restart-the-numbering again
for the part after the pipe?

[Unfortunately transforms.conf specification does not support
conditional-logic presently].

Any quick insights will be deeply appreciated!

[testTransformPara]

REGEX=((\w+?)_(\w+)=(\d+):(\d+)) | ((\w+?)_(\w+)=(\d+):(\d+)@(\w+)=(\d+))

FORMAT = (FIELD1::$1 FIELD2::$2 FIELD3::$3 FIELD4::$4) | (FIELD1::$1 FIELD2::$2 FIELD3::$3 FIELD4::$4 FIELD5::$5 FIELD4::$6)

SOURCE_KEY = InComingList

MV_ADD = true

Re: transforms.conf – supporting alternatives in REGEX and numbering the alternative-groups in the FORMAT lines

bmacias84 — Wed, 01 May 2013 22:13:10 GMT

Can you give some data samples. There may be a better approach.

Re: transforms.conf – supporting alternatives in REGEX and numbering the alternative-groups in the FORMAT lines

Ayn — Thu, 02 May 2013 06:50:48 GMT

Why not just have two separate transforms? Like you say transforms.conf does not support this kind of conditional extraction and I think it's likely it never will either since you can solve the problem by using separate transforms.

Re: transforms.conf – supporting alternatives in REGEX and numbering the alternative-groups in the FORMAT lines

SonnyB — Fri, 03 May 2013 17:18:12 GMT

Ayn: If there is a constraint (say, for some reason), that the corresponding props.conf can not be changed. So, the new transforms' para creation is not possible. In that case, we have to design the alternatives of the REGEX and FORMAT in the existing paragraph expressions only. What is the formal guideline from Splunk on this, if it exists. Could not find any direct examples/discussion of this topic yet.

Re: transforms.conf – supporting alternatives in REGEX and numbering the alternative-groups in the FORMAT lines

Ayn — Fri, 03 May 2013 17:52:33 GMT

Could you give me a reason why you wouldn't be able to change/add props settings?

Re: transforms.conf – supporting alternatives in REGEX and numbering the alternative-groups in the FORMAT lines

SonnyB — Fri, 03 May 2013 17:56:38 GMT

some frozen-code issues etc.

Re: transforms.conf – supporting alternatives in REGEX and numbering the alternative-groups in the FORMAT lines

Ayn — Fri, 03 May 2013 18:07:58 GMT

...that relate to props.conf but not to transforms.conf? This sounds very much like a constructed problem.

Re: transforms.conf – supporting alternatives in REGEX and numbering the alternative-groups in the FORMAT lines

dwaddle — Fri, 03 May 2013 19:27:10 GMT

I agree with Ayn here. The proper solution is multiple transforms.conf stanzas with multiple references to them within props.conf. If your change management / code freeze processes are such that you can change one Splunk config file but not a related Splunk config file, perhaps your processes are not aligned with reality in a live Splunk deployment.

Re: transforms.conf – supporting alternatives in REGEX and numbering the alternative-groups in the FORMAT lines

SonnyB — Fri, 03 May 2013 21:09:02 GMT

Ayn: Can't get into more specifics here, but, imagine a scenario, where you already have thousands of lines of sourcetypes defined in props.conf and if any of the existing paras are changed, that then mandates a regression testing cycle for all the related clients. If the props is not touched in this fashion, then this dilemma is circumvented. transforms.conf's existing fields are matched perfectly and remain unperturbed. Some optional new fields are added, that are then tested out for the new client-apps.

Re: transforms.conf – supporting alternatives in REGEX and numbering the alternative-groups in the FORMAT lines

Ayn — Sat, 04 May 2013 07:23:30 GMT

Refer to dwaddle's comments - if you have that complicated of a process for changing props.conf but for some reason not transforms.conf, you have other problems to solve than this.

Re: transforms.conf – supporting alternatives in REGEX and numbering the alternative-groups in the FORMAT lines

dwaddle — Mon, 06 May 2013 16:49:22 GMT

Also, "better practices" (I would say "best" but that is subjective) suggest that you build context-specific apps for your various inputs and their processing. If you have a single props.conf with 'thousands of lines' in it, then you may want to consider breaking those down into different apps and managing your revisions in smaller units.