Splunk Search

transforms.conf – supporting alternatives in REGEX and numbering the alternative-groups in the FORMAT lines

Explorer

In the transforms.conf file, how do I support the alternatives on the REGEX line with the
corresponding FORMAT line group’s numbering? See the example attempt below:
I can have either 4 groups or 6 groups on the REGEX line
(that’s why there is a pipe "|" there;
InComingList SOURCE_KEY could have either 4 groups or 6 groups).
Corresponding FORMAT line then need to be able to: either support 4 groups’ matches
or 6 groups’ matches. How to do that?
How to number the groups - increase the numbers or restart-the-numbering again
for the part after the pipe?

[Unfortunately transforms.conf specification does not support
conditional-logic presently].

Any quick insights will be deeply appreciated!


[testTransformPara]

REGEX=((\w+?)_(\w+)=(\d+):(\d+)) | ((\w+?)_(\w+)=(\d+):(\d+)@(\w+)=(\d+))

FORMAT = (FIELD1::$1 FIELD2::$2 FIELD3::$3 FIELD4::$4) | (FIELD1::$1 FIELD2::$2 FIELD3::$3 FIELD4::$4 FIELD5::$5 FIELD4::$6)

SOURCE_KEY = InComingList

MV_ADD = true

Legend

Why not just have two separate transforms? Like you say transforms.conf does not support this kind of conditional extraction and I think it's likely it never will either since you can solve the problem by using separate transforms.

SplunkTrust
SplunkTrust

Also, "better practices" (I would say "best" but that is subjective) suggest that you build context-specific apps for your various inputs and their processing. If you have a single props.conf with 'thousands of lines' in it, then you may want to consider breaking those down into different apps and managing your revisions in smaller units.

0 Karma

Legend

Refer to dwaddle's comments - if you have that complicated of a process for changing props.conf but for some reason not transforms.conf, you have other problems to solve than this.

Explorer

Ayn: Can't get into more specifics here, but, imagine a scenario, where you already have thousands of lines of sourcetypes defined in props.conf and if any of the existing paras are changed, that then mandates a regression testing cycle for all the related clients. If the props is not touched in this fashion, then this dilemma is circumvented. transforms.conf's existing fields are matched perfectly and remain unperturbed. Some optional new fields are added, that are then tested out for the new client-apps.

0 Karma

SplunkTrust
SplunkTrust

I agree with Ayn here. The proper solution is multiple transforms.conf stanzas with multiple references to them within props.conf. If your change management / code freeze processes are such that you can change one Splunk config file but not a related Splunk config file, perhaps your processes are not aligned with reality in a live Splunk deployment.

Legend

...that relate to props.conf but not to transforms.conf? This sounds very much like a constructed problem.

Explorer

some frozen-code issues etc.

0 Karma

Legend

Could you give me a reason why you wouldn't be able to change/add props settings?

0 Karma

Explorer

Ayn: If there is a constraint (say, for some reason), that the corresponding props.conf can not be changed. So, the new transforms' para creation is not possible. In that case, we have to design the alternatives of the REGEX and FORMAT in the existing paragraph expressions only. What is the formal guideline from Splunk on this, if it exists. Could not find any direct examples/discussion of this topic yet.

0 Karma

Champion

Can you give some data samples. There may be a better approach.

State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!