topic Re: why are props.conf keys not recognized in Getting Data In

why are props.conf keys not recognized

FritzWittwer_ol — Tue, 29 Sep 2020 16:37:09 GMT

I have a verry strange behaviour in props.conf, the following stanza does not work:

[SDCS-liveclone-xxxxxx-st_XmlWinEventLog:Security]
TRANSFORMS-SDCS-liveclone-xxxxxx-wes-route1 = SDCS-liveclone-xxxxxx-wes_Filter
TRANSFORMS-SDCS-liveclone-xxxxxx-wes-route2 = SDCS-liveclone-xxxxxx-ls_SwitchIndex
TRANSFORMS-SDCS-liveclone-xxxxxx-wes-route3 = SDCS-common-xxxxxx-DiscardEvents
TRANSFORMS-SDCS-liveclone-xxxxxx-wes-route4 = SDCS-liveclone-xxxxxx-wse_UsernameAnonymizer
TRANSFORMS-SDCS-liveclone-xxxxxx-wes-route5 = SDCS-liveclone-xxxxxx-wse_UsernameAnonymizer
TRANSFORMS-SDCS-liveclone-xxxxxx-wes-route6 = SDCS-liveclone-xxxxxx-wse_UsernameAnonymizer
TRANSFORMS-SDCS-liveclone-xxxxxx-wes-route7 = SDCS-liveclone-xxxxxx-ls_SwitchRoute
TRANSFORMS-SDCS-liveclone-xxxxxx-wes-route8 = SDCS-common-xxxxxx-CleanSourceType

it stops processing at the transform TRANSFORMS-SDCS-liveclone-xxxxxx-wes-route7

if an X is added to this rule as follows: (line 8 after SDCS, more to the end e.g. ...SDCS-livecloneX... does not work)

[SDCS-liveclone-xxxxxx-st_XmlWinEventLog:Security]
TRANSFORMS-SDCS-liveclone-xxxxxx-wes-route1 = SDCS-liveclone-xxxxxx-wes_Filter
TRANSFORMS-SDCS-liveclone-xxxxxx-wes-route2 = SDCS-liveclone-xxxxxx-ls_SwitchIndex
TRANSFORMS-SDCS-liveclone-xxxxxx-wes-route3 = SDCS-common-xxxxxx-DiscardEvents
TRANSFORMS-SDCS-liveclone-xxxxxx-wes-route4 = SDCS-liveclone-xxxxxx-wse_UsernameAnonymizer
TRANSFORMS-SDCS-liveclone-xxxxxx-wes-route5 = SDCS-liveclone-xxxxxx-wse_UsernameAnonymizer
TRANSFORMS-SDCS-liveclone-xxxxxx-wes-route6 = SDCS-liveclone-xxxxxx-wse_UsernameAnonymizer
TRANSFORMS-SDCSX-liveclone-xxxxxx-wes-route7 = SDCS-liveclone-xxxxxx-ls_SwitchRoute
TRANSFORMS-SDCS-liveclone-xxxxxx-wes-route8 = SDCS-common-xxxxxx-CleanSourceType

the props get applied

Just as background info

the same key is not used anywhere else
similar keys are used, e.g. the same set but wit -ls- instead of -wes- this works
xxxxxx is a company name, in this case 6 characters
the transforms have to be applied in this order (clone event, change _tcp_routing, change index, discard events whose routing has not been changed, over write usernames which follow to a certain pattern, change routing to the final destination, reset original source type. This is needed to hand over some events to another Splunk plattform, but masking the usernames while doing so.

below are the referenced transforms, the ones named SDCS-common-... are kept in another app

[SDCS-liveclone-xxxxxx-Clone_XmlWinEventLogSecurity]
REGEX = .
DEST_KEY=_TCP_ROUTING
FORMAT=nothing
CLONE_SOURCETYPE = SDCS-liveclone-xxxxxx-st_XmlWinEventLog:Security

[SDCS-liveclone-xxxxxx-wes_Filter]
SOURCE_KEY = MetaData:Source
REGEX = WinEventLog:Security
DEST_KEY=_TCP_ROUTING
FORMAT=SDCS-liveclone-xxxxxx-target-route

[SDCS-liveclone-xxxxxx-ls_SwitchIndex]
REGEX = .
DEST_KEY=_MetaData:Index
FORMAT=yyyyyy
WRITE_META = true

[SDCS-common-xxxxxx-DiscardEvents]
SOURCE_KEY = _TCP_ROUTING
REGEX = nothing
DEST_KEY = queue
FORMAT = nullQueue

[SDCS-common-xxxxxx-CleanSourceType]
SOURCE_KEY=MetaData:Sourcetype
REGEX = (?mi)^[^:]*::SDCS-(live|verify)clone-xxxxxx-st_(.*)$
DEST_KEY=MetaData:Sourcetype
FORMAT=sourcetype::$2
WRITE_META = true

[SDCS-liveclone-xxxxxx-wse_UsernameAnonymizer]
REGEX = (?mi)^(.*? __pattern_to_match_username__ (<.*?)$
FORMAT = $1**HiddenSwisscomUser**$2
DEST_KEY = _raw

[SDCS-liveclone-xxxxxx-ls_SwitchRoute]
SOURCE_KEY=_TCP_ROUTING
REGEX = SDCS-liveclone-xxxxxx-target-route
#REGEX = .
DEST_KEY=_TCP_ROUTING
FORMAT=zzzzzz

Re: why are props.conf keys not recognized

horsefez — Fri, 03 Nov 2017 08:50:08 GMT

Hi FritzWittwer,

Please for the sake of readability make it more clear where you are adding an "x" to. It took me quite a while to figure it out.
Did you check if your corresponding transforms.conf stanza has a typo?
Could you show the corresponding transforms.conf stanza?

Re: why are props.conf keys not recognized

koshyk — Fri, 03 Nov 2017 10:01:20 GMT

Are all xxxx different from each other? if NOT there are duplicates.
Can you run btool on props and transforms to see how it looks finally?

Re: why are props.conf keys not recognized

jgbricker — Fri, 03 Nov 2017 10:21:20 GMT

When you add the X you may have caused Splunk to perform route 7 last due to ASCII ordering. SDCS would come before SDCSX.

Re: why are props.conf keys not recognized

FritzWittwer_ol — Fri, 03 Nov 2017 10:23:51 GMT

the xxxxxx are always the same, the transforms have all different key values as they are numbered. The transform SDCS-liveclone-xxxxxx-wse_UsernameAnonymizer is listed 3 times (lines 5 to 7 ) on purpose as it should be applied 3 times, the username may occur up to 3 times in the same event. Exactly the same configuration works for if the Keys are SDCS-verifyclone.... instead of SDCS-liveclone...

Re: why are props.conf keys not recognized

FritzWittwer_ol — Fri, 03 Nov 2017 12:36:20 GMT

No, i don't think this is the issue, after figuring out my workaround with the x, I cleaned it up a bit by adding a letter on each key:

[SDCS-liveclone-xxxxxx-st_XmlWinEventLog:Security]
TRANSFORMS-SDCSa-liveclone-xxxxxx-wes-route1 = SDCS-liveclone-xxxxxx-wes_Filter
TRANSFORMS-SDCSb-liveclone-xxxxxx-wes-route2 = SDCS-liveclone-xxxxxx-ls_SwitchIndex
TRANSFORMS-SDCSc-liveclone-xxxxxx-wes-route3 = SDCS-common-xxxxxx-DiscardEvents
TRANSFORMS-SDCSd-liveclone-xxxxxx-wes-route4 = SDCS-liveclone-xxxxxx-wse_UsernameAnonymizer
TRANSFORMS-SDCSe-liveclone-xxxxxx-wes-route5 = SDCS-liveclone-xxxxxx-wse_UsernameAnonymizer
TRANSFORMS-SDCSf-liveclone-xxxxxx-wes-route6 = SDCS-liveclone-xxxxxx-wse_UsernameAnonymizer
TRANSFORMS-SDCSg-liveclone-xxxxxx-wes-route7 = SDCS-liveclone-xxxxxx-ls_SwitchRoute
TRANSFORMS-SDCSh-liveclone-xxxxxx-wes-route8 = SDCS-common-xxxxxx-CleanSourceType

This keeps the order the same as the original and it works just fine.

Re: why are props.conf keys not recognized

jgbricker — Fri, 03 Nov 2017 17:29:31 GMT

Are you saying adding the letters corrected the ordering? That would be fixing ASCII ordering as I suggested. Otherwise, just to clarify you are saying that the TRANSFORMS-SDCSg-liveclone-xxxxxx-wes-route7 = SDCS-liveclone-xxxxxx-ls_SwitchRoute isn't processing or it doesn't process TRANSFORMS-SDCSh-liveclone-xxxxxx-wes-route8 = SDCS-common-xxxxxx-CleanSourceType

Re: why are props.conf keys not recognized

FritzWittwer_ol — Thu, 09 Nov 2017 07:02:52 GMT

No, adding the letter after TRANSFORMS-SDCS on all rules does not change the ASCII ordering, it is the same ordering as I started with, there it is enforced with the number at the end of the name.
I have in fact the exact same configuration without the letters which works, the only difference it that the rules are named TRANSFORMS-SDCSh-verifyclone... instead of TRANSFORMS-SDCSh-liveclone....

Re: why are props.conf keys not recognized

FritzWittwer_ol — Tue, 20 Mar 2018 07:56:49 GMT

Giving an answer myself just do document what i found:
I am using key names as below, the number 1 is the part which is used to order the keys
TRANSFORMS-SDCSa-liveclone-xxxxxx-wes-route1

It works in the folksinging two cases

the number is in the first 20 chars after TRANSFORMS-
there are no - (minus) signs in the key after TRANSFORMS-