topic Re: Why does a transforms report stanza have issues with source key? in Splunk Dev

Why does a transforms report stanza have issues with source key?

landen99 — Wed, 02 Aug 2017 16:14:25 GMT

Specifically, I have a search-time transform which works against _raw but not against a source key. When it only specifies a regex and a source key, it does not work, but when the source key is removed, it does work. It would be really good to also gain a general understanding of what contributes to source key not working.

sourcetype: splunkd
regex: \/+handshake\/+reply\/+(?<deploy_client_name>[^\/]+)\/+(?<deploy_client>\S+)
source key: _raw OR channel; the first works and the second does not.
https://regex101.com/r/eI7V1B/2

The field channel extracts normally by default but does not seem to be available for the transform. Is it possible to get the transfer to work against the field, channel?

In any Splunk environment, you should see that channel is extracted by default with the following search:

index=_internal sourcetype=splunkd component=PubSubSvr reply handshake PubSubSvr | dedup channel | table channel deploy*

You can use my regex (above) in a transform on your own search head to duplicate the issue that I am describing. Here is the transform:

[ds_msg_client_handshake]
CLEAN_KEYS = 0
REGEX = \/handshake\/reply\/(?P<deploy_client_name>[^\/]+)\/(?P<deploy_client>\S+)
SOURCE_KEY = channel

Re: Why does a transforms report stanza have issues with source key?

somesoni2 — Wed, 02 Aug 2017 16:22:30 GMT

This transform is doing index-time activity or search time?

Re: Why does a transforms report stanza have issues with source key?

bheemireddi — Tue, 29 Sep 2020 15:11:38 GMT

how is the channel field getting extracted? using another transforms?KV_MODE? did notice something before it doesn't do much if it is from the KV_MODE.

Can you paste sample transforms where you are using SOURCE_KEY? and also how the channel field getting extracted in this case?

Re: Why does a transforms report stanza have issues with source key?

landen99 — Wed, 02 Aug 2017 17:20:05 GMT

search time

Re: Why does a transforms report stanza have issues with source key?

landen99 — Wed, 02 Aug 2017 17:22:57 GMT

The channel field is extracted by default against the splunkd sourcetype. I didn't do anything to extract "channel". You can see the events in your own deployments by searching:

index=_internal sourcetype=splunkd component=PubSubSvr reply handshake PubSubSvr | dedup channel | table channel deploy*

Re: Why does a transforms report stanza have issues with source key?

landen99 — Wed, 02 Aug 2017 17:30:44 GMT

Here is the transform stanza:

[ds_msg_client_handshake]
CLEAN_KEYS = 0
REGEX = \/handshake\/reply\/(?P<deploy_client_name>[^\/]+)\/(?P<deploy_client>\S+)
SOURCE_KEY = channel

Re: Why does a transforms report stanza have issues with source key?

bheemireddi — Tue, 29 Sep 2020 15:11:44 GMT

Yes, all the field extractions for the splunkd source type are through KV_MODE, since this is auto by default. So I am not surprised it did not work as the SOURCE_KEY, as the field extractions also have the precedence of their operations.

REPORT goes first before KV_MODE, and the channel field won't be available for your REPORT in the transforms.conf

For more understanding on the precedence, probably below thread might help
https://answers.splunk.com/answers/475935/for-a-field-user-which-has-precedence-an-eval-defi.html

Re: Why does a transforms report stanza have issues with source key?

landen99 — Wed, 02 Aug 2017 18:42:39 GMT

So is the solution then to configure a props to extract the field first so that it is available before the transform's report call? Or is there a better solution?

Re: Why does a transforms report stanza have issues with source key?

bheemireddi — Tue, 29 Sep 2020 15:11:52 GMT

Yes, In order to use for the field to be SOURCE_KEY in the REPORT/transforms - you either get that through EXTRACT OR REPORT and make sure the stanza that is extracting the SOURCE_KEY evaluates before the stanza where you want to use it

Re: Why does a transforms report stanza have issues with source key?

landen99 — Thu, 03 Aug 2017 14:04:02 GMT

I think that you meant to make this a reply to my comment on your answer above...