Splunk Dev

Why does a transforms report stanza have issues with source key?

landen99
Motivator

Specifically, I have a search-time transform which works against _raw but not against a source key. When it only specifies a regex and a source key, it does not work, but when the source key is removed, it does work. It would be really good to also gain a general understanding of what contributes to source key not working.

sourcetype: splunkd
regex: \/+handshake\/+reply\/+(?<deploy_client_name>[^\/]+)\/+(?<deploy_client>\S+)
source key: _raw OR channel; the first works and the second does not.
https://regex101.com/r/eI7V1B/2

The field channel extracts normally by default but does not seem to be available for the transform. Is it possible to get the transfer to work against the field, channel?

In any Splunk environment, you should see that channel is extracted by default with the following search:

index=_internal sourcetype=splunkd component=PubSubSvr reply handshake PubSubSvr | dedup channel | table channel deploy*

You can use my regex (above) in a transform on your own search head to duplicate the issue that I am describing. Here is the transform:

[ds_msg_client_handshake]
CLEAN_KEYS = 0
REGEX = \/handshake\/reply\/(?P<deploy_client_name>[^\/]+)\/(?P<deploy_client>\S+)
SOURCE_KEY = channel
Tags (1)
0 Karma
1 Solution

bheemireddi
Communicator

Yes, all the field extractions for the splunkd source type are through KV_MODE, since this is auto by default. So I am not surprised it did not work as the SOURCE_KEY, as the field extractions also have the precedence of their operations.

REPORT goes first before KV_MODE, and the channel field won't be available for your REPORT in the transforms.conf

For more understanding on the precedence, probably below thread might help
https://answers.splunk.com/answers/475935/for-a-field-user-which-has-precedence-an-eval-defi.html

View solution in original post

bheemireddi
Communicator

Yes, In order to use for the field to be SOURCE_KEY in the REPORT/transforms - you either get that through EXTRACT OR REPORT and make sure the stanza that is extracting the SOURCE_KEY evaluates before the stanza where you want to use it

0 Karma

landen99
Motivator

I think that you meant to make this a reply to my comment on your answer above...

0 Karma

bheemireddi
Communicator

Yes, all the field extractions for the splunkd source type are through KV_MODE, since this is auto by default. So I am not surprised it did not work as the SOURCE_KEY, as the field extractions also have the precedence of their operations.

REPORT goes first before KV_MODE, and the channel field won't be available for your REPORT in the transforms.conf

For more understanding on the precedence, probably below thread might help
https://answers.splunk.com/answers/475935/for-a-field-user-which-has-precedence-an-eval-defi.html

landen99
Motivator

So is the solution then to configure a props to extract the field first so that it is available before the transform's report call? Or is there a better solution?

0 Karma

bheemireddi
Communicator

how is the channel field getting extracted? using another transforms?KV_MODE? did notice something before it doesn't do much if it is from the KV_MODE.

Can you paste sample transforms where you are using SOURCE_KEY? and also how the channel field getting extracted in this case?

0 Karma

landen99
Motivator

Here is the transform stanza:

[ds_msg_client_handshake]
CLEAN_KEYS = 0
REGEX = \/handshake\/reply\/(?P<deploy_client_name>[^\/]+)\/(?P<deploy_client>\S+)
SOURCE_KEY = channel
0 Karma

landen99
Motivator

The channel field is extracted by default against the splunkd sourcetype. I didn't do anything to extract "channel". You can see the events in your own deployments by searching:

index=_internal sourcetype=splunkd component=PubSubSvr reply handshake PubSubSvr | dedup channel | table channel deploy*
0 Karma

somesoni2
Revered Legend

This transform is doing index-time activity or search time?

0 Karma

landen99
Motivator

search time

0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

SignalFlow: What? Why? How?

What is SignalFlow? Splunk Observability Cloud’s analytics engine, SignalFlow, opens up a world of in-depth ...

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...