Getting Data In

Possible conflict between the "props.conf" documentation and the "Route and filter data" documentation?

masonmorales
Influencer

http://docs.splunk.com/Documentation/Splunk/6.2.3/Admin/Propsconf

TRANSFORMS-<class> = <transform_stanza_name>, <transform_stanza_name2>,...
...
* Use a comma-separated list to apply multiple transform stanzas to a single TRANSFORMS
  extraction. Splunk applies them in the list order. For example, this sequence ensures that
  the [yellow] transform stanza gets applied first, then [blue], and then [red]:
        [source::color_logs]
        TRANSFORMS-colorchange = yellow, blue, red

I thinks this makes sense. The order is clearly going to be yellow first, then blue, then red.

However, the opposite seems to be inferred in an example within "Route and filter data":

http://docs.splunk.com/Documentation/Splunk/6.2.3/Forwarding/Routeandfilterdatad#Keep_specific_event...

Keep specific events and discard the rest

Here's the opposite scenario. In this example, you use two transforms to keep only the sshd events. One transform routes sshd events to indexQueue, while another routes all other events to nullQueue.

Note: In this example, the order of the transforms in props.conf matters. The null queue transform must come first; if it comes later, it will invalidate the previous transform and route all events to the null queue.

1. In props.conf:

[source::/var/log/messages]
TRANSFORMS-set= setnull,setparsing
2. In transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = \[sshd\]
DEST_KEY = queue
FORMAT = indexQueue

This is stating that the "null queue transform must come first", but according to the props.conf documentation, that means it will be applied first. Since the "setnull" stanza has a REGEX = . (anything/wildcard), then wouldn't everything get routed to the null queue BEFORE the "setparsing" stanza captured the [sshd] events and sent them to the indexQueue?

1 Solution

dshpritz
SplunkTrust
SplunkTrust

They aren't conflicting, but the second one could be a little clearer.

The transforms are setting the queue, and the last one to run will win. That is, the setnull transform will match all events, and set them to the nullQueue. The second (setparsing) will then run, and set the queue to indexQueue. While the data is still raw (not cooked) you can still manipulate the queue, as shown in the docs.

If the setnull were to run second, then all of the events would be sent to the nullQueue, which is not desired in this example.

So, yes, they are run in the order of the list.

View solution in original post

dshpritz
SplunkTrust
SplunkTrust

They aren't conflicting, but the second one could be a little clearer.

The transforms are setting the queue, and the last one to run will win. That is, the setnull transform will match all events, and set them to the nullQueue. The second (setparsing) will then run, and set the queue to indexQueue. While the data is still raw (not cooked) you can still manipulate the queue, as shown in the docs.

If the setnull were to run second, then all of the events would be sent to the nullQueue, which is not desired in this example.

So, yes, they are run in the order of the list.

masonmorales
Influencer

That makes sense, thank you for the explanation. Hopefully we can get the docs improved. 🙂

0 Karma

Richfez
SplunkTrust
SplunkTrust

Yes, the docs could be more clear but your explanation helps.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...