Getting Data In
Highlighted

How to use sourcetype to route data from a heavy forwarder to a specific index?

Communicator

I am trying to route data from a heavy forwarder to a specific index. I would prefer the rule be run on the indexers for a more scale-able solution. I using the sourcetype to route the data but this is not working. Any ideas are much appreciated.

props.conf

[cisco:asa]
TRANSFORMS-force-sourcetype_for_cisco_devices = route_to_firewall_index

transforms.conf

[route_to_firewall_index]
SOURCE_KEY = MetaData:Sourcetype
REGEX = (%ASA)
DEST_KEY = _MetaData:Index
FORMAT = net_firewall

Thanks!

Highlighted

Re: How to use sourcetype to route data from a heavy forwarder to a specific index?

SplunkTrust
SplunkTrust

Hi ebailey,

you got it almost, try something like this:

props.conf

 [cisco:asa]
 TRANSFORMS-routing_for_cisco_sourcetype = route_to_cisco_asa_index

transforms.conf

[route_to_cisco_asa_index]
 REGEX = .
 DEST_KEY = _MetaData:Index
 FORMAT = <your new index name>

You don't need to regex on the sourcetype, because you assigned this special transforms already to the sourcetype.

hope this helps ...

cheers, MuS

Highlighted

Re: How to use sourcetype to route data from a heavy forwarder to a specific index?

Communicator

I did some testing and this did not work. From the comment above it seems I cannot route cooked data.

0 Karma
Highlighted

Re: How to use sourcetype to route data from a heavy forwarder to a specific index?

SplunkTrust
SplunkTrust

This should be done on the heavy forwarder and it seems I cannot route cooked data is not true. You cannot do this by default, but if you change your splunktcp stanza in inputs.conf on the indexer to something like this:

[splunktcp] route=has_key:_utf8:parsingQueue;has_key:_linebreaker:parsingQueue;absent_key:_utf8:parsingQueue;absent_key:_linebreaker:parsingQueue

it will re-parse cooked data. By doing this, you somehow make the heavy forwarder obsolete 😉
So, either but it on the heavy ( props/transforms) or use as mentioned the light weight forwarder instead.

0 Karma
Highlighted

Re: How to use sourcetype to route data from a heavy forwarder to a specific index?

Splunk Employee
Splunk Employee

PLEASE do not redirect already parsed events back into the parsing queue. This ability will be removed in the future and you may be left stuck, and there is no way that we can guarantee it will work as we continue to fix bugs and handle the expected usecases.
This should be considered similar to reconfiguring the pipelines in splunk which technically is still possible but is totally unsupportable.

Highlighted

Re: How to use sourcetype to route data from a heavy forwarder to a specific index?

SplunkTrust
SplunkTrust

As I wrote it makes no sense to do something like that, even when it is possible. Thanks for your input, I'll keep that in mind!

0 Karma
Highlighted

Re: How to use sourcetype to route data from a heavy forwarder to a specific index?

Splunk Employee
Splunk Employee

How is your ASA sending to your heavy? If it's coming in over the wire, it may be assigned an initial sourcetype of udp:514 (or whatever is tied to the network inputs.conf defnition). If it's a transform that's invoked on [udp:514] to change the sourcetype to cisco:asa, then you've already missed your change to (also) set the target index. You get one pass through TRANSFORMS at index time, keyed from the initial sourcetype of the data.

More information about your setup is required to be able to tell precisely what's going on.

0 Karma
Highlighted

Re: How to use sourcetype to route data from a heavy forwarder to a specific index?

Communicator

I have the Splunk ASA TA installed on the HF so the data from the ASA is correctly assigning the cisco:asa sourcetype to the events. So I cannot route cooked data once it hits the indexer? The reason why I want to move the routing rule is that the queue is blocking on the rule so I would prefer to move the rule sot the indexing tier where we have more indexers to spread the load instead of just one queue in the HF. Am I out of luck?

0 Karma
Highlighted

Re: How to use sourcetype to route data from a heavy forwarder to a specific index?

Splunk Employee
Splunk Employee

One thing: "queue blocking on the rule" you could simply make the heavy a light forwarder so that it's not parsing, and simply acting as a passthrough. But I'm guessing that you have a heavy for other reasons, so let's proceed.

0 Karma
Highlighted

Re: How to use sourcetype to route data from a heavy forwarder to a specific index?

Communicator

yes indeed - but I can make some adjustments if I do not have any other options.

0 Karma