I am trying to route data from a heavy forwarder to a specific index. I would prefer the rule be run on the indexers for a more scale-able solution. I using the sourcetype to route the data but this is not working. Any ideas are much appreciated.
[cisco:asa] TRANSFORMS-force-sourcetype_for_cisco_devices = route_to_firewall_index
[route_to_firewall_index] SOURCE_KEY = MetaData:Sourcetype REGEX = (%ASA) DEST_KEY = _MetaData:Index FORMAT = net_firewall
you got it almost, try something like this:
[cisco:asa] TRANSFORMS-routing_for_cisco_sourcetype = route_to_cisco_asa_index
[route_to_cisco_asa_index] REGEX = . DEST_KEY = _MetaData:Index FORMAT = <your new index name>
You don't need to regex on the sourcetype, because you assigned this special transforms already to the sourcetype.
hope this helps ...
I did some testing and this did not work. From the comment above it seems I cannot route cooked data.
This should be done on the heavy forwarder and
it seems I cannot route cooked data is not true. You cannot do this by default, but if you change your
splunktcp stanza in
inputs.conf on the indexer to something like this:
it will re-parse cooked data. By doing this, you somehow make the heavy forwarder obsolete 😉
So, either but it on the heavy ( props/transforms) or use as mentioned the light weight forwarder instead.
PLEASE do not redirect already parsed events back into the parsing queue. This ability will be removed in the future and you may be left stuck, and there is no way that we can guarantee it will work as we continue to fix bugs and handle the expected usecases.
This should be considered similar to reconfiguring the pipelines in splunk which technically is still possible but is totally unsupportable.
As I wrote it makes no sense to do something like that, even when it is possible. Thanks for your input, I'll keep that in mind!
How is your ASA sending to your heavy? If it's coming in over the wire, it may be assigned an initial sourcetype of udp:514 (or whatever is tied to the network inputs.conf defnition). If it's a transform that's invoked on [udp:514] to change the sourcetype to cisco:asa, then you've already missed your change to (also) set the target index. You get one pass through TRANSFORMS at index time, keyed from the initial sourcetype of the data.
More information about your setup is required to be able to tell precisely what's going on.
I have the Splunk ASA TA installed on the HF so the data from the ASA is correctly assigning the cisco:asa sourcetype to the events. So I cannot route cooked data once it hits the indexer? The reason why I want to move the routing rule is that the queue is blocking on the rule so I would prefer to move the rule sot the indexing tier where we have more indexers to spread the load instead of just one queue in the HF. Am I out of luck?
One thing: "queue blocking on the rule" you could simply make the heavy a light forwarder so that it's not parsing, and simply acting as a passthrough. But I'm guessing that you have a heavy for other reasons, so let's proceed.
yes indeed - but I can make some adjustments if I do not have any other options.