How do I route pre-processed data to a specified index, based on field value?

[atari1050]

I have app data routing from one set of Relay Forwarders (DEV) into another set of Relay Forwarders (sandbox) and then on to a set of indexers. I need to route the data to a specific index if the field COMPONENT has a specific value (web or cam). All other values will continue to route the data to the existing index (dev).
I have looked at all Splunk Answers posts I could find. I do not have enough Karma points to post them here. What did I miss?

props.conf

[CAMrouting]
TRANSFORMS-routing_to_cam_index = fieldCAM_routing

[WEBrouting]
TRANSFORMS-routing_to_web_index = fieldWEB_routing
transforms.conf

[fieldWEB_routing]
REGEX = WEB
FORMAT = web
DEST_KEY = _MetaData:Index

[fieldCAM_routing]
REGEX = CAM
FORMAT = cam
DEST_KEY = _MetaData:Index

[acharlieh]

I think I can see a few potential issues here...

First, it sounds like you're routing based on a field value, but your props.conf has stanzas for two different sourcetypes. If your inputs.conf is actually reading two distinct sources, then assign index at input time separately as well and you're done. (I would note that Xrouting are terrible sourcetype names, you should think about following Splunk's naming conventions for sourcetypes. Even if same sourcetype but two different inputs can be sent to different indexes on the forwarders. Much easier than applying a regex to each event)

If you have a single stream of events, then your props stanzas are likely incorrect as they should be named for sourcetype. Barring HEADER_MODE, HEC (if you look at it the right way), and some esoteric setups that I'll get into in a minute, A single stream of data would not have differing sourcetypes at this point in the parsing process therefore the separate props conf stanzas is likely incorrect that way as well.

Now your architecture... You describe a sequence of "Relay Forwarders." If these are Heavy Forwarders (I.e. full Splunk package as opposed to the Splunk universal forwarder package), then the first one to see your data is executing the regexReplacement processor, not your indexers and your appropriate props and transforms need to be there. (And the appropriate restarting of Splunk after landing config of course).This comes from Splunk trying to be efficient in ingestion processing allowing fully parsed data to skip parsing steps when forwarded to other Splunk instances. With some adjustments to inputs.conf on your indexers you can force re-baking of the data, but very rarely is that the correct answer and misconfiguration there can really screw things up.

I'm not going to go deep into it but intermediate heavy forwarders are also usually not the right answer, and present a number of issues.

Talking about the steps of ingestion, I usually keep this page bookmarked from the Splunk wiki: https://wiki.splunk.com/Community:HowIndexingWorks

Hopefully this gives you some ideas that may or may not be applicable to your specific scenario.

[atari1050]

Hi acharlieh -
Thanks for the response. Sorry about the terms of use, Heavy Forwarder = Relay Forwarder.
The second set of Heavy Forwarders are for completion of architecture. What I mean by that is that when we simulate load, test apps, we want to test from the entire hierarchy: HF, Indexers, SH's.
Back to our show:
For ease of use, we are splitting off data from our dev environment into our sandbox environment just so we can test against it. It is pre-processed, single sourcetype (originally udp SYSLOG, when forwarded, it changes to tcp, different port).

Can you explain the syntax of why it appears that my props.conf is written for sourcetypes? What I was trying to do there was define 2 field values, but sounds like I cannot do that.

I hope this clarifies.

[acharlieh]

So if you are going HF- HF- Indexer, the place where these would have any effect is the first HF.

In props.conf, the stanza names e.g. [CAMrouting] are the sourcetype, host or source that the attributes underneath apply to. So you have currently defined two sourcetypes each applying one transform. Spend some time with the spec file.

You will also find that you can specify multiple TRANSFORM-x lines (so long as x varies), or specify multiple transforms in one attribute as that attribute is a comma separated list.

[atari1050]

Thanks. I corrected the stanzas in props.conf, as I was wrong about the source specification as you stated.
As I understand, the changes/transforms MUST be done on the first set of Relays?

Here are the corrected entries, however, still not getting results in separate indexes. Should I change it from source to sourcetype in the props?

props.conf:
[source::udp:9514]
TRANSFORMS-camIndex = index_cam
TRANSFORMS-webIndex = index_web

transforms.conf:
[index_cam]
REGEX = .*COMPONENT=CAM.*
DEST_KEY = _MetaData:Index
FORMAT = cam

[index_web]
REGEX = .*COMPONENT=WEB.*
DEST_KEY = _MetaData:Index
FORMAT = web

[atari1050]

I have tried all types of iterations of the above props.conf and transforms.conf paying attention to what is documented in the .spec files, but to no avail. It appears that the stanzas are ignored by Splunk.
I am now going on acharlieh's statement that this should be done from the first layer of RF's and will perform the change tonight. Will update when complete.