In our environment all data from syslog sources and UFs come to our HFs before they get forwarded to indexers. Now we have got the following requirement:
Linux systems will be sending linsec data and capacity planning data to our HFs. Linsec data is configured to go to index A and capacity planning data is configured to go to index B. All the data comes to our HFs cooked. It is not the raw data. Now we want to forward linsec data to one set of indexers and capacity planning data to other set of indexers.
Is there a way we can do it in Splunk? How to filter the cooked data on HFs which is specific to index and then route it to separate set of indexers? Can I use props and transforms to filter cooked data? I tried props and transforms and seems it did not work.
Yes, you can set the data to go to different sets of indexers, but be sure to get it right. You have to use an outputs.conf file that will define both seta and the indexers they will be sent to. Then you have to set up the inputs.conf to send the data to the right indexer set. Here is an example outputs.conf file:
The _TCP_ROUTING is the key to sending it to the other indexers. You may have to play around with the names, etc., but the configuration is solid. I've been using the same type stuff for a couple of years and it works.
You may wish to only send that to machines that need to send to the other indexers. It will try to connect on any machine that it is set up on, though it won't send to that indexer set unless you specify it in the inputs.conf file.
In one case where I have used this method the data on the source UF has this configured to send to a different HF, not just an indexer. So this still works as long as the HF's are sending their data to different indexers. You probably could set up the HF to do this instead, but in my case it worked better to send certain parts of the data from the UF to a different place (non-default) than the indexers for filtering (on the HF).
So the server with the HF gets the inputs.conf as I showed, but instead of using the indexers in the server, use the server config to an HF, which forwards its received data to a set of indexers.