How to edit my props and transforms to route data to an index based on host?

[jtroanlw]

I've searched Splunk Answers and Googledom with no luck, leaving me to possibly repeat the same question as others may have already done so...

Background --
We're setting up a brand new distributed environment to replace our current single-instance server. We'll eventually have two indexers (each with two indexes) and a layer of intermediate forwarders to accept data from the various equipment. Routing to an indexer (and eventually which index within) will be based solely on the "host" field value, which could be a hostname, a FQDN, or an IP address. (All of this is for requirements I've been handed, so I can't change this part of the architecture.)

Technical Setup and Question --
In my current test environment, I've got one indexer and an intermediate forwarder receiving from a test client. The intent is for anything from "splunk-if" or "splunk-if.domain.com" is to be routed into the index test2.

props.conf currently is:

[host::splunk-if*]
TRANSFORMS-splunk-if = TR_splunk-if

transforms.conf currently is:

[TR_splunk-if]
SOURCE_KEY = MetaData:Host
DEST_KEY = _MetaData:Index
REGEX = (.*)
FORMAT = test2

I've tried several different ways to set these up, based on what I've found in the Splunk documentation and Answers forum (with the latest formats shown above). Unfortunately, nothing has worked and everything for splunk-if is heading into the main index (which fails my configuration testing).

So my (obvious) questions are -- What am I doing wrong? What's the correct syntax? Will the same syntax work when I start setting up my intermediate forwarders' outputs/props/transforms .conf files?

Many Thanks!

[woodcock · Accepted solution]

You have already discriminated based on the host in your props.conf stanza so you don't need to do it again in transforms.conf; try this:

[TR_splunk-if]
DEST_KEY = _MetaData:Index
REGEX = .
FORMAT = test2

[jtroanlw]

Unfortunately, that didn't work. I changed the transforms.conf stanza as suggested and restarted splunk, but the newly-generated events still routed into main instead of test2.

[woodcock]

Usually when this happens (valid configuration does not work), it is because the stanza header is bad. In this case, the stanza header is host-based so I am guessing that elsewhere in your configurations you are overriding your host field. In this case, you have to use the original host value, not the overridden one. Do you override the host?

[jtroanlw]

Not that I know of, but I could be and not know it. When I look at the data that's sitting in the index, I can see that that the host value is "splunk-if".

(I'm very new to Splunk and still learning, especially the system side while I'm designing and building the new architecture.)

[woodcock]

OK, if you have not deliberately overridden it, then this is not the problem. Where have you put these files? They need to go to your Heavy Forwarder or all of your Indexers and then you need to restart all of the Splunk instances there. Have you done that?

[jtroanlw]

Based on the files being used to direct which index the data goes into, I put the files are on my indexer and that Splunk instance has been manually restarted after each change.

I took your comment to mean that I also need to restart the heavy forwarder instance after each props/transforms change on the Indexer. That's just been done, but it didn't help -- I restarted the sendmail service to generate events after the HF restart and those events went to main (instead of test2).

[woodcock]

If you have a Heavy forwarder, then the indexing is done there and the files must be there and the restarts must happen there. Once the HF does the indexing, it sends the finished data to the Indexer which just stores it. If it is a HF, you need those files THERE and not on your Indexers.

[jtroanlw]

Not seeing that mentioned anywhere before, I didn't know that critical part. (Does this mean that I should plan for more CPU/RAM for the HFs?)

I've copied the props.conf and transforms.conf content onto the HF and renamed both files on the indexer, then restarted both Splunk instances. Unfortunately, the events I generated after the restarts still ended up in index main.

[woodcock]

Do note that a *Heavy* Forwarder is different than a Universal Forwarder. I don't know which you have but they are very different. When you use a HF, the majority of the Indexing work is done there, instead of on the Indexers. Are you sure that you are using HFs?

[jtroanlw]

That's one part I am sure of -- I used the full splunk installer .rpm (vs the UF .rpm) to build the HF layer.

It sounds like maybe I should use the UF instead for my intermediate forwarders? Their demands are simply "accept direct feeds from other devices (w/ and w/o UFs on them) and relay to indexer A if in a given list of hosts/IPs, else relay to indexer B", then let the indexers do the intensive work since they're the ones with horsepower and RAM. From above, I would also move the props and transforms .conf's back to the indexers in this scenario? Thanks.

[woodcock]

You can architect your desire several ways, but we need to be clear about how you did it so that the correct answer can be given. If you need a HV, then you should use one, but you don't get a HV just by installing full Splunk. The simplest way is to install the HV binary as indicated here:

http://docs.splunk.com/Documentation/Splunk/6.2.4/Forwarding/Typesofforwarders

For now, the safest thing to do is to put the 2 files on your Indexers and on your Forwarders, until you figure out what you have exactly.

[jtroanlw]

After thinking through the comments and conversing with one of our other admins, I decided to change the intermediate forwarder to a UF install. (We will always route data based on the host field, so the HF would've been overkill. Plus, I had some concerns about the parsing/indexing load on HFs that were built to be only very lightweight.)

After adding a stanza in props.conf to look for splunk-if by IP and pointing it to the splunk-if stanza in transforms.conf (all on the indexer), I now have that data set routing into the desired alternate index.

I still have a few other things on the To Do list before building the production servers, but this has helped me tremendously. Thanks for your help in getting me through this stage.

[somesoni2]

This should get you started.

http://docs.splunk.com/Documentation/Splunk/6.2.4/Forwarding/Routeandfilterdatad

[jtroanlw]

Thanks. This will help with tackling the last of my questions, but it doesn't cover the route-to-index problem that's currently at the top of my list of road blocks.