Getting Data In

How to edit my props and transforms to route data to an index based on host?

jtroanlw
Explorer

I've searched Splunk Answers and Googledom with no luck, leaving me to possibly repeat the same question as others may have already done so...

Background --
We're setting up a brand new distributed environment to replace our current single-instance server. We'll eventually have two indexers (each with two indexes) and a layer of intermediate forwarders to accept data from the various equipment. Routing to an indexer (and eventually which index within) will be based solely on the "host" field value, which could be a hostname, a FQDN, or an IP address. (All of this is for requirements I've been handed, so I can't change this part of the architecture.)

Technical Setup and Question --
In my current test environment, I've got one indexer and an intermediate forwarder receiving from a test client. The intent is for anything from "splunk-if" or "splunk-if.domain.com" is to be routed into the index test2.

props.conf currently is:

[host::splunk-if*]
TRANSFORMS-splunk-if = TR_splunk-if

transforms.conf currently is:

[TR_splunk-if]
SOURCE_KEY = MetaData:Host
DEST_KEY = _MetaData:Index
REGEX = (.*)
FORMAT = test2

I've tried several different ways to set these up, based on what I've found in the Splunk documentation and Answers forum (with the latest formats shown above). Unfortunately, nothing has worked and everything for splunk-if is heading into the main index (which fails my configuration testing).

So my (obvious) questions are -- What am I doing wrong? What's the correct syntax? Will the same syntax work when I start setting up my intermediate forwarders' outputs/props/transforms .conf files?

Many Thanks!

1 Solution

woodcock
Esteemed Legend

You have already discriminated based on the host in your props.conf stanza so you don't need to do it again in transforms.conf; try this:

[TR_splunk-if]
DEST_KEY = _MetaData:Index
REGEX = .
FORMAT = test2

View solution in original post

woodcock
Esteemed Legend

You have already discriminated based on the host in your props.conf stanza so you don't need to do it again in transforms.conf; try this:

[TR_splunk-if]
DEST_KEY = _MetaData:Index
REGEX = .
FORMAT = test2

jtroanlw
Explorer

Unfortunately, that didn't work. I changed the transforms.conf stanza as suggested and restarted splunk, but the newly-generated events still routed into main instead of test2.

0 Karma

woodcock
Esteemed Legend

Usually when this happens (valid configuration does not work), it is because the stanza header is bad. In this case, the stanza header is host-based so I am guessing that elsewhere in your configurations you are overriding your host field. In this case, you have to use the original host value, not the overridden one. Do you override the host?

jtroanlw
Explorer

Not that I know of, but I could be and not know it. When I look at the data that's sitting in the index, I can see that that the host value is "splunk-if".

(I'm very new to Splunk and still learning, especially the system side while I'm designing and building the new architecture.)

0 Karma

woodcock
Esteemed Legend

OK, if you have not deliberately overridden it, then this is not the problem. Where have you put these files? They need to go to your Heavy Forwarder or all of your Indexers and then you need to restart all of the Splunk instances there. Have you done that?

0 Karma

jtroanlw
Explorer

Based on the files being used to direct which index the data goes into, I put the files are on my indexer and that Splunk instance has been manually restarted after each change.

I took your comment to mean that I also need to restart the heavy forwarder instance after each props/transforms change on the Indexer. That's just been done, but it didn't help -- I restarted the sendmail service to generate events after the HF restart and those events went to main (instead of test2).

0 Karma

woodcock
Esteemed Legend

If you have a Heavy forwarder, then the indexing is done there and the files must be there and the restarts must happen there. Once the HF does the indexing, it sends the finished data to the Indexer which just stores it. If it is a HF, you need those files THERE and not on your Indexers.

0 Karma

jtroanlw
Explorer

Not seeing that mentioned anywhere before, I didn't know that critical part. (Does this mean that I should plan for more CPU/RAM for the HFs?)

I've copied the props.conf and transforms.conf content onto the HF and renamed both files on the indexer, then restarted both Splunk instances. Unfortunately, the events I generated after the restarts still ended up in index main.

0 Karma

woodcock
Esteemed Legend

Do note that a *Heavy* Forwarder is different than a Universal Forwarder. I don't know which you have but they are very different. When you use a HF, the majority of the Indexing work is done there, instead of on the Indexers. Are you sure that you are using HFs?

0 Karma

jtroanlw
Explorer

That's one part I am sure of -- I used the full splunk installer .rpm (vs the UF .rpm) to build the HF layer.

It sounds like maybe I should use the UF instead for my intermediate forwarders? Their demands are simply "accept direct feeds from other devices (w/ and w/o UFs on them) and relay to indexer A if in a given list of hosts/IPs, else relay to indexer B", then let the indexers do the intensive work since they're the ones with horsepower and RAM. From above, I would also move the props and transforms .conf's back to the indexers in this scenario? Thanks.

0 Karma

woodcock
Esteemed Legend

You can architect your desire several ways, but we need to be clear about how you did it so that the correct answer can be given. If you need a HV, then you should use one, but you don't get a HV just by installing full Splunk. The simplest way is to install the HV binary as indicated here:

http://docs.splunk.com/Documentation/Splunk/6.2.4/Forwarding/Typesofforwarders

For now, the safest thing to do is to put the 2 files on your Indexers and on your Forwarders, until you figure out what you have exactly.

0 Karma

jtroanlw
Explorer

After thinking through the comments and conversing with one of our other admins, I decided to change the intermediate forwarder to a UF install. (We will always route data based on the host field, so the HF would've been overkill. Plus, I had some concerns about the parsing/indexing load on HFs that were built to be only very lightweight.)

After adding a stanza in props.conf to look for splunk-if by IP and pointing it to the splunk-if stanza in transforms.conf (all on the indexer), I now have that data set routing into the desired alternate index.

I still have a few other things on the To Do list before building the production servers, but this has helped me tremendously. Thanks for your help in getting me through this stage.

0 Karma

somesoni2
Revered Legend
0 Karma

jtroanlw
Explorer

Thanks. This will help with tackling the last of my questions, but it doesn't cover the route-to-index problem that's currently at the top of my list of road blocks.

0 Karma
Get Updates on the Splunk Community!

Splunk Classroom Chronicles: Training Tales and Testimonials

Welcome to the "Splunk Classroom Chronicles" series, created to help curious, career-minded learners get ...

Access Tokens Page - New & Improved

Splunk Observability Cloud recently launched an improved design for the access tokens page for better ...

Stay Connected: Your Guide to November Tech Talks, Office Hours, and Webinars!

🍂 Fall into November with a fresh lineup of Community Office Hours, Tech Talks, and Webinars we’ve ...