topic Re: How to selectively index and forward with filtering? in Getting Data In

How to selectively index and forward with filtering?

dottom — Wed, 15 Jun 2011 05:49:46 GMT

Is there a way to selectively index and forward by using filtering criteria such as hostname, sourcetype, or REGEX in transforms.conf? Currently, I can selectively index and forward on a per input stanza basis in inputs.conf, but I don't want to forward everything coming into an input.

If I were to only forward (and not index locally), I would use a REGEX in transforms.conf with a [stanza] in props.conf to filter what to forward. But it looks like using transforms.conf and props.conf is not supported for selective index and forwarding.

Re: How to selectively index and forward with filtering?

MuS — Wed, 15 Jun 2011 06:07:37 GMT

Hi dottom

either take a look here:
http://splunk-base.splunk.com/answers/1888/how-do-i-configure-splunk-to-filter-out-events-i-dont-want-to-index

or read the docs here:
http://www.splunk.com/base/Documentation/4.2.1/Deploy/Routeandfilterdatad

both is working as designed, but be aware about this here:
http://splunk-base.splunk.com/answers/13139/wineventlogsecurity-filtering-does-not-work

I just run into this bug last week. but as said, beside this, all is working like in the docs written.

regards

Re: How to selectively index and forward with filtering?

dottom — Wed, 15 Jun 2011 06:32:48 GMT

My scenario is different in that I don't want to filter out events from being indexed. What I want to do is filter events to be forwarded, i.e. do not forward some events (only index it), forward specific sourcetype to remoteHostA, forward specific REGEX string to remoteHostB, etc.

The scenario:

A single inputs.conf stanza receives logs from 100 different systems.
I want to index all of them (using "indexAndForward" and _INDEX_AND_FORWARD_ROUTING in each input stanza).
But I want to selectively forward some logs to some other log consumer devices (using props.conf and transforms.conf, which does not work for "indexAndForward").

I don't want to just forward using LWF/HF/UF which is very flexible to customize using props.conf and transforms.conf. This is a "index and selectively forward" approach.

As a kludge, I've considered running both a forwarder and index instance (two Splunk instances) and have the forwarder forward locally what I want indexed, and forward remotely what I want sent off to other log collection devices. But I really don't want to run two Splunk instances just to have flexible filtering capability for a "index and forward" design.

Re: How to selectively index and forward with filtering?

MuS — Wed, 15 Jun 2011 07:15:27 GMT

Hi dottom

well basicly an indexer can do the same filtering/routing of data like a forwarder. here is a post about how to configure forwarder to send different information to 2 different indexers

so your indexer can be setup to filter data to different indexes or forward any data to 3rd party systems.

regrads

Re: How to selectively index and forward with filtering?

dottom — Mon, 28 Sep 2020 09:42:12 GMT

That solution doesn't work for this environment because I cannot configure the forwarders to send to different indexers. I have a single indexer that is the central hub for many different data inputs. What I want to do is selectively forward and index from a single indexer

Right now, with Splunk, an indexer can selective forward without indexing.

I am using "indexAndForward" and _INDEX_AND_FORWARD_ROUTING in each input stanza. So at the moment, my forwarding granularity is limited to a stanza in inputs.conf. What I want to do is be able to use props.conf and transforms.conf to selective decide:

What to index only, and not forward.
What to index and forward.