Getting Data In

How to index all locally and forward specific sourcetype to a 3rd party indexer?

Communicator

I have the following scenario to achieve:

1.I have a cluster of indexers receiving misc. events
2. By default, all events are indexed locally
3. For events matching specific condition (for instance, a specific sourcetype) I want to forward them to a 3rd party indexer without indexing them locally.

In the documentation, I found several examples closed to what I need:
http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad

So I configured the following:

outputs.conf

[tcpout]
defaultGroup=nothing

[tcpout:externalIDX]
server = x.x.x.x:9997

props.conf

[mysourcetype]
TRANSFORMS-routing = ForwardToEXT

transforms.conf

[ForwardToEXT]
REGEX = .
DEST_KEY = _TCP_ROUTING
FORMAT = externalIDX

Results
When I do apply configuration, my external system does receive the events but my local indexer no longer index anything.

Alternatively, I tried to remove defaultGroup=nothing from outputs.conf. This time, everything (any event) is duplicated to the external indexer.

Anybody, to help me to define the right syntax?
Regards.

1 Solution

Champion

As mentioned in a comment above, it's easy to selectively forward, and it's easy to selectively index.

What's more difficult is when you want to selectively not index.

If you set queue to nullQueue to try to prevent local indexing, they also won't be forwarded.

You can't unset a DESTKEY (though you can set it to be empty, but the mere presence of `INDEXANDFORWARDROUTING` will result in indexing locally). This means you can't set _INDEXANDFORWARDROUTING for everything (either in inputs or a default props stanza) and then clear it later. But, you can override (in this case clear) the value of the transform class that would set that value. This is how I was able to get this working as I believe you want.

outputs.conf:

[indexAndForward]
index = true
selectiveIndexing = true

[tcpout]
defaultGroup = nothing

[tcpout:externalIDX]
server = <forwarder>:9997

props.conf:

[default]
# unless a more specific stanza clears the value of this class, the transform will be run
TRANSFORMS-selectiveIndexing = selectiveIndexing

[<sourcetype to forward>]
TRANSFORMS-routing = ForwardToEXT
# note the empty list of transforms to run in this class, overridden from the [default]
TRANSFORMS-selectiveIndexing =

transforms.conf:

[ForwardToEXT]
REGEX = . 
DEST_KEY = _TCP_ROUTING
FORMAT = externalIDX

[selectiveIndexing]
REGEX = . 
DEST_KEY = _INDEX_AND_FORWARD_ROUTING
# FORMAT doesn't really matter here, the key _INDEX_AND_FORWARD_ROUTING just has to be set
FORMAT = local

View solution in original post

Champion

As mentioned in a comment above, it's easy to selectively forward, and it's easy to selectively index.

What's more difficult is when you want to selectively not index.

If you set queue to nullQueue to try to prevent local indexing, they also won't be forwarded.

You can't unset a DESTKEY (though you can set it to be empty, but the mere presence of `INDEXANDFORWARDROUTING` will result in indexing locally). This means you can't set _INDEXANDFORWARDROUTING for everything (either in inputs or a default props stanza) and then clear it later. But, you can override (in this case clear) the value of the transform class that would set that value. This is how I was able to get this working as I believe you want.

outputs.conf:

[indexAndForward]
index = true
selectiveIndexing = true

[tcpout]
defaultGroup = nothing

[tcpout:externalIDX]
server = <forwarder>:9997

props.conf:

[default]
# unless a more specific stanza clears the value of this class, the transform will be run
TRANSFORMS-selectiveIndexing = selectiveIndexing

[<sourcetype to forward>]
TRANSFORMS-routing = ForwardToEXT
# note the empty list of transforms to run in this class, overridden from the [default]
TRANSFORMS-selectiveIndexing =

transforms.conf:

[ForwardToEXT]
REGEX = . 
DEST_KEY = _TCP_ROUTING
FORMAT = externalIDX

[selectiveIndexing]
REGEX = . 
DEST_KEY = _INDEX_AND_FORWARD_ROUTING
# FORMAT doesn't really matter here, the key _INDEX_AND_FORWARD_ROUTING just has to be set
FORMAT = local

View solution in original post

Communicator

It works, thanks a lot again !
I just added the following to get ride of a warning at startup:
[acceptedkeys]
is
valid=INDEXANDFORWARDROUTING

0 Karma

Communicator

thanks again for your investigation. I confirm I already tried by myself the idea to set queue to nullQueue with another transform in parallel, it does work.
I will test your other idea in parallel and let you know.
thanks a lot !

0 Karma

Champion

Edit: ignore this answer. I posted a new answer with a different solution due to the requirement not to index events that were forwarded.

Don't forget you need to configre indexAndForward in outputs.conf.

[indexAndForward]
index=true

Communicator

It looks like an alternative method to the one I am trying to use. I saw what you suggest but it looks the offered granularity is only per-input which does not match my need.
In my case (because working at indexer level), I have a single input receiving all data and I need to re-forward some data based on other critieria (such as sourcetype, or more complex regex condition).
Let me know if I missed something. Thanks.

0 Karma

Champion

The configuration I included is required in order to enable local indexing for events that are also forwarded. This would be used in addition to the configuration you included in your original question.

Note that this is a component of what you were referring to (selective indexing), and does not involve setting selectiveIndexing=true.

From the doc:

Index all the data before forwarding it. To do this, just enable the indexAndForward attribute in outputs.conf.

By doing this you are essentially stating that you want every event forwarded. But, your original configuration sets the default tcpout group to one that doesn't exist, so only after you change _TCP_ROUTING to externalIDX will the event actually be forwarded.

0 Karma

Communicator

So I found time to make a new test... Basically same than my above config + adding what you suggested.

It is almost perfect
1. I can get the specified sourcetype to be forwarded
2. Other sourcetypes are untouched (still locally indexed as before and not forwarded)

But (that was part of my requirement, the specified sourcetype is still locally indexed. I would like to have forwarded only

0 Karma

Champion

I'm looking into this. The opposite (forward everything, but only index these few events) is very easy, but index everything and only forward these events is proving trickier.

Did I just miss the "and don't index if forwarded" part of the question at first, or did you edit your question to add that? I'm only wondering if my reading comprehension skills need work. 🙂

0 Karma

Communicator

Yes you missed it, it was there from the beginning 🙂 but anyway thanks for your suggestion so far, I learnt a lot.

0 Karma

Communicator

Thanks a lot for clarification. I will test that.

0 Karma

SplunkTrust
SplunkTrust

Syntax looks good, can you please try to check your config using btool command for every conf file (Specially props and transforms)

Btool command for props is $SPLUNK_HOME/bin/splunk cmd btool props --debug list mysourcetype , you can modify same command to check configuration for transforms.conf or any other configuration file.

0 Karma

Communicator

I will check and come back

0 Karma