I have the following scenario to achieve:
1.I have a cluster of indexers receiving misc. events
2. By default, all events are indexed locally
3. For events matching specific condition (for instance, a specific sourcetype) I want to forward them to a 3rd party indexer without indexing them locally.
In the documentation, I found several examples closed to what I need:
http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad
So I configured the following:
outputs.conf
[tcpout]
defaultGroup=nothing
[tcpout:externalIDX]
server = x.x.x.x:9997
props.conf
[mysourcetype]
TRANSFORMS-routing = ForwardToEXT
transforms.conf
[ForwardToEXT]
REGEX = .
DEST_KEY = _TCP_ROUTING
FORMAT = externalIDX
Results
When I do apply configuration, my external system does receive the events but my local indexer no longer index anything.
Alternatively, I tried to remove defaultGroup=nothing
from outputs.conf. This time, everything (any event) is duplicated to the external indexer.
Anybody, to help me to define the right syntax?
Regards.
As mentioned in a comment above, it's easy to selectively forward, and it's easy to selectively index.
What's more difficult is when you want to selectively not index.
If you set queue
to nullQueue
to try to prevent local indexing, they also won't be forwarded.
You can't unset
a DEST_KEY (though you can set it to be empty, but the mere presence of _INDEX_AND_FORWARD_ROUTING
will result in indexing locally). This means you can't set _INDEX_AND_FORWARD_ROUTING for everything (either in inputs or a default props stanza) and then clear it later. But, you can override (in this case clear) the value of the transform class that would set that value. This is how I was able to get this working as I believe you want.
outputs.conf:
[indexAndForward]
index = true
selectiveIndexing = true
[tcpout]
defaultGroup = nothing
[tcpout:externalIDX]
server = <forwarder>:9997
props.conf:
[default]
# unless a more specific stanza clears the value of this class, the transform will be run
TRANSFORMS-selectiveIndexing = selectiveIndexing
[<sourcetype to forward>]
TRANSFORMS-routing = ForwardToEXT
# note the empty list of transforms to run in this class, overridden from the [default]
TRANSFORMS-selectiveIndexing =
transforms.conf:
[ForwardToEXT]
REGEX = .
DEST_KEY = _TCP_ROUTING
FORMAT = externalIDX
[selectiveIndexing]
REGEX = .
DEST_KEY = _INDEX_AND_FORWARD_ROUTING
# FORMAT doesn't really matter here, the key _INDEX_AND_FORWARD_ROUTING just has to be set
FORMAT = local
As mentioned in a comment above, it's easy to selectively forward, and it's easy to selectively index.
What's more difficult is when you want to selectively not index.
If you set queue
to nullQueue
to try to prevent local indexing, they also won't be forwarded.
You can't unset
a DEST_KEY (though you can set it to be empty, but the mere presence of _INDEX_AND_FORWARD_ROUTING
will result in indexing locally). This means you can't set _INDEX_AND_FORWARD_ROUTING for everything (either in inputs or a default props stanza) and then clear it later. But, you can override (in this case clear) the value of the transform class that would set that value. This is how I was able to get this working as I believe you want.
outputs.conf:
[indexAndForward]
index = true
selectiveIndexing = true
[tcpout]
defaultGroup = nothing
[tcpout:externalIDX]
server = <forwarder>:9997
props.conf:
[default]
# unless a more specific stanza clears the value of this class, the transform will be run
TRANSFORMS-selectiveIndexing = selectiveIndexing
[<sourcetype to forward>]
TRANSFORMS-routing = ForwardToEXT
# note the empty list of transforms to run in this class, overridden from the [default]
TRANSFORMS-selectiveIndexing =
transforms.conf:
[ForwardToEXT]
REGEX = .
DEST_KEY = _TCP_ROUTING
FORMAT = externalIDX
[selectiveIndexing]
REGEX = .
DEST_KEY = _INDEX_AND_FORWARD_ROUTING
# FORMAT doesn't really matter here, the key _INDEX_AND_FORWARD_ROUTING just has to be set
FORMAT = local
It works, thanks a lot again !
I just added the following to get ride of a warning at startup:
[accepted_keys]
is_valid=_INDEX_AND_FORWARD_ROUTING
thanks again for your investigation. I confirm I already tried by myself the idea to set queue to nullQueue with another transform in parallel, it does work.
I will test your other idea in parallel and let you know.
thanks a lot !
Edit: ignore this answer. I posted a new answer with a different solution due to the requirement not to index events that were forwarded.
Don't forget you need to configre indexAndForward in outputs.conf.
[indexAndForward]
index=true
It looks like an alternative method to the one I am trying to use. I saw what you suggest but it looks the offered granularity is only per-input which does not match my need.
In my case (because working at indexer level), I have a single input receiving all data and I need to re-forward some data based on other critieria (such as sourcetype, or more complex regex condition).
Let me know if I missed something. Thanks.
The configuration I included is required in order to enable local indexing for events that are also forwarded. This would be used in addition to the configuration you included in your original question.
Note that this is a component of what you were referring to (selective indexing), and does not involve setting selectiveIndexing=true
.
From the doc:
Index all the data before forwarding it. To do this, just enable the indexAndForward attribute in outputs.conf.
By doing this you are essentially stating that you want every event forwarded. But, your original configuration sets the default tcpout group to one that doesn't exist, so only after you change _TCP_ROUTING
to externalIDX
will the event actually be forwarded.
So I found time to make a new test... Basically same than my above config + adding what you suggested.
It is almost perfect
1. I can get the specified sourcetype to be forwarded
2. Other sourcetypes are untouched (still locally indexed as before and not forwarded)
But (that was part of my requirement, the specified sourcetype is still locally indexed. I would like to have forwarded only
I'm looking into this. The opposite (forward everything, but only index these few events) is very easy, but index everything and only forward these events is proving trickier.
Did I just miss the "and don't index if forwarded" part of the question at first, or did you edit your question to add that? I'm only wondering if my reading comprehension skills need work. 🙂
Yes you missed it, it was there from the beginning 🙂 but anyway thanks for your suggestion so far, I learnt a lot.
Thanks a lot for clarification. I will test that.
Syntax looks good, can you please try to check your config using btool command for every conf file (Specially props and transforms)
Btool command for props is $SPLUNK_HOME/bin/splunk cmd btool props --debug list mysourcetype
, you can modify same command to check configuration for transforms.conf or any other configuration file.
I will check and come back