I am trying to route certain events to a specific index based on a field value. All events are sent to a heavy forwarder, which then forwards traffic to my indexers. The system that generates the events I want to reroute required a custom data input, so on the heavy forwarder I have a data input defined on a specific UDP port. The data input places all events from this system into a specific index,
indexA for simplicity. Some events from this system contain a certain field value, we'll call it
reRouteMe, in a certain field,
fieldA. For these events, I want to move them to a different index,
So basically, for all events from this specific source - if
fieldA = reRouteMe I want this to go to
indexB, otherwise, the event should go to its normal index
indexA defined in the data input. This is the current configuration on the HFWD, which is not working for some reason:
[sourcetypeA] TRANSFORMS-reroute = reRouteMe
[reRouteMe] SOURCE_KEY = _raw REGEX = (reRouteMe) DEST_KEY = _MetaData:Index FORMAT = indexB
I've also tried applying this on my indexers to no avail, and I've tried REGEX variances such as
fieldA=reRouteMe, etc. I'm leaning toward the issue being the custom data input that's defined on the forwarder inferring with the props / transforms, but I would think the props / transforms on the indexers would catch it.
If I were you, I would just do it after the fact and copy the special events into a summary index using the
collect command; this will be FAR easier (it moves from exceedingly difficult, perhaps even impossible, to absolutely trivial):
We receive hundreds of events a day that have to be moved to this other index in a streaming fashion. If it was just a one-off case I'd agree this is a good solution.
Hundreds/day is nothing. I would still use the
collect command run every hour.
[field]=[value] | collect index=[new] did not result in the events moving? They are still in the original index.
Actually, I missed that
collect moves to a summary index. We aren't permitted to have these events in this index, so I need to move these events to the new index and ensure they aren't in the original.
collect command copies them, you still need to use
delete to remove (hide) them from the original index. As far as going to a
summary index, that only means that you you will not be charged twice for the volume against your license, which is a very good thing.
I also missed that it doesn't keep the sourcetype, so they are all in the new index with a
host field as the search head and the
stash. So I should be able to do
[field]=[value] | collect index=[new] sourcetype=[sourcetype] host=[host[ to maintain the metadata correct?
This resolved my issue well enough for now until I can figure out why the routing stopped working. Thanks for the help!
What is the stanza for your inputs.conf?
Can you run a btool on the sourcetype? Make sure the reRouteMe applies to the sourcetype.
$SPLUNK_HOME/bin/splunk btool list props sourcetypeA --debug Will show (with file paths) the configurations applied to the sourcetype. So if the
inputs.conf sourcetype matches the
props.conf sourcetype, and btool reports it correctly, this should work. I verified the config in my test environment, so I know what you have listed will work.