topic Re: Can't route forwarded data to different index? in Getting Data In

Can't route forwarded data to different index?

gljiva — Thu, 29 Apr 2010 04:52:16 GMT

Hi, i saw many suggestions to routing data to different index from light forwarder but none seems to work. I have set up light forwarder that sends cooked data to my indexer. Connection is ok, because data gets indexed but in wrong index.

Here are some of configurations i have tried (on indexer side):

inputs.conf 
[splunktcp://9997]
index = test_2

Next one:
inputs.conf 
[splunktcp://9997]

props.conf
[host::Simpson-test]
index = test_2
TRANSFORMS-foo = routeIndex

transforms.conf
[routeIndex]
SOURCE_KEY = source
REGEX = = WinEventLog
DEST_KEY =_MetaData:Index
FORMAT = test_2

Next one:
inputs.conf 
[splunktcp://9997]
sourcetype=REMOTE

props.conf
[REMOTE]
TRANSFORMS-simremote = sremote

transforms.conf
[sremote]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = test_2

Data allways ends up in index that was defined on lightforwarder (test_1). I would like to route data that comes to indexer based on sourcetype and host. Data comes from lightforwarder and is cooked so i suppose that sourcetype and host allready exist and i can do REGEX search on SOURCE_KEY?

Any hints abouth troubleshooting this?

Re: Can't route forwarded data to different index?

dwaddle — Thu, 29 Apr 2010 06:27:18 GMT

The light forwarder can send to a specific index if you set the destination index in inputs.conf on the forwarder itself. This can be overridden at the indexer using props.conf/transforms.conf for a specific host or sourcetype.

props.conf:

[myfavoritesourcetype]
    TRANSFORMS-index=sendtomyindex

transforms.conf:

[sendtomyindex]
SOURCE_KEY=MetaData:Sourcetype
REGEX=(.*)
FORMAT=index::myindex
WRITE_META=true

Update: it looks like this works just as well. I'm not sure which is preferable...

[sendtomyindex]
SOURCE_KEY=_MetaData:Index
DEST_KEY=_MetaData:Index
REGEX=(.*)
FORMAT=myindex

Re: Can't route forwarded data to different index?

gljiva — Mon, 28 Sep 2020 09:12:10 GMT

Thanks alot, got it to work!
First one doesn't exactly work. It really adds new _index field to data, but doesn't delete old one. So I end up having two _index fields, but only first _index field is used to store data, and second one seems to get ignored.

Second one works great if i write it like this:
[routeIndex]
SOURCE_KEY = MetaData:Sourcetype
REGEX=(.*)
DEST_KEY=_MetaData:Index
FORMAT=test_2
WRITE_META=true

Re: Can't route forwarded data to different index?

dwaddle — Thu, 29 Apr 2010 18:26:30 GMT

Interesting, I tried both of these (and some variations that didn't work) and only saw one "index" field in each case. But then again, I'm struggling to see "_index" in the UI. How are you seeing that?

Re: Can't route forwarded data to different index?

Lowell — Thu, 29 Apr 2010 20:52:34 GMT

Side note on your regex: You should be able to use REGEX=.?, which should be more efficient way to say match-all.

Re: Can't route forwarded data to different index?

Jason — Thu, 13 May 2010 06:05:01 GMT

The easiest way is to specify the index in inputs.conf on the forwarder.

It seemed counter-intuitive to me at first, as if I was directing the forwarder to index something locally. But data is sent along to the indexer and placed into the desired index (as defined in an indexes.conf on the indexer).

On the forwarder, inputs.conf:

[monitor:///filename/or/other/input/type/]
disabled = false
index=ghtest5