Getting Data In

Can't route forwarded data to different index?

gljiva
Path Finder

Hi, i saw many suggestions to routing data to different index from light forwarder but none seems to work. I have set up light forwarder that sends cooked data to my indexer. Connection is ok, because data gets indexed but in wrong index.

Here are some of configurations i have tried (on indexer side):

inputs.conf 
[splunktcp://9997]
index = test_2

Next one:
inputs.conf 
[splunktcp://9997]

props.conf
[host::Simpson-test]
index = test_2
TRANSFORMS-foo = routeIndex

transforms.conf
[routeIndex]
SOURCE_KEY = source
REGEX = = WinEventLog
DEST_KEY =_MetaData:Index
FORMAT = test_2

Next one:
inputs.conf 
[splunktcp://9997]
sourcetype=REMOTE

props.conf
[REMOTE]
TRANSFORMS-simremote = sremote

transforms.conf
[sremote]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = test_2

Data allways ends up in index that was defined on lightforwarder (test_1). I would like to route data that comes to indexer based on sourcetype and host. Data comes from lightforwarder and is cooked so i suppose that sourcetype and host allready exist and i can do REGEX search on SOURCE_KEY?

Any hints abouth troubleshooting this?

Tags (2)
1 Solution

dwaddle
SplunkTrust
SplunkTrust

The light forwarder can send to a specific index if you set the destination index in inputs.conf on the forwarder itself. This can be overridden at the indexer using props.conf/transforms.conf for a specific host or sourcetype.

props.conf:

[myfavoritesourcetype]
    TRANSFORMS-index=sendtomyindex

transforms.conf:

[sendtomyindex]
SOURCE_KEY=MetaData:Sourcetype
REGEX=(.*)
FORMAT=index::myindex
WRITE_META=true

Update: it looks like this works just as well. I'm not sure which is preferable...

[sendtomyindex]
SOURCE_KEY=_MetaData:Index
DEST_KEY=_MetaData:Index
REGEX=(.*)
FORMAT=myindex

View solution in original post

Jason
Motivator

The easiest way is to specify the index in inputs.conf on the forwarder.

It seemed counter-intuitive to me at first, as if I was directing the forwarder to index something locally. But data is sent along to the indexer and placed into the desired index (as defined in an indexes.conf on the indexer).

On the forwarder, inputs.conf:

[monitor:///filename/or/other/input/type/]
disabled = false
index=ghtest5

dwaddle
SplunkTrust
SplunkTrust

The light forwarder can send to a specific index if you set the destination index in inputs.conf on the forwarder itself. This can be overridden at the indexer using props.conf/transforms.conf for a specific host or sourcetype.

props.conf:

[myfavoritesourcetype]
    TRANSFORMS-index=sendtomyindex

transforms.conf:

[sendtomyindex]
SOURCE_KEY=MetaData:Sourcetype
REGEX=(.*)
FORMAT=index::myindex
WRITE_META=true

Update: it looks like this works just as well. I'm not sure which is preferable...

[sendtomyindex]
SOURCE_KEY=_MetaData:Index
DEST_KEY=_MetaData:Index
REGEX=(.*)
FORMAT=myindex

Lowell
Super Champion

Side note on your regex: You should be able to use REGEX=.?, which should be more efficient way to say match-all.

dwaddle
SplunkTrust
SplunkTrust

Interesting, I tried both of these (and some variations that didn't work) and only saw one "index" field in each case. But then again, I'm struggling to see "_index" in the UI. How are you seeing that?

0 Karma

gljiva
Path Finder

Thanks alot, got it to work!
First one doesn't exactly work. It really adds new _index field to data, but doesn't delete old one. So I end up having two _index fields, but only first _index field is used to store data, and second one seems to get ignored.

Second one works great if i write it like this:
[routeIndex]
SOURCE_KEY = MetaData:Sourcetype
REGEX=(.*)
DEST_KEY=_MetaData:Index
FORMAT=test_2
WRITE_META=true

Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Can’t Make It to Boston? Stream .conf25 and Learn with Haya Husain

Boston may be buzzing this September with Splunk University and .conf25, but you don’t have to pack a bag to ...

Splunk Lantern’s Guide to The Most Popular .conf25 Sessions

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Unlock What’s Next: The Splunk Cloud Platform at .conf25

In just a few days, Boston will be buzzing as the Splunk team and thousands of community members come together ...