topic Re: Routing to index based on host, etc. in Getting Data In

Routing to index based on host, etc.

mmletzko — Sat, 30 Oct 2010 03:49:54 GMT

We have an indexer and two forwarders. The forwarders are installed on other syslog servers to forward their syslogs to the indexer. The indexer has 2 indexes - main and index1

I have successfully configured Splunk via props/transforms to route syslog sent directly from a Juniper to the indexer to the "index1" index instead of the default "main" by using the following config on the indexer:

props.conf

[syslog]
TRANSFORMS-syslog-NSM=syslog-NSM

transforms.conf

[syslog-NSM]
DEST_KEY = _MetaData:Index
REGEX = NSM
FORMAT = index1

I am now trying to route any syslog that includes "XUS" in the host name to index1 but no matter what I try I cannot get it to work. I'm suspecting that there is relevance to the syslog being sent directly to the indexer vs syslog being forwarded to the indexer via splunk forwarder. Is this true?

This is my latest of many attempts to get this to work. What am I missing? I understand my regex may not be correct, but I've tried more 'open' regexs and I still can't route anything coming from the forwarder. I can get other syslogs being sent directly to the indexer to route to index1, but nothing from the forwarder:

props.conf (updated)

[syslog]
TRANSFORMS-syslog-NSM=syslog-NSM
TRANSFORMS-syslog-vault=syslog-vault

transforms.conf (updated)

[syslog-NSM]
DEST_KEY = _MetaData:Index
REGEX = NSM
FORMAT = index1

[syslog-vault]
SOURCE_KEY = _MetaData:Index  (have tried with/without this)
DEST_KEY = _MetaData:Index
REGEX = XUS
FORMAT = index1
WRITE_META = true  (have tried with/without this)

Is there something I can do on the indexer to get this to work?..meaning...can I configure routing on the indexer for syslog received via Splunk forwarder? Or do I need to perform the routing on the forwarder itself? If so, how should I do it?

Thanks!!

Re: Routing to index based on host, etc.

chris — Mon, 01 Nov 2010 00:23:48 GMT

Hi Mmletzko

You are right there is a difference between the messages being sent directly to the indexer and forwarded messages. It depends on whether you are using the light forwarder or the regular forwarder aswell.

There is good wiki page that helps a lot which is here (Where do I configure my Splunk settings)

If I understand you correctly you have:

a regular forwarder and an indexer
a juniper device sends messages containing "NSM" directly to the indexer
other devices that send "XUS" messages to the forwarder

-> Both messages should end up in index1

I think this is what you need:

On the indexer
props.conf [syslog] TRANSFORMS-syslog-NSM=syslog-NSM

transforms.conf [syslog-NSM] DEST_KEY = _MetaData:Index REGEX = NSM FORMAT = index1

On the forwarder
props.conf [syslog] TRANSFORMS-syslog-vault=syslog-vault

transforms.conf [syslog-vault] DEST_KEY = _MetaData:Index REGEX = XUS FORMAT = index1

That is almost what you had, I hope this helps

Re: Routing to index based on host, etc.

mmletzko — Mon, 01 Nov 2010 19:46:53 GMT

Chris - THANK YOU!!!

Looks like I danced around this, but didn't quite have it. I had the correct configuration in the transforms/props files, but I think the problem was that I had the routing configured on the forwarder (full forwarder, btw) AND the indexer at the same time. The one thing I hadn't tried was having it configured ONLY on the forwarder and not the indexer. Once I removed the settings from the indexer and then enabled them ONLY on the forwarder, it did the trick!!!

So here are the final settings that did the trick:

indexer - props.conf

[syslog]
TRANSFORMS-syslog-NSM=syslog-NSM

indexer - transforms.conf

[syslog-NSM]
DEST_KEY = _MetaData:Index
REGEX = NSM
FORMAT = index1

forwarder - props.conf

[syslog]
TRANSFORMS-syslog-vault=syslog-vault

forwarder - transforms.conf

[syslog-vault]
DEST_KEY = _MetaData:Index
REGEX = XUS
FORMAT = index1

Thanks again Chris - you restored my sanity!!

-Matt

Re: Routing to index based on host, etc.

chris — Tue, 09 Nov 2010 17:32:49 GMT

I'm glad that it works now