topic Re: Sending to different index based on host from syslog stream in Monitoring Splunk

Sending to different index based on host from syslog stream

dkoski — Wed, 22 Sep 2010 20:40:50 GMT

Ok, so I'm not sure what I'm doing wrong here. I need to send data coming in on the standard syslog UDP 514 stream based on hostname (Most of it going to main, the rest going to a custom index).

i created a new props.conf in etc/system/local with just:

[host::10.10.10.*]
TRANSFORM-test = route_to_test

And a new transforms.conf with just:

[route_to_test]
REGEX = .*
DEST_KEY = _MetaData:Index
FORMAT = test

My intention was to route anything coming in from 10.10.10.* to an index named test, however everything is still going to main.

What am I missing?

Re: Sending to different index based on host from syslog stream

southeringtonp — Wed, 22 Sep 2010 20:46:44 GMT

Does the REGEX statement require a capture group here? I can't remember...

What happens if you change it to:

REGEX = (.)

Re: Sending to different index based on host from syslog stream

ziegfried — Wed, 22 Sep 2010 20:47:02 GMT

This is how you need to configure the transform:

[route_to_test]
SOURCE_KEY=MetaData:Host
DEST_KEY=_MetaData:Index
REGEX=^host::(10\.10\.10\.\d+)$
FORMAT=test
WRITE_META=true

and you could use the sourcetype or source in the props:

[syslog]
TRANSFORMS-test = route_to_test

[source::udp:514]
TRANSFORMS-test = route_to_test

Re: Sending to different index based on host from syslog stream

dkoski — Wed, 22 Sep 2010 21:36:01 GMT

So how would you do multiple subnets (or hosts) to multiple indexes then coming in on a syslog stream?

Re: Sending to different index based on host from syslog stream

ziegfried — Wed, 22 Sep 2010 21:44:43 GMT

Like this: REGEX=^host::(10.10.10.\d+|192.168.0.\d+)$

Re: Sending to different index based on host from syslog stream

ziegfried — Wed, 22 Sep 2010 21:46:13 GMT

You could probably use your approach with the WRITE_META=true stuff as well. Didn't try that one exactly myself.

Re: Sending to different index based on host from syslog stream

gkanapathy — Thu, 23 Sep 2010 01:32:21 GMT

No, you don't need WRITE_META = true if writing to the _MetaData:Index DEST_KEY.

Re: Sending to different index based on host from syslog stream

gkanapathy — Thu, 23 Sep 2010 01:33:06 GMT

Really though, it seems to me his original solution should work as long as the incoming host name is in fact 10.10.10.*. I guess I would set the sourcetype to something not syslog (something that doesn't set the host) so you can see what's in the incoming data to be sure.

Oh, wait. Can you try:

TRANSFORMS-test = route_to_test

instead of:

TRANSFORM-test = route_to_test

And also as southeringtonp suggests, do not use the regex .*, use just .? or (?=) instead.

Re: Sending to different index based on host from syslog stream

ziegfried — Thu, 23 Sep 2010 01:38:11 GMT

Are you sure? I experienced it in the past that it did not work without the WRITE_META...

Re: Sending to different index based on host from syslog stream

satishp — Thu, 10 Mar 2011 23:53:11 GMT

Hey Guys,

I have same question I have 4 hosts 2 development and 2 production. I have created two index dev & prod.

I am using syslog TCP 514 port so how should i transfer production logs on prod and dev logs on dev index ?

I tried every combination but no luck... please help to find solution.