topic Re: Route data to index based on host in Getting Data In

Route data to index based on host

Yancy — Sat, 03 Apr 2010 09:04:22 GMT

Hi folks,

I'd like to route WMI logs to different indexes based off the host name (I have a few environments)

Going off the directions here: http://www.splunk.com/support/forum:SplunkAdministration/3468 I'm not sure why this isn't working. My data from my remote collected WinEventLog keeps going into 'main'. Ideas?

props.conf

[host::foo*]
TRANSFORMS-foo = route_to_foo

transforms.conf

[route_to_foo]
SOURCE_KEY = _raw
REGEX = (?m)^wmi_type=(WinEventLog:System|WinEventLog:Application)
DEST_KEY = _MetaData:Index
FORMAT = foo

EDIT: Per suggestions this is now working as below

props.conf

[wmi]
TRANSFORMS-FIELDS = wmi-host, wmi-source, wmi-sourcetype, wmi-color

transforms.conf

default transform for wmi-host

[wmi-host]
REGEX = (?m)ComputerName=(.+)
DEST_KEY = MetaData:Host
FORMAT = $1

Environment host names are unique in the three characters after "FOO" ie for the BAR & BAD environment, host names would be "FOOBAR1", "FOOBAR2", "FOOBAD1", "FOOBAD2"

[wmi-color]
SOURCE_KEY = MetaData:Host
REGEX = (?m)FOO(.{3})(.+)
DEST_KEY = _MetaData:Index
FORMAT = $1

Re: Route data to index based on host

Simon — Sat, 03 Apr 2010 15:56:01 GMT

Check the host field of the events in your 'main' index. Make sure it matches with your wildcard in props.conf (foo*).

Re: Route data to index based on host

Dan — Sat, 03 Apr 2010 21:06:42 GMT

I would test with a simpler REGEX. It looks like yours might have unmatched parentheses. I'd also try matching on MetaData:Source as your SOURCE_KEY rather than _raw. Seems simpler and would have better performance.

Re: Route data to index based on host

gkanapathy — Sat, 03 Apr 2010 23:12:40 GMT

In the case of wmi inputs, unfortunately the source is simply set to "wmi", and is set during the same phase as this TRANSFORM.

Re: Route data to index based on host

gkanapathy — Sat, 03 Apr 2010 23:16:29 GMT

Oh, yeah, I just thought of another thing while commenting on another answer to this. By default, [wmi] sourcetype also does a TRANSFORM that sets the host, set in etc/system/default/props.conf and etc/system/default/transforms.conf. It's likely that it's running after your host one and therefore overwriting your change.

You might need to override that by setting:

[wmi]
TRANSFORMS-FIELDS = wmi-source, wmi-sourcetype
# the original was TRANSFORMS-FIELDS = wmi-host, wmi-source, wmi-sourcetype

Update:

oops sorry, got confused, thought you were trying to update "host" not "index". Ignore the override I said and look at kbains answer.

Re: Route data to index based on host

Yancy — Mon, 05 Apr 2010 22:51:16 GMT

Thanks, I think this is on the right track.. just not quite there yet.

Overriding the props.conf [wmi] per your suggestion, it now simply leaves the host as the Splunk Server. Index is still set to 'main'.

This is on 4.0.10

Re: Route data to index based on host

kbains — Tue, 06 Apr 2010 00:14:31 GMT

since the hostname is extract via a transform stanza itself, you cannot call the routing transform based on host. here are the configuration changes you should make:

props.conf:

[wmi]
TRANSFORMS-FIELDS = wmi-host, wmi-source, wmi-sourcetype, wmi-foo1, wmi-foo2

#notice the wmi-foo1 and wmi-foo2 transforms are called AFTER the wmi-host transform

transforms.conf:

#make everything from this host go to a different index 
[wmi-foo1]
SOURCE_KEY = MetaData:Host
REGEX = <the host you want to route>
DEST_KEY = _MetaData:Index
FORMAT = foo

#now revert the index for the events you DON'T want to in the foo index for the host
[wmi-foo2]
SOURCE_KEY = _raw
REGEX = <some regex here based on raw text event>
DEST_KEY = _MetaData:Index
FORMAT = <non foo index>

Re: Route data to index based on host

gkanapathy — Tue, 06 Apr 2010 01:50:31 GMT

FORMAT = index::$1 should be FORMAT = $1 in stanza [wmi-color]

Re: Route data to index based on host

Ledio_Ago — Thu, 08 Apr 2010 08:12:56 GMT

Another example redirecting Windows event logs using the log channel name:

http://answers.splunk.com/questions/970/how-do-i-configure-splunk-to-index-windows-event-log-data-in-separate-indexes

Re: Route data to index based on host

jblaine — Thu, 03 Mar 2011 04:08:33 GMT

I'm having a hell of a time with this, and it seems right to continue the thread here instead of posting a nearly identical question.

"All" I am trying to do is route ANY data (syslog, WMI, whatever ... anything) from host::ats-* to the index 'ats'

I have tried:

# props.conf
[host::ats-*]
TRANSFORMS-whatever = ats

with

# transforms.conf
[ats]
REGEX = .+
DEST_KEY = _MetaData:Index
FORMAT = ats

and the only thing I see in index=ats is some notifications from the Splunk forwarder(s) on ats-* hosts due to my restarting our main Splunk server. There is no WMI data showing up in index=ats, as it is all still going to index=main. This WMI data that I can see in index=main clearly shows fields of host=ats-17.our.org, etc.

Any advice would greatly be appreciated.

Re: Route data to index based on host

Yancy — Fri, 11 Mar 2011 02:45:02 GMT

Hi, it's probably a good idea to open a new question and link back to this one, as new questions are much more visible. This question has been answered, and while your scenario is similar, I'm not quite sure what the next steps would be.