Getting Data In

Route data to index based on host

Yancy
Path Finder

Hi folks,

I'd like to route WMI logs to different indexes based off the host name (I have a few environments)

Going off the directions here: http://www.splunk.com/support/forum:SplunkAdministration/3468 I'm not sure why this isn't working. My data from my remote collected WinEventLog keeps going into 'main'. Ideas?

props.conf

[host::foo*]
TRANSFORMS-foo = route_to_foo

transforms.conf

[route_to_foo]
SOURCE_KEY = _raw
REGEX = (?m)^wmi_type=(WinEventLog:System|WinEventLog:Application)
DEST_KEY = _MetaData:Index
FORMAT = foo

EDIT: Per suggestions this is now working as below

props.conf

[wmi]
TRANSFORMS-FIELDS = wmi-host, wmi-source, wmi-sourcetype, wmi-color

transforms.conf

default transform for wmi-host

[wmi-host]
REGEX = (?m)ComputerName=(.+)
DEST_KEY = MetaData:Host
FORMAT = $1

Environment host names are unique in the three characters after "FOO" ie for the BAR & BAD environment, host names would be "FOOBAR1", "FOOBAR2", "FOOBAD1", "FOOBAD2"

[wmi-color]
SOURCE_KEY = MetaData:Host
REGEX = (?m)FOO(.{3})(.+)
DEST_KEY = _MetaData:Index
FORMAT = $1
1 Solution

kbains
Splunk Employee
Splunk Employee

since the hostname is extract via a transform stanza itself, you cannot call the routing transform based on host. here are the configuration changes you should make:

props.conf:

[wmi]
TRANSFORMS-FIELDS = wmi-host, wmi-source, wmi-sourcetype, wmi-foo1, wmi-foo2

#notice the wmi-foo1 and wmi-foo2 transforms are called AFTER the wmi-host transform   

transforms.conf:

#make everything from this host go to a different index 
[wmi-foo1]
SOURCE_KEY = MetaData:Host
REGEX = <the host you want to route>
DEST_KEY = _MetaData:Index
FORMAT = foo

#now revert the index for the events you DON'T want to in the foo index for the host
[wmi-foo2]
SOURCE_KEY = _raw
REGEX = <some regex here based on raw text event>
DEST_KEY = _MetaData:Index
FORMAT = <non foo index>

View solution in original post

jblaine
Explorer

I'm having a hell of a time with this, and it seems right to continue the thread here instead of posting a nearly identical question.

"All" I am trying to do is route ANY data (syslog, WMI, whatever ... anything) from host::ats-* to the index 'ats'

I have tried:

# props.conf
[host::ats-*]
TRANSFORMS-whatever = ats

with

# transforms.conf
[ats]
REGEX = .+
DEST_KEY = _MetaData:Index
FORMAT = ats

and the only thing I see in index=ats is some notifications from the Splunk forwarder(s) on ats-* hosts due to my restarting our main Splunk server. There is no WMI data showing up in index=ats, as it is all still going to index=main. This WMI data that I can see in index=main clearly shows fields of host=ats-17.our.org, etc.

Any advice would greatly be appreciated.

0 Karma

Yancy
Path Finder

Hi, it's probably a good idea to open a new question and link back to this one, as new questions are much more visible. This question has been answered, and while your scenario is similar, I'm not quite sure what the next steps would be.

0 Karma

Ledio_Ago
Splunk Employee
Splunk Employee

Another example redirecting Windows event logs using the log channel name:

http://answers.splunk.com/questions/970/how-do-i-configure-splunk-to-index-windows-event-log-data-in...

gkanapathy
Splunk Employee
Splunk Employee

FORMAT = index::$1 should be FORMAT = $1 in stanza [wmi-color]

0 Karma

kbains
Splunk Employee
Splunk Employee

since the hostname is extract via a transform stanza itself, you cannot call the routing transform based on host. here are the configuration changes you should make:

props.conf:

[wmi]
TRANSFORMS-FIELDS = wmi-host, wmi-source, wmi-sourcetype, wmi-foo1, wmi-foo2

#notice the wmi-foo1 and wmi-foo2 transforms are called AFTER the wmi-host transform   

transforms.conf:

#make everything from this host go to a different index 
[wmi-foo1]
SOURCE_KEY = MetaData:Host
REGEX = <the host you want to route>
DEST_KEY = _MetaData:Index
FORMAT = foo

#now revert the index for the events you DON'T want to in the foo index for the host
[wmi-foo2]
SOURCE_KEY = _raw
REGEX = <some regex here based on raw text event>
DEST_KEY = _MetaData:Index
FORMAT = <non foo index>

gkanapathy
Splunk Employee
Splunk Employee

Oh, yeah, I just thought of another thing while commenting on another answer to this. By default, [wmi] sourcetype also does a TRANSFORM that sets the host, set in etc/system/default/props.conf and etc/system/default/transforms.conf. It's likely that it's running after your host one and therefore overwriting your change.

You might need to override that by setting:

[wmi]
TRANSFORMS-FIELDS = wmi-source, wmi-sourcetype
# the original was TRANSFORMS-FIELDS = wmi-host, wmi-source, wmi-sourcetype

Update:

oops sorry, got confused, thought you were trying to update "host" not "index". Ignore the override I said and look at kbains answer.

Yancy
Path Finder

Thanks, I think this is on the right track.. just not quite there yet.

Overriding the props.conf [wmi] per your suggestion, it now simply leaves the host as the Splunk Server. Index is still set to 'main'.

This is on 4.0.10

0 Karma

Dan
Splunk Employee
Splunk Employee

I would test with a simpler REGEX. It looks like yours might have unmatched parentheses. I'd also try matching on MetaData:Source as your SOURCE_KEY rather than _raw. Seems simpler and would have better performance.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

In the case of wmi inputs, unfortunately the source is simply set to "wmi", and is set during the same phase as this TRANSFORM.

0 Karma

Simon
Contributor

Check the host field of the events in your 'main' index. Make sure it matches with your wildcard in props.conf (foo*).

0 Karma
Get Updates on the Splunk Community!

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...

Thank You for Celebrating CX Day with Splunk!

Yesterday the entire team at Splunk &#43; Cisco joined the global celebration of CX Day - celebrating our ...