How can I collect events from several groups of Windows servers with separate dedicated indexes?

[evgenyv]

My goal is to create a multi-tenant environment for monitoring several groups of Windows Servers.
In other words, I’d like to index every group with a separate dedicated index.
The Splunk Universal Forwarder is installed on every Windows Server and its output is directed to my Splunk Enterprise.

My original idea was:
1. To create a Server Class for each group of servers
2. To specify a separate index to each Sever Class

For the 1st group everything went ok:
Windows Events Logs -> New ->
Select Forwarders: Here I selected servers for the 1st group (group1) -> Next
Select Source: Here I selected relevant event log channels (Application) -> Next
Input Setting: Here I selected an index for the 1st group (idx_group1)-> Review -> Save

As a result, the new deployment application was created for the Server Class:
In /opt/splunk/etc/deployment-apps/_server_app_group1/local/inputs.conf

[WinEventLog://Application]
disabled = 0
index = idx_group1

Indeed, the events from the server are indexed by idx_group1!

However, when I did the same for the 2nd group, I’ve got an error:

Cannot create another input for the event log "Application", one already exists.

Splunk says “The event log monitor runs once for every event log input defined in Splunk.”

So my question is – how can I collect the events from several groups of servers, when each group is indexed by a dedicated index?

[javiergn]

That's one of the reasons I don't like the Forwarder Management when using advanced configurations.

Try editing the serverclass.conf file manually:

http://docs.splunk.com/Documentation/Splunk/6.3.2/Updating/Useserverclass.conf

[evgenyv]

Thanks, javiergn.
Unfortunately, I didn't find a way to associate the serverclass with the index.

The approach Route specific events to a different index is described in
http://docs.splunk.com/Documentation/Splunk/6.3.2/Indexer/Setupmultipleindexes

Does it work for events sent by a Forwarder? Should I have only one Server Class for all server in this case?

Not clear to me to what props.conf and transforms.conf can I apply it in case of Windows Events Log.
Will it overwrite the index setting in inputs.conf of the application?

Is there any limitation on number of stanza's in transforms.conf? (I expect dozens of groups and dozens of server in each group)

[javiergn]

Hi,

It might be easier to just create one serverclass for all the windows servers and then group by app. That is, every app will go to a different group of servers based on the whitelist/blacklist regex you write in your serverclass.conf. See this

For instance (based on Example 3 from the link above):

[global]
# whitelist.0=* at the global level ensures that the machineTypesFilter attribute
# invoked later will apply.
whitelist.0=*

[serverClass:WindowsMachineTypes]
machineTypesFilter=windows-*

[serverClass:WindowsMachineTypes:app:Group1_EventLogsApp]
whitelist.0=YOURWHITELIST
blacklist.0=YOURBLACKLIST

You need to make sure every app has its own inputs.conf and that intputs.conf uses a different index.
This should work just fine providing you don't deploy several apps to the same server, that is, all the whitelists/blacklists need to be mutually exclusive.

An alternative that might increase performance but will probably simplify deployment is as follows:

• Create one serverclass and one app for all your Windows servers you want to monitor
• Do not specify the index name during collection. Just define a sourcetype so that you can easily filter out by that later on.
• Create a transform in your heavy forwarder/indexer (whichever is your next hop) to overwrite the index name before indexing based on whichever filter you want to use. See this

[evgenyv]

I did this, but got the following errors:

stanza=serverClass:vcpe_win:app:_server_app_group1 property=whitelist.0 reason='unsupported at this level'
stanza=serverClass:vcpe_win:app:_server_app_group2 property=whitelist.0 reason='unsupported at this level'

my serverclasses.conf is

[serverClass:vcpe_win]
restartSplunkd = true

[serverClass:vcpe_win:app:_server_app_group1]
whitelist.0 = 10.20.4.213

[serverClass:vcpe_win:app:_server_app_group2]
whitelist.0 = 10.20.4.214