Why can't Splunk_TA_windows redirect to a custom index?

[rlaan]

I have added the latest version of Splunk_TA_windows to my environment using a deployment server.

The app has been pushed to all windows machines, the search heads and the heavy forwarders.

I have only been receiving data into the "Main" index and be unsuccessful at redirecting the data to our preferred collection points index =  wineventlog.

on the deployment server i have created a  Splunk_TA_windows/local/inputs.conf file containing the following.

[WinEventLog://ForwardedEvents]
index = wineventlog
disabled = 0

[WinEventLog://Application]
index = wineventlog
disabled = 0

[WinEventLog://System]
index = wineventlog
disabled = 0

[XmlWinEventLog]
index = wineventlog

[WinEventLog]
index = wineventlog

I am primarily a linux guy for splunk admin and only have 1 windows host monitored at the moment (all windows events are forwarded to and collected from this node), is there something that needs to be done differently to redirect the index for this applications? Next consideration I have is using props/transforms to change the index although am worried about the hardware impact of that on 5 million events a day.

[rlaan · Accepted solution]

Thank you everyone for all the help, realized the deployment was pushing successfully and after getting access to the servers. Noticed that the deployment server wasn't triggering restarts to apply the conf changes.

The issue ended up being a serverclass.conf configuration where an entry for restartSplunkd = true... was instead restartSplunkD = true

The capital D was halting and splunkd restarts on the windows machines so the new configurations were never applied.

[gcusello]

Hi @rlaan,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the Contributors;-)

[rlaan]

Did some more testing, it appears the windows forwarder had a locally configured app 
SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\local\inputs.conf

Containing the following entry
[WinEventLog://ForwardedEvents]
checkpointInterval = 5
current_only = 0
disabled = 0
start_from = oldest

I had a windows admin remove the extra configuration in the above local app and added the same entry to the Splunk_TA_windows/local/inputs.conf on the deployment server and confirmed it was pushed to the windows forwarder.

This broke the data inputs, so now i am wondering why the Splunk_TA_windows app pushed via deployment server doesn't seem to be detected by the universal forwarder. While the locally added file in apps\SplunkUniversalForwarder\local\inputs.conf redirected the forwarded events. Anyone have an idea on what to check to get the deployment server managed applications to function on the forwarder?

[PickleRick]

It doesn't work that way. You can have a default settings in [default] stanza but you can't use general "input type" stanza.

So you need to either define a default setting or add index parameter to each individual input.

[rlaan]

How would i go about adding a "add index parameter to each individual input" I thought that was already what i was doing with the individual stanza such as 

[WinEventLog://Application]
index = wineventlog

 I guess there is a gap in my understanding of how the monitors are defined on windows, i have tried adding index=wineventlog to all stanza present within the default/inputs.conf into my local version without any events being sent to the new index.

Many windows servers are sending data to the single universal forward before passing it into splunk, is this aggregation using window tools possibly an issue with how the indexes are defined? 

[PickleRick]

OMG, my bad. I didn't notice that you do have the index specified under each input as well. I suppose I noticed the entries in the "genre" stanzas and assumed you didn't put them there.

Did you check with btool what's your effective input conifg?

c:\program files\splunkuniversalforwarder\bin\splunk btool inputs list --debug

The installation path might of course differ, it might be c:\program files\splunk for older forwarder versions and if you installed it somewhere else completely, look for it there 😉

[rlaan]

if the default stanza is added would this cause all things sent to main to go to the new index (potentially from unrelated applications?) I am curious to potential environmental level effects as the application is deployed on the heavy forwarders and could effect all inputs.

[PickleRick]

Short answer - it could.

Long answer - it depends on how other inputs are defined. But the typical approach is to explicitly define destination index per input. This way you avoid such unexpected changes in behaviour.

BTW, remember that efective config depends on config files priority so such "default" setting could theoretically be overwritten by another default setting.

[gcusello]

Hi @rlaan,

id you deployed the TA-Windows modified as you said to all Windows Universal Forwarder, you should have events in the wineventlog index.

If not, you could also override index on Indexers or (when present) on Heavy Forwarders.

To override index you have to create a props.conf

[your_sourcetype1]
TRANSFORMS-index = overrideindex

[your_sourcetype2]
TRANSFORMS-index = overrideindex

[your_sourcetype3]
TRANSFORMS-index = overrideindex

and a transforms.conf

[overrideindex]
DEST_KEY =_MetaData:Index
REGEX = .
FORMAT = wineventlog

Ciao.

Giuseppe