topic Re: 'Invalid Key in Stanza' errors being generated at startup for inputs.conf whitelist on a 6.1.4 Heavy Forwarder that docs say should work in Getting Data In

'Invalid Key in Stanza' errors being generated at startup for inputs.conf whitelist on a 6.1.4 Heavy Forwarder that docs say should work

wrangler2x — Tue, 15 Dec 2015 00:26:02 GMT

Per these docs http://docs.splunk.com/Documentation/Splunk/6.1.4/Data/MonitorWindowsdata I have changed from the old way of using transforms to filter out unwanted Windows Events from logs I am monitoring to using a whitelist in inputs.conf. I am sending these to forwarders on various windows systems using deployment monitor. While restarting a Splunk forwarder that had died for some reason, I got this error on startup:

Invalid key in stanza [WinEventLog:Security] in C:\Program Files\Splunk\etc\apps\OIT_WINEVENT_DC_INDEX_WIN_01\default\inputs
.conf, line 23: whitelist (value: 528-535,539-540,624-626,632,636,659,642-644,660,675-676,671-672,680-681,1100,1102,1104,1108,4612,4616,4618,4624-4625,4634,4720,4728,4732,4738,4740,4756,4767-4768,4771-4772,4776,5461 )

Here is the stanza from the inputs.conf file in question:

[WinEventLog:Security]
disabled = 0
index= winevent_dc_index
whitelist = 528-535,539-540,624-626,632,636,659,642-644,660,675-676,671-672,680-681,1100,1102,1104,1108,4612,4616,4618,4624-4625,4634,4720,4728,4732,4738,4740,4756,4767-4768,4771-4772,4776,5461

It looks just like the example in the documentation. Also, this blog entry says it should work in Splunk 6: http://blogs.splunk.com/2013/10/14/windows-event-logs-in-splunk-6/

So, why is this not working?

I also ran btool (it says the same thing):

C:\Program Files\Splunk\bin>splunk btool check --debug

Invalid key in stanza [WinEventLog:Security] in C:\Program Files\Splunk\etc\apps\OIT_WINEVENT_DC_INDEX_WIN_01\default\inputs.conf, line 23: whitelist  (value:  528-535,539-540,624-626,632,636,659,642-644,660,675-676,671-672,680-681,1100,1102,1104,1108,4612,4616,4618,4624-4625,4634,4720,4728,4732,4738,4740,4756,4767-4768,4771-4772,4776,5461 )

Re: 'Invalid Key in Stanza' errors being generated at startup for inputs.conf whitelist on a 6.1.4 Heavy Forwarder that docs say should work

MuS — Tue, 15 Dec 2015 00:32:56 GMT

Hi wrangler2x,

just a hint, look at the inputs.conf.spec file if this is listed

  whitelist = <regular expression>

Usually if you get this error something is missing in the .conf.spec file.

cheers, MuS

Re: 'Invalid Key in Stanza' errors being generated at startup for inputs.conf whitelist on a 6.1.4 Heavy Forwarder that docs say should work

wrangler2x — Tue, 15 Dec 2015 00:46:45 GMT

In the link you have there, there is a section called EventLog filtering which shows what the 6.1.4 documentation page I linked to (original question) describes. However, if I look at the actual 6.1.4 inputs.conf.spec I don't find that. Looks like my release may not support it, and the docs are wrong.

Re: 'Invalid Key in Stanza' errors being generated at startup for inputs.conf whitelist on a 6.1.4 Heavy Forwarder that docs say should work

MuS — Tue, 15 Dec 2015 01:18:49 GMT

No, it more means someone forgot to add in the inputs.conf.spec and therefore splunk does not know about it and thinks it is an error.

BTW just downloaded a 6.1.4 Windows UF 64Bit and the input.conf.spec contains this on line 174:

 whitelist = <regular expression>

Maybe download your version again?

Re: 'Invalid Key in Stanza' errors being generated at startup for inputs.conf whitelist on a 6.1.4 Heavy Forwarder that docs say should work

javiergn — Tue, 15 Dec 2015 10:38:46 GMT

Hi, your stanza looks all right to me.

Have you tried adding your event codes one at a time and see what happens?
Maybe the line is too long?
Or:

[MODE PARANOID ON]
Maybe the hyphen separating your event ID ranges is not the right type of hyphen.
See this: https://www.cs.tut.fi/~jkorpela/dashes.html

Or maybe there are hidden characters that your editor does not display
[MODE PARANOID OFF]

Hope that helps,
J

Re: 'Invalid Key in Stanza' errors being generated at startup for inputs.conf whitelist on a 6.1.4 Heavy Forwarder that docs say should work

wrangler2x — Tue, 15 Dec 2015 20:13:39 GMT

@MuS I have that same thing on line 174 also. But that is in the Monitor section. If you look at the latest documentation, under this section:

###
# Windows Event Log Monitor
###

You will find a sub-section which looks like this:

# EventLog filtering
#
# Filtering at the input layer is desirable to reduce the total processing load
# in network transfer and computation on the Splunk nodes acquiring and
# processing the data.

and in this section there is this:

whitelist = <list of eventIDs> | key=regex [key=regex]
blacklist = <list of eventIDs> | key=regex [key=regex]

And this:

The base unumbered whitelist and blacklist support two formats, a list of integer event IDs, and a list of key=regex pairs.

Now, in my 6.1.4 spec, in the Windows Event Log Monitor section, there is no subsection called EventLog filtering. However, just below the evt_dns_name = and index = specs, I do find these two (lines 1130 and 1141, respectively) :

whitelist = <list> | key=regex [key=regex]
blacklist = <list> | key=regex [key=regex]

And there are two comments with the whitelist which read:

 * In list form, tells Splunk which event IDs and/or event ID ranges that incoming events must have
  in order to be indexed.
 * In list form, A comma-separated list of event ID and event ID ranges to include (example: 4,5,7,100-200).

So, I'd say they are in the spec, although documented much differently than in the current documentation!

So, what next?

Re: 'Invalid Key in Stanza' errors being generated at startup for inputs.conf whitelist on a 6.1.4 Heavy Forwarder that docs say should work

wrangler2x — Tue, 15 Dec 2015 20:32:05 GMT

I removed all of the hyphens and used just a full list with commas only. Same error. Then I delected the whitelist line and manually added a new line which reads whitelist = 528 to keep it really simple, and so there was no chance of a hidden character. Same error on restart.

Re: 'Invalid Key in Stanza' errors being generated at startup for inputs.conf whitelist on a 6.1.4 Heavy Forwarder that docs say should work

Michael_Carlisl — Wed, 06 Apr 2016 18:47:54 GMT

So it seems that the issue is missing "\". If you update your inputs.conf file to be [WinEventLog://Security] it should work.

Re: 'Invalid Key in Stanza' errors being generated at startup for inputs.conf whitelist on a 6.1.4 Heavy Forwarder that docs say should work

wrangler2x — Wed, 06 Apr 2016 19:13:36 GMT

Wait, that's confusing... You say it is missing "\" then show "//" in [WinEventLog://Security]

But yeah, the docs at http://docs.splunk.com/Documentation/Splunk/6.0.2/Data/MonitorWindowsdata show it as "//".

I did not set up the stanzas we are using (which otherwise work fine without the "//") and the blog at http://blogs.splunk.com/2013/10/14/windows-event-logs-in-splunk-6/ shows it without the "//" so I did not event know these were missing until I read your comment. Did you test to see if adding them in allows the whitelist? Which version of Splunk?

Re: 'Invalid Key in Stanza' errors being generated at startup for inputs.conf whitelist on a 6.1.4 Heavy Forwarder that docs say should work

Michael_Carlisl — Wed, 06 Apr 2016 19:50:45 GMT

I tested it in version 6.3.1. Sorry about putting "\". There's also an issue with [WinEventLog:Application] if you ever use the Citrix Add Ons and use their inputs.conf file.