Why does migrating universal forwarder to 6.4.3 display "unconfigured/disabled/deleted index='wineventlog'" message and event logs stopped forwarding?

[morganfw]

Hi all,
I've 3 Splunk 6.4.1 Indexers and a Splunk 6.4.1 Search Head + Distributed Management Console (DMC) on Linux Red Hat 6.6.
I've tested Windows Event Log in Windows 2008 R2 Domain Controller Servers in Preproduction environment with 1 Splunk 6.4.1 Indexer + Search Head + DMC, forwarded Event Logs all ok.

I've migrated Splunk Universal Forwarder (SUF) in Production from 5.0.2 to 6.4.3 with clean Installation (Uninstall SUF 5.0.2, Reboot Server and Reinstall SUF 6.4.3), and before with SUF 5.0.2 Windows Events was forwarded with no problem, after SUF Clean Upgrade to 6.4.3 I receive once following message:

Received event for unconfigured/disabled/deleted index='wineventlog' with source='source::WinEventLog:Security' host='host::my-host' sourcetype='sourcetype::WinEventLog:System' (1 missing total)

and Event Logs stopped to be forwarded.
I haven't changed configuration on my Indexers and Search Head, below my configuration:

• $SPLUNK_HOME/etc/system/local/serverclass.conf
• List item

[serverclass:domain_controller]
host = my-dc-host
[serverclass:domain_controller:app:domain_controller]
* $SPLUNK_HOME/etc/deployment-apps/domain_controller/default/inputs.conf
[WinEventLog://Security]
disabled = 0
(I've also tried to add "index = main" on bottom of above stanza with no results).

And other configurations to send logs globally from deployment clients to deployment server...

I've tried to uninstall and reinstall SUF 6.4.3, but no issue resolved, I've also read all Splunk Answers on same problem, but before SUF upgrade Windows Event Logs was Forwarded with no problem, and in Preproduction all works fine.

Any suggestion?
Regards.

[jwelch_splunk · Accepted solution]

After working thru issue.

Splunk_TA_windows from the new install of the UF, and that would over-ride the app of
domain_controller because a Capital S takes a higher priority on an app than a lowercase d.

Okie

[jwelch_splunk]

It appears that your events are going to a new index called wineventlog, perhaps on the older system they were going to winevents.

The solution here is to create a new index called wineventlog, or use your inputs.conf to direct the data back to the old existing index name.

I assume this is a result of the Windows TA upgrade.

Okie

[morganfw]

I know that events going to new index that not exists, but my trouble is another.

I've plain Splunk Enterprise Core, no TA, my issue is particular because before with SUF 5.0.2 to Indexer 6.4.1 Windows Event Log was Forwarded, after "clean" upgrade to SUF 6.4.3 Forwarder stop to send Event Log to Indexer 6.4.1.

Before SUF upgrade Event Logs were going to "main" index with no explicit declaration in inputs.conf, so I've tried to configure explicitly the previous used index in inputs.conf and reload serverclasses with no success, and I don't need to create new index because prior upgrade this worked.

In Preproduction Environment all works fine and SUF forward events log to Indexer.

I hope I explained myself.

[gjanders]

With this point

$SPLUNK_HOME/etc/deployment-apps/domain_controller/default/inputs.conf

[WinEventLog://Security]
disabled = 0

(I've also tried to add "index = main"
on bottom of above stanza with no
results).

Are you overriding in the local directory? The local version will always override as per Configuration file precedence

I would recommend you run:
splunk btool inputs list --debug

To determine what index= setting is been applied to the stanza you mention on the forwarder, the above command should be run on a forwarder with the issue. Splunk btool is documented here

[morganfw]

No override, local directory is empty.

[jwelch_splunk]

I think there is an echo in here..... ( =

[gjanders]

You posted 2 minutes after me 😛
Nice comment by the way 🙂

[jwelch_splunk]

According to my screen you were two minutes after me..... And since you are not my wife....I know I am right. haha.

My bet is he has

Splunk_TA_windows from the new install of the UF, and that would over-ride the app of
domain_controller because a Capital S takes a higher priority on an app than a lowercase d.

We shall see.

Okie

[jwelch_splunk]

Not trying to be difficult here, but I am having an issue following you.

do this command on the UF that had 5.0.4 Uninstalled and then the new 6.4.3 installed

splunk btool inputs list --debug |findstr wineventlog > support.txt

Paste me what is in that file

Okie

[morganfw]

Ok I did the trick!
Forwarder Upgrade with no custom options create Splunk_TA_windows that override my "domain_controller" app under UF path "$SPLUNK_HOME\etc\apps", I ran btool command and output following result:

C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf index = wineventlog
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf index = wineventlog
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf index = wineventlog

so I've uninstalled UF and reinstalled with "Customize Options" and uncheck all flags under "Windows Event Logs", after installation I've checked that "Splunk_TA_windows" directory that not exists under "$SPLUNK_HOME\etc\apps" then Start SplunkUniversalForwarder Service, Deployment Server pushed configurations to Deployment Client and now I receive logs.

Thx all for support!!!

[morganfw]

Ok I'll try to execute above command and paste output.