Getting Data In

Why does migrating universal forwarder to 6.4.3 display "unconfigured/disabled/deleted index='wineventlog'" message and event logs stopped forwarding?

morganfw
Path Finder

Hi all,
I've 3 Splunk 6.4.1 Indexers and a Splunk 6.4.1 Search Head + Distributed Management Console (DMC) on Linux Red Hat 6.6.
I've tested Windows Event Log in Windows 2008 R2 Domain Controller Servers in Preproduction environment with 1 Splunk 6.4.1 Indexer + Search Head + DMC, forwarded Event Logs all ok.

I've migrated Splunk Universal Forwarder (SUF) in Production from 5.0.2 to 6.4.3 with clean Installation (Uninstall SUF 5.0.2, Reboot Server and Reinstall SUF 6.4.3), and before with SUF 5.0.2 Windows Events was forwarded with no problem, after SUF Clean Upgrade to 6.4.3 I receive once following message:

Received event for unconfigured/disabled/deleted index='wineventlog' with source='source::WinEventLog:Security' host='host::my-host' sourcetype='sourcetype::WinEventLog:System' (1 missing total)

and Event Logs stopped to be forwarded.
I haven't changed configuration on my Indexers and Search Head, below my configuration:

  • $SPLUNK_HOME/etc/system/local/serverclass.conf
  • List item

[serverclass:domain_controller]
host = my-dc-host
[serverclass:domain_controller:app:domain_controller]
* $SPLUNK_HOME/etc/deployment-apps/domain_controller/default/inputs.conf
[WinEventLog://Security]
disabled = 0
(I've also tried to add "index = main" on bottom of above stanza with no results).

And other configurations to send logs globally from deployment clients to deployment server...

I've tried to uninstall and reinstall SUF 6.4.3, but no issue resolved, I've also read all Splunk Answers on same problem, but before SUF upgrade Windows Event Logs was Forwarded with no problem, and in Preproduction all works fine.

Any suggestion?
Regards.

0 Karma
1 Solution

jwelch_splunk
Splunk Employee
Splunk Employee

After working thru issue.

Splunk_TA_windows from the new install of the UF, and that would over-ride the app of
domain_controller because a Capital S takes a higher priority on an app than a lowercase d.

Okie

View solution in original post

jwelch_splunk
Splunk Employee
Splunk Employee

After working thru issue.

Splunk_TA_windows from the new install of the UF, and that would over-ride the app of
domain_controller because a Capital S takes a higher priority on an app than a lowercase d.

Okie

jwelch_splunk
Splunk Employee
Splunk Employee

It appears that your events are going to a new index called wineventlog, perhaps on the older system they were going to winevents.

The solution here is to create a new index called wineventlog, or use your inputs.conf to direct the data back to the old existing index name.

I assume this is a result of the Windows TA upgrade.

Okie

0 Karma

morganfw
Path Finder

I know that events going to new index that not exists, but my trouble is another.

I've plain Splunk Enterprise Core, no TA, my issue is particular because before with SUF 5.0.2 to Indexer 6.4.1 Windows Event Log was Forwarded, after "clean" upgrade to SUF 6.4.3 Forwarder stop to send Event Log to Indexer 6.4.1.

Before SUF upgrade Event Logs were going to "main" index with no explicit declaration in inputs.conf, so I've tried to configure explicitly the previous used index in inputs.conf and reload serverclasses with no success, and I don't need to create new index because prior upgrade this worked.

In Preproduction Environment all works fine and SUF forward events log to Indexer.

I hope I explained myself.

0 Karma

gjanders
SplunkTrust
SplunkTrust

With this point

$SPLUNK_HOME/etc/deployment-apps/domain_controller/default/inputs.conf

[WinEventLog://Security]
disabled = 0

(I've also tried to add "index = main"
on bottom of above stanza with no
results).

Are you overriding in the local directory? The local version will always override as per Configuration file precedence

I would recommend you run:
splunk btool inputs list --debug

To determine what index= setting is been applied to the stanza you mention on the forwarder, the above command should be run on a forwarder with the issue. Splunk btool is documented here

0 Karma

morganfw
Path Finder

No override, local directory is empty.

0 Karma

jwelch_splunk
Splunk Employee
Splunk Employee

I think there is an echo in here..... ( =

0 Karma

gjanders
SplunkTrust
SplunkTrust

You posted 2 minutes after me 😛
Nice comment by the way 🙂

0 Karma

jwelch_splunk
Splunk Employee
Splunk Employee

According to my screen you were two minutes after me..... And since you are not my wife....I know I am right. haha.

My bet is he has

Splunk_TA_windows from the new install of the UF, and that would over-ride the app of
domain_controller because a Capital S takes a higher priority on an app than a lowercase d.

We shall see.

Okie

0 Karma

jwelch_splunk
Splunk Employee
Splunk Employee

Not trying to be difficult here, but I am having an issue following you.

do this command on the UF that had 5.0.4 Uninstalled and then the new 6.4.3 installed

splunk btool inputs list --debug |findstr wineventlog > support.txt

Paste me what is in that file

Okie

0 Karma

morganfw
Path Finder

Ok I did the trick!
Forwarder Upgrade with no custom options create Splunk_TA_windows that override my "domain_controller" app under UF path "$SPLUNK_HOME\etc\apps", I ran btool command and output following result:

C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf index = wineventlog
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf index = wineventlog
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf index = wineventlog

so I've uninstalled UF and reinstalled with "Customize Options" and uncheck all flags under "Windows Event Logs", after installation I've checked that "Splunk_TA_windows" directory that not exists under "$SPLUNK_HOME\etc\apps" then Start SplunkUniversalForwarder Service, Deployment Server pushed configurations to Deployment Client and now I receive logs.

Thx all for support!!!

0 Karma

morganfw
Path Finder

Ok I'll try to execute above command and paste output.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...