Why am I getting error "Received event for unconfigured/disabled/deleted index="wineventlog"" when the index does exist?

[Epicism1]

Hello,

I am trying to log the Sysmon/Operational Windows event logs via the Sysmon TA app:

[WinEventLog://Microsoft-Windows-Sysmon/Operational]
disabled = false
renderXml = true
index=wineventlog

But when I push the app to the Universal Forwarders on my Windows boxes, I am receiving the error:

Received event for unconfigured/disabled/deleted index="wineventlog" with source="source::WinEventLog:Microsoft-Windows-Sysmon/Operational" host="host::XX" sourcetype="sourcetype::XmlWinEventLog:Microsoft-Windows-Sysmon/Operational". So far received events from 1 missing index(es).

The challenge is that the index does exist and is enabled:

It was originally created by the Windows TA app, so I deleted it, recreated it, put it in indexes.conf, but nothing. I cannot see the issue.

Any help would be appreciated.

[Epicism1 · Accepted solution]

So I figured out what the issue is. I set the index in inputs.conf which made the hard-coded checksum fail and refuse to install

The universal forwarder splunkd.log has the error:

02-06-2016 05:15:54.061 -0500 INFO  DeployedApplication - Checksum mismatch 0 <> 7566232508823169641 for app=TA-microsoft-sysmon. Will reload from='splunk.projectmayhem.local:8089/services/streams/deployment?name=default:Windows%20Event%20Logs:TA-microsoft-sysmon'
02-06-2016 05:15:54.076 -0500 INFO  DeployedApplication - Downloaded url=splunk.projectmayhem.local:8089/services/streams/deployment?name=default:Windows%20Event%20Logs:TA-microsoft-sysmon to file='C:\Program Files\SplunkUniversalForwarder\var\run\Windows Event Logs\TA-microsoft-sysmon-1454471855.bundle' sizeKB=30
02-06-2016 05:15:54.076 -0500 WARN  DeployedApplication - app=TA-microsoft-sysmon, installed_via="search head cluster deployer, UI, CLI, or REST API", checksum=0b39270c03e818fb6bcadcf51781171cc69e07ce
02-06-2016 05:15:54.076 -0500 WARN  DeployedApplication - app=TA-microsoft-sysmon was already installed via search head cluster deployer, UI, CLI, or REST API; it may not be overridden via deployment server; remove existing app=TA-microsoft-sysmon via search head cluster deployer, UI, CLI, or REST API if you wish to install it via deployment server
02-06-2016 05:15:54.076 -0500 ERROR DeployedServerclass - name=Windows Event Logs Failed to install app=TA-microsoft-sysmon

To fix it I removed the checksum in the app.conf file.

Thanks for your help.

[s2_splunk]

Why that would cause the message you saw on the indexer I am not sure, but I am glad you figured it out!

[Epicism1]

Yeah, that part is weird because it seems to me like it shouldn't be attempting to send any logs to the indexer. Weird.

[bandit]

If you haven't already done so, I would try restarting your indexers to ensure they are initializing the indexes.conf config updates.

[s2_splunk]

Can you run sudo ./splunk cmd btool indexes list --debug | grep -A 10 "\[wineventlog" on your indexer and share the output, please?

[Epicism1]

/opt/splunk/etc/system/local/indexes.conf                    [wineventlog]
/opt/splunk/etc/system/default/indexes.conf                  assureUTF8 = false
/opt/splunk/etc/system/default/indexes.conf                  bucketRebuildMemoryHint = auto
/opt/splunk/etc/system/local/indexes.conf                    coldPath = $SPLUNK_DB/wineventlog/colddb
/opt/splunk/etc/system/default/indexes.conf                  coldPath.maxDataSizeMB = 0
/opt/splunk/etc/system/default/indexes.conf                  coldToFrozenDir =
/opt/splunk/etc/system/default/indexes.conf                  coldToFrozenScript =
/opt/splunk/etc/system/default/indexes.conf                  compressRawdata = true
/opt/splunk/etc/system/default/indexes.conf                  defaultDatabase = main
/opt/splunk/etc/system/default/indexes.conf                  enableDataIntegrityControl = false
/opt/splunk/etc/system/default/indexes.conf                  enableOnlineBucketRepair = true

[s2_splunk]

Hmmm, that is weird. When you run a search on the indexer using index=wineventlog, do you see data? Your screenshot suggests there should be events in it.
The only other thing I can think of is some weird non-printable control character in your inputs.conf.

[Epicism1]

Yes, security and application event logs are streaming in no problem. I'll try running dos2unix on the inputs.conf file...

[s2_splunk]

I think that is a great idea... Please report back.