Why is no data being indexed after deploying the Splunk Add-on for Microsoft Windows and "send to indexer" apps?

[flynnhandley]

I have deployed the Splunk Add-on for Microsoft Windows and the send to indexer app successfully to my forwarders.

I have a server called activedirectory which I can see has downloaded the apps, however, no data is being indexed. Actually, I can't see data from any of the servers that have the app deployed.

I can see on the splunk server that activedirectory has made a connection on 9997.

if I search for host=activedirectory there are no events

Any ideas where I should start troubleshooting?

Data Summary

Host Count Last Update
SERVER1 135 3/24/15 2:30:30.000 PM
splunk 1 3/24/15 12:45:33.000 PM

C:\Program Files\Splunk\etc\deployment-apps\Splunk_TA_windows\local

OS Logs

[WinEventLog://Application]
disabled = 0
start_from = oldest
current_only = 0
checkpointInterval = 5
index = wineventlog
renderXml=false

[WinEventLog://Security]
disabled = 0
start_from = oldest
current_only = 0
evt_resolve_ad_obj = 1
checkpointInterval = 5
blacklist1 = EventCode="4662" Message="Object Type:\s+(?!groupPolicyContainer)"
blacklist2 = EventCode="566" Message="Object Type:\s+(?!groupPolicyContainer)"
index = wineventlog
renderXml=false

[WinEventLog://System]
disabled = 1
start_from = oldest
current_only = 0
checkpointInterval = 5
index = wineventlog
renderXml=false

C:\Program Files\Splunk\etc\deployment-apps\sendtoindexer\local

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = splunk:9997

[tcpout-server://splunk:9997]

[alemarzu]

Hi Flynn,

• Have you installed the Supporting Add-on for Active Directory (LDAP-search) ?
  • Take a look in here for any error message, $SPLUNK_HOME/var/log/splunk/SA-ldapsearch.log
  • Add-on Docs (troubleshooting): http://docs.splunk.com/Documentation/SA-LdapSearch/2.1.2/User/UseSA-ldapsearchtotroubleshootproblems
• Did you map "winfra-admin" role to your user account ?
• Did you deploy "TA-DomainController-NTx" Add-on on your Domain Controller based on OS version ?

You should read this http://docs.splunk.com/Documentation/MSApp/1.2.0/MSInfra/TroubleshoottheSplunkAppforWindowsInfrastru...

Hope it helps.

[uktechnologyser]

How did you see the deployment client made a connection on 9997? Did you get a fix for this? I have the same problem.

[hcanivel]

I have a few troubleshooting questions for you:

a) What do your splunkd logs say for both forwarder and indexer? Are there any errors at the forwarder? Maybe the user running Splunk doesn't have winevent log read access?

b) on indexer side, you're writing to index wineventlog. How do you know that it isn't indexing? Have you tried specified the index as part of your query?

c) how do you know that your host is activedirectory and not activedirectory?

Always start with the splunkd logs for any basic errors.

[uktechnologyser]

a) My splunkd log on the deployment client says:

ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.

Does this mean it isnt referencing the outputs.conf file in the send to indexer app? This has the configuration telling it to tcp out to the indexers.

b) I know it isnt indexing because i cant see any files populating in the wineventlog folder on the indexers.

c) my host doesnt appear,

Interestingly if i install the Splunk_TA_windows app directly on the indexers it starts getting logs straight away and storing them locally. From there i can search them from the search head. It must be a problem with the send to indexer app.