Splunk App Windows Infrastructure upgrade - No "sourcetype="MSAD*" found

[linusHillyard]

I've recently upgraded to Splunk App for Windows Infrastructure 1.1.1 from version 1.0.4. Previously I had no issues with Active Directory data detection or Splunk App for Active Directory(SA-ldapsearch) version 2.0.1 and can still successfully search for queries like '|ldapsearch domain=DOMAIN search="(cn=Administrator)"'). Since the upgrade, when I run through the first-time setup wizard I get the error, "ERROR: Search "sourcetype="MSAD*" | head 5" did not return any events in the last 24 hours" when checking for data being provided by the environment.

I'm then provided with a link named, "Splunk Add-on for Microsoft Windows Active Directory for Splunk Universal Forwarder" however the link takes me to setup instruction for the Windows Infrastructure App. Since I'm still able to perform ldapsearch queries from the search app I'd assume the Splunk App for Active Directory is working correctly.

Also, when viewing the upgrade instruction(http://docs.splunk.com/Documentation/MSApp/1.1.1/MSInfra/UpgradetheSplunkAppforWindowsInfrastructure) you're instructed to download 'Splunk Supporting Add-on for Active Directory version 2.0.2 or later' however version 2.0.1 is the latest version I can currently find for download.

I'd appreciate any insight as I've hit a wall and cannot proceed with the upgraded version of the Windows Infrastructure app.

[dolejh76]

We had a similar issue - did you make sure that msad where default search indexes?

[dolejh76]

Settings > Access Controls > Roles > winfra-admin > Scroll down to "Indexes searched by default" and add the appropriate indexes. (msad in this case)

John

[schultet]

Thanks! That's what I was missing. Setup is progressing past the msad error in the Check Data step now.

[schultet]

where do I check that setting?

[malmoore]

Settings > Roles > "Winfra-admin" role

[malmoore · Accepted solution]

Hi,

Have you made sure that the user that you log into Splunk Enterprise with has the 'winfra-admin' role? That role lets you search the proper default indexes that come with the app.

The Splunk Supporting Add-on for Active Directory is currently at version 2.0.1. The reference to 2.0.2 has been corrected. Apologies for any confusion.

[linusHillyard]

Yes, sorry. I should've stated the Prerequisites checks are all successful(green check marks). Also, I'm seeing no errors on the forwarders regarding the PS scripts used to collect AD info by the TA-DomainController-NT6 add on. Again, this application wias working correctly with the previous version Splunk App Windows Inf 1.0.4.

[IRF2233]

I did the below mentioned but this didn't help me. Please recommend any other solution for this!

Settings > Access Controls > Roles > winfra-admin > Scroll down to "Indexes searched by default" and add the appropriate indexes. (msad in this case)

[schultet]

I have all the apps installed in the correct directory on my SH (a single server used for Splunk) E:\Program Files\Splunk\etc\apps

on the SH, I have the SA-ldapsearch in the etc\apps directory. LDAP is configured with a domain admin user currently. I performed a test connection and it was successful.

I do have the same APPs installed on both Domain Controllers and one is forwarding events

Splunk_TA_windows
SplunkUniversalForwarder
TA-DNSServer-NT6
TA-DomainCOntrollerNT6

Both DCs are 64bit - one is 2012r2 the other DC that is not forwarding events is 2008R2

[schultet]

If you run 2012R2

Did you mean on the SH or the DCs or both?

I have neither on my DC's but the 2012 DC is forwarding while the 2008 does not appear to be forwarding events. How could I check if my second DC (2008r2) is forwarding any events at all?

[malmoore]

Only on the DCs.

If your 2008 DC is not forwarding events make sure:

• that you have a forwarder installed on it.
• That the forwarder has the TA-DomainController-NT6 installed in it.
• That the forwarder has been configured to send data to indexers.
• That there is a clean IP connection available from DC to indexer (no firewalls blocking ports)
• That you have changed AD audit policy to log the Additional AD events that the app needs.
• That the indexer has the appropriate indexes configured on it.
• That the user account on the SH has been configured to search those indexes.

A quick search with"index=msad host=Hostname-of-DC" is a great start. If you're not seeing data, walk back through this list. It's usually network connectivity or misconfiguration (and sometimes both.)

[barrydow]

OK, I'm still highly lost at this point. I see no TA-DomainController-NT6 available to add into my deployed apps. Nor do I see it in my list of available apps. I'm not at the console, so I might find it directly there if I can get back on the Linux console and look within the list of apps to find it, but thus far, I don't see where it is to find it from to be sure to get it deployed back to the Windows Domain Controllers.

[malmoore]

Hi, can you show what is in your $SPLUNK_HOME/etc/deployment_apps folder on your deployment server? Is the deployment serve running on your Linux host?

Log into Splunk on the deployment server. If you put the apps in the right directory you should see them in Forwarder Management.

[barrydow]

I definitely (now) have the Add-ons in the right path and see them below forwarder management. They are "deployed" to the Windows servers where I should be getting data from yet I am still getting the errors related to Search "sourcetype="ActiveDirectory*" | head 5" and same for sourcetype="MSAD*" | head 5

That is the same on both of the Splunk servers I'm working with (the Add-ons for Windows 2008 domain are in place, and yet we're not getting those events capture for the Windows Infrastructure App.

Next suggestion?

(And stupid question follow-up... The add-ons were originally kept below the Windows Infrastructure App. I copied them up and into the Deployment-Apps area, as I also did with the Infrastructure App. Should I not have the Infrastructure App there as an App for deployment? Should I have had just the Add-ons? Why don't the add-ons go along for the ride when the Infrastructure App is being deployed??)

[dolejh76]

So you place the TA-DomainController-NT6 folder and its contents in the $SPLUNK_HOME/etc/deployment-apps folder correct? I have not deployed any mew apps in a while but I think after that you have to run

$SPLUNK_HOME/bin/splunk reload deploy-server

[malmoore]

This is correct. you can also restart Splunk on the deployment server.

[schultet]

I have checked all of the above - I have found records with the search command you provided:

sourcetype = MSAD:NT6:Replication, MSAD:NT6:Health, MSAD:NT6:DNS-Zone-Information, MSAD:NT6:SiteInfo

I just don't see any of the AD user, password and unlock events coming from this DC.

Thanks for any help

[dolejh76]

Did you make the GPO changes to have these events logged on the DC? page 35 and 36 of the install guide...

[schultet]

Yes the GPO settings for monitoring are in place and the same for both the DCs - One is logging events and one is not.

I used GPUPDATE /force did a GPRESULT and it audit polices are applied to this DC. Maybe I need a reboot?

[dolejh76]

Changes are only reported on the DC that made the change. These are not replicated events. That is why you collect the events in Splunk into one central location. If you want to test the auditing make sure you are connected to a specific DC and make the change. Then test that it is being logged in the event viewer.