Splunk 6.2, indexer is a Debian 7.7 64 VM, universal forwarder is installed on a Windows 2008 R2 64 VM. I've just finished configuring Splunk App for Windows Infrastructure following the manual to the letter (includes Windows Add-on and Splunk Supportint Add-on for Active Directory, AKA ta_ldapsearch). I prepared my AD according to the manual, installed the apps, have the forwarder starting automatically as a service in my AD box, deployed with my deployment server both Splunk_TA_Windows and the add-ons needed for Infrastructure (ta-dnsserver-nt6 and ta-domaincontroller-nt6). I can see the data coming in, I can run searches but the app dashboards do not show anything, they either say "no results found" or the drop-down boxes do not get filled and the dashboards stay "waiting for input" forever. If I try to configure the app via "Tools and Settings" -> "App Configuration" and let it detect ("Detect" button) it only sellects "Group Policy" and "Organizational Units", and shows nothing in the dashboards. If I force it (select manually all boxes) it does show some summary info but still does not show anything in the dashboards. I have only a custom
inputs.conf file for Windows Add-on with all sources enabled, but nothing else. Ldapsearch add-on seems to be working correctly since it can connect to my AD using the "test" button in the app's configuration page.
I am not sure where should I start looking for clues, but I noticed the searches for the dashboards are not working with specific field values (like
eventtype=powershell), I've checked this listing and opening the objects of the app and copied/pasted some of the searches into a search windows; nevertheless, if a precede the searches with
eventtype=* they work flawlessly for all fields that have already indexed data, which is really odd. The format I MUST use all times in order to get anything is:
eventtype=* eventtype=powershell sourcetype="blah blah blah..." | stats (more blah blah blah)
The data surely is coming in, since today we reached our daily limit for index data (500 MB trial license) again and the main data is coming from the AD server and going to the indexes of the related apps, I've checked it in "license usage" and in the date for the newest events in the indexes.
Any ideas? I was trying to help a client that has the exact same issue, only for him it is an enterprise environment, mine is just a lab. I thought I would be lucky, install everything and then just show up the answer to the client, but it seems there is more to it than meets the eye, sort of. I've already double-checked the app documentation to see if I didn't skip any important steps, but it seems it has all been done by the book, so where is the catch? The client is not happy and Splunk is not looking good for him, I would like to avoid a negative review, even if they do not buy the product it would be nice if they do not spread a bad word about it, coz they are BIG, nay HUGE and they get HEARD around here...
I had an issue with the lookup generation apparently failing. It was failing because at one point I had removed the default admin account. I've run into similar issues in the past and have since added the account rather than change the owner on all the default reports, etc.
Came across this after pulling my beard out - followed the stock instructions and data seemed to be indexed. However the detection and dashboards drew blanks.
The key for me was "index=*" - that clued me in. It was just the indexes that were being searched by default.
Settings > Access Controls > Roles > Admin > Indexes searched by default (I added msad, permon and winevents)
The detection was then much more successful and I'm seeing data in the dashboards. Some tweaking required, but still!
Thanks for your answer. I've included the indexes in admin role and restarted Splunk (just in case) but it did not change anything 😞
Basically, some lookup tables are not being populated, don't have a clue why is this happening. Detection always leaves updates (windows) and computers/groups/users (domain) dashboards unchecked. Nevertheless, if I check it manually, I can see the dashboards and some data in most of it (for instance, updates show all updates applied since the VM was deployed, plus software installed - it is not detected by the app but it has all data needed, weird stuff I've seen in 20 years of IT so I am not surprised). Other than that, some performance data is not coming in since last Friday, namely, disk data, so the dashboards are empty. I am not sure what should I do next, but it would be nice to know where should I look for clues. I can't just start telling the client to downgrade stuff if this does not solve the issue. And I would really like to know why some data stopped coming in from the forwarder after last Friday, but the logs are not helpful.
Make sure the user you are using has permissions to the indices. We have added roles which have access to the appropriate indices. You need to add the admin user plus other users (whoever uses the app) to these roles.
Other than that, it's a basic diagnostics process - click on the "open in search" button and see what you can see. Check the search inspector and see if any events are being processed.
Thanks for your reply. I've added the roles windows-admin and winfra-admin to user admin (the one I am employing during these tests). It solved partially the issue, some dashboards are getting populated now, others remain empty. Should there be other roles or these 2 are the ones I should select?
I've also downgraded the app from ver. 2 to ver. 1.1.13 as instructed but did not see any improvement. I am configuring the app via ldap.conf file, as this version has no GUI for configuration, copied the old config I saved in another directory from ver. 2 before uninstalling it to the local folder of ver. 1.1.13, seems to be working but as I said, only half of the dashboards get populated. More than that, now when I try to configure Windows Infrastructure app via the "Detect" button, almost all boxes get selected but "Windows -> Applications and Updates" and "Active Directory -> Users/Computers/Groups". Nevertheless, the dashboards related to disk/memory/cpu, for instance, are still empty. Where else should I look for clues?
The "for instance" worries me - we are not getting a complete picture of what is wrong.
1) Is the data coming in? Check Splunk_TA_Windows that you are deploying to your endpoints to ensure that you are collecting the disk, memory and CPU data
2) Are the lookups being built? Check the lookups directory for CSV files - are they populated?
These are probably the two major causes for concern.
More frustration. I have ldapsearch downgraded to 1.1.13 as per instructions, but there are Java errors for some searches from the dashboards. Java is version 1.8, when I run the search from "Org Units:All" dashboard:
I get this:
ERROR: java.lang.NullPointerException: null External search command 'ldapsearch' returned error code 1.
Which info do you need, pray? Not that I haven't posted a truckload of info here, but if you need more details just ask. I am about to downgrade Java 1.8 to 1.7, would be rather demoralizing to see another failed downgrade. Should I wait?
Found another thing. I've run a search with the values of "Disk Information" dashboard, and it seems there is something wrong with the fields the dashboard is looking for. The search starts with:
eventtype=windows_hostmon Type=Disk ...
The search does not find any results. If I replace the
Type=* then it returns results, but none of
Type=Disk. What I have for
Information Not persistent Persistent Boot-time Warning Error Site SiteLink Computer Value Added Active Directory Domain Services 0x0000000A System.String
Another example is "Host Monitoring" dashboard, the search from the app is as follow:
eventtype="windows_hostmon" (Type=Computer OR Type=OperatingSystem)| stats latest(OS) as OS, latest(Domain) as Domain, latest(Architecture) as Architecture by host | search (host="*") host="*" OS="*" Domain="*" Architecture="*"
If I get rid of the second search after the pipe, I get values for host and Domain, but not for OS and Architecture. If I leave it like that, no values are returned. This is most frustrating, to tell you the truth.
Your comments indicate that data is missing. We can't tell what data is missing. Based on the comments, at least the WinHostMon data is missing. Check out the documentation to the WinHostMon data input and add that to the TA. Docs link: http://docs.splunk.com/Documentation/Splunk/6.1.4/Data/MonitorWindowshostinformation
Other than that, it looks like some of the perfmon you are looking for is not getting gathered. Check out the documentation to the Perfmon data input and add what you need to the TA. Docs link: http://docs.splunk.com/Documentation/Splunk/6.1.4/Data/Real-timeWindowsperformancemonitoring
I've run a search with the following parameters:
index=* eventtype=windows_performance collection=LogicalDisk OR collection=PhysicalDisk
... and I can see fields named as
% Idle Time and
Disk Writes/sec with their respective values. But then the last event is from Friday the 31st of October, no more info has been collected since last Friday. If I run the dashboards with "all time" as time range, I can get them all to show values, but for system, disk (logical and physical), memory and process metrics the last values are all from last Friday, 5.50 PM approximately. Same goes for updates that are not detected by the app, but - and this is a big "but" - the computers/users/groups show up the values correctly, although these dashboards are not detected by the app, I have to check their boxes manually in "Tools and Settings", so it is a complete mess. Where should I check to see why the forwarder stopped sending this data?
As for the CSV files, I've checked all inside the lookups directory within the TA Windows app directory and most are populated, but some seem to be empty:
fs_notification_change_type.csv msad_group_type.csv object_category.csv status.csv user_types.csv
The lookups are being built only for the dashboards that the app can find data for. What can be the possible cause and where should I look for clues?
Thanks for your help so far.
Also, there is a known issue with SA-ldapsearch 2.0 right now - your comments suggest this is the version you are running. I'd recommend downgrading to SA-ldapsearch 1.1.13 until v2.0.1 comes out when you are using SA-ldapsearch in conjunction with Windows Infrastructure.