Hi there! I've spent about 8 hours trying to get Splunk working and I'm at the end of my rope. The server was easy enough to set up; I spun up a new CentOS vm, installed and configured the Splunk server (I'm able to log into the web interface and install apps) but what's killing me is installing the Universal Forwarder. I'm trying to install it on my "utility" Windows 2008 R2 vm that hosts all my apps like HP System Insight, Spiceworks, Netwrix, etc. I want Splunk's Universal Forwarder to grab remote Windows data and provide AD Monitoring.
I created an AD user called Splunk and followed the directions here under "Prepare Active Directory for Splunk installation as a domain user" to create & configure AD groups: http://docs.splunk.com/Documentation/Splunk/5.0.5/Installation/PrepareyourWindowsnetworkforaSplunkin...
No matter what I do to install the Universal Forwarder I get the error "Splunk Installer was unable to start Splunk services. Please make sure you have provided the correct username and/or password, and the user you are trying to run Splunk as has the correct priviliges. Exitcode='4'." I can install it for local monitoring without issue.
What the heck am I doing wrong? Neither the log in appdata - local - temp nor the log in var - splunk - log provide any clues. And everyone else here who has posted the question has found the most bizarre solutions, or apparently, no solution at all, which seems rather strange. Thanks for any advice or pointers!!!
No sweat. They make it sound scary, but it should go pretty smooth. The biggest problem is folks start enabling all of the logging on the DC, and end up bogging down both the DC and the indexer.
From the log it looks like you did not specify a destination indexer IP with port 9997.
In the service manager you should verify that the user name and password are correct. You should use the domain\username format for the username.
I tried installing the Universal Forwarder onto another server and got the same error but with an exitcode of 2. The log says this:
Splunk> Needle. Haystack. Found.
Checking mgmt port : open
Checking conf files for typos... Done
All preliminary checks passed.
Starting splunk server daemon (splunkd)...
SplunkForwarder: Starting (pid 2928)
Timed out waiting for splunkd to start.
Removing service SplunkForwarder
What could this mean?
You are trying to collect AD data remotely with a universal forwarder? If so, then you should rethink it, because it does not work that way.
To monitor AD, at minimum the forwarder needs to be on the domain controller and it needs to send data to an indexer either directly or through a heavy forwarder.