Our splunk administrator jumped ship shortly after getting splunk set up, and there wasn't anybody that officially took over his role, thus, no real splunk experts here. I'm trying to fill in some gaps. 😞
We recently decommissioned a couple of domain controllers and replaced them with new ones, and then some reports stopped sending events related to accounts getting locked out, password resets, and users that weren't logging out at night. (there's probably more but these are the three that management is specifically complaining about)
I went into "Settings | Data Inputs | Active Directory monitoring" and added the names of the two new domain controllers - it was just running with "default" before. That didn't seem to fix the issue.
Where else in splunk would any pointers to domain controllers be configured that might affect the splunk server not getting these events?
This is splunk version 6.6.4, running on Windows.
(hopefully this question isn't too vague to work with 😞 )
Note to soutamo,
There was a deploymentclient.conf file, and it was pointing to the correct server:port#.
It didn't have a [deployment-client] stanza though, so I added that, per the document you linked to.
Still not feeling the love yet 😞
I saw this error at the top of the page this afternoon:
"the lookup table 'audit_host_inventory' does not exist it is referenced by configuration linux:audit"
There are a couple of Windows workstations that are sending event logs to splunk, but not the Linux workstations.
When I looked in Settings | Lookups | Automatic lookups, none of the Automatic Lookups have an owner - they all say "no owner". Is that normal?
Every Auto lookup IS set to "Global" for Sharing, though
Neither of the ../local folders had an outputs.conf file in it
The rule o& thumb: You should do all changes on ../local/outputs.conf file, never on default directory!
Do you know if you have deployment server in use? Any deploymentclient.conf on those or other servers? See https://docs.splunk.com/Documentation/Splunk/8.0.2/Updating/Configuredeploymentclients if needed. If those are n use then configurations must do with those otherwise via local files.
Ismo
I found 2 "outputs.conf" - one in "\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default"
and one in "\Program Files\SplunkUniversalForwarder\etc\system\default".
Is that first one the correct one to edit? (The 2nd one had a warning about "don't edit this file directly")
Neither one had a "server =" parameter set.
I added "server = :8089" to the first one - let's see if it starts generating some events now ....
Have you configured the new forwarders on the domain controllers with outputs.conf? Or, if you're using a Deployment Server, did you configure the forwarders to connect to it? (It can get outputs that way, assuming you're using a DS and it has an outputs app)
One other thing that is probably relevant - the domain controllers that were replaced were also the DNS servers. Possibly the issue is being caused by different IP addresses for DNS, rather than the domain controllers per se?
The events stopped coming in right around the time the DNS/domain changes were made.