I have 4 domain controllers with Splunk Universal Forwarders installed on them. I'm trying to get the Windows Security logs and Cisco ASA logs sent to my Splunk Light server.
I get the ASA sys logs from all the forwarders except one and I get Windows Security logs from one of the forwards, but the other three I don't get them from.
Nothing makes sense. There are no firewall issues. All the domain controllers can ping one another. I don't have any of the Splunk ports blocked.
I checked the splunkd.txt log files and there are no errors. The inputs and outputs conf files are all set up exactly the same, but still only some forwarders send data while others don't.
I followed this article http://docs.splunk.com/Documentation/SplunkLight/6.4.1/GettingStarted/GettingdataintoSplunkLightusin... and still can't get every forwarded to communicate,
Under Forwarder Management -> Server Classes all of them are checking in, but there not all sending the data asked them to send.
Any help would be appreciated.
Possibly this Answers topic can help resolve your situation. I've also reached out to some others that might be able to offer some suggestions.
Unfortunately this does not help. I forgot to mention one more thing. I have set these forwarders up and they were receiving the security logs fine. But for testing purposes I uninstall and reinstall these forwarders and it never just comes back up and gives me the logs. I always have to do some troubleshooting and more troubleshooting to finally get them communicating again.
Then I will reinstall them and again I have to troubleshoot them to get the logs.
How about the CISO ASA logs. Why are some of them being forwarded just fine and others not. Is there something special I need to do.
We also installed WireShark and we can see that the forwarders are seeing the data and even sending the data. And yet at the Splunk server only some of the data is getting through. It just doesn't make any sense. It seems very random. We have a mixture of 2008 R2 and Windows 2012 servers and some send and receive without an issue while others don't.
We can either get the Security log or the Cisco ASA logs from every server, And some of them we get both. So all the servers are communicating fine. It does not make sense that we don't get both logs. Is there something on the Splunk Light server that could be blocking some of the packets from coming through?
Just checking back on this.
I've tried reinstalling the forwarder several times, I'm getting my Cisco ASA logs but I'm not getting the Windows Security event logs.
Any other suggestions?
So, to get this straight, you have:
Believe it or not, the fact that only one DC is sending WinEventLog::Security sort of makes sense. Unless these DCs all operate for different domains/forests, it makes sense to use only one DC to send AD-related logs to Splunk to reduce network chatter. But, we'll still work to resolve this anyway.
The ASA problem is probably the easier one to diagnose based on the limited info we have. But first I need some more info:
Give us whatever information you can. If you don't know, that's fine, we can find out more based on what you do know. Thanks for your assistance on sleuthing this.
Hello Malmoore and thank you for the response.
The DC's are a mix of 2008R2 and 2012R2
The ASA (Cisco Adaptive Security Appliance [Firewall]) is only able to send syslog messages to an IP on the inside interface. Therefore in 4 of the 5 sites the ASA directs the syslog messages to the local DC running Splunk Forwarder. Packet captures show the logs arrive and Netstat confirms Splunk.exe (forwarder) is listening on the port. This configuration is working for all servers running 2008R2. The one server running 2012R2 does not appear to forward the syslog events from the ASA (nor is it any longer sending Windows logs), however I suspect the problem lies with Splunk Light not logging them in the DB (ignoring them). The 2012R2 server was sending Windows logs at one time but these are no longer coming through.
We are only interested in the security logs on the DC's. Specifically we are interested to know when a user logs onto their local machine. Only the local DC logs this event so we need the events from all DC's.
DC's not sending Windows logs:
2 server running 2008R2 (both were sending logs at one time but stopped 1 week ago - no changes were implemented at that time - we are reinstalling forwarder again to try to bring it back online.)
1 server running 2012R2 (was sending logs at originally but stopped after a reinstall of the forwarder while troubleshooting ASA logs not coming through)
DC's not forwarding ASA logs
Same 2012R2 server no longer sending security logs. Splunk light has never registered the IP of the ASA as a source. The IP has appeared in the Hosts list automatically for the other firewalls. Packet capture shows the same syslog entries leaving the forwarder destined for the Splunk server.
Searching for DC's by hostname returns entries from the security logs from that host (results returned for all hosts - not all of them have recent events).
In troubleshooting we had tried to get all AD events to see if that would work and in many cases that allowed events to start flowing. As a result some servers are sending more information than desired. We attempted to change this under Home, Add Data, forward, Existing, (select server class for windows servers [all 5 DC's are listed as forwarders]), Next, Local Event Logs, Security, Review, Submit. However the Sources and Sourcetypes lists on the home page indicates last update for WinEventLog:Security 1/25. Additionally this seems to stop all Windows logs coming in including Security logs.
I suspect this to be a key part of the problem. Splunk seems to be ignoring logs due to some setting in the Add Data area or elsewhere. We have tried so many different combinations and are at a loss as to what may produce the expected result. We have run packet captures on one forwarder not showing Windows logs in Splunk and see what look like Windows event logs being sent from the forwarder yet not appearing in the DB.
Config from: C:\Program Files\SplunkUniversalForwarder\etc\system\local
targetUri = (IP of Splunk Server):8089
host = B-DC01
disabled = 0
defaultGroup = default-autolb-group
server = (Splunk IP):9997
serverName = B-DC01
pass4SymmKey = (redacted)
sslKeysfilePassword = (redacted)
description = autogeneratedpoolforwarder
quota = MAX
slaves = *
stack_id = forwarder
description = autogeneratedpoolfree
quota = MAX
slaves = *
stack_id = free
Well, it seems like you have done quite a bit of sleuthing on this.
When you say "we had tried to get all AD events" during the troubleshooting phase, did you mean changing the Group Policy on your DCs to collect all AD related events, or did you mean using the ADmon input? These are two separate things.
My apologies for not including a time constraint in your search for hosts previously. You can easily rectify that and see what data is coming in for each host currently by setting the time picker to a range that falls outside of where most events that have come in for that host, for example, "Last 4 hours" would be good as a time constraint for the host that has stopped sending ASA logs and EventLog data a while back.
Some additional questions:
There seems to be multiple issues in play here. A network failure on the indexer would result in no data coming in from any host. I do not believe that it is a problem with the indexing of data either, as a problem with indexing should result in failure to find data for all hosts, not just one.
"All AD events" refers to the a setting set on the splunk light web portal under "Add Data". I referenced the exact area in Splunk where we changed this in the previous post: Home, Add Data, forward, Existing, (select server class for windows servers [all 5 DC's are listed as forwarders]), Next, Local Event Logs, Security, Review, Submit. No idea what admon is. We have the logging policy on the DCs functioning correctly.
Time constraint was not a problem. I only mentioned it to provide as much information as possible.
Received event for unconfigured/disabled/deleted index=wineventlog with source="source::WinEventLog:Security" host="host::IC-DC01" sourcetype="sourcetype::WinEventLog:Security". So far received events from 1 missing index(es).
2/11/2017, 7:42:51 AM
I still feel the problem is in the configuration settings in the Add Data section of splunk. I have no documentation on how these settings should be. How does one configure within Splunk the data it should assimilate?
On Forwarder management all 5 forwarders appear and have phoned home within a minute. Yet no data now flows for windows logs - only firewall data. I suspect settings within the Server Class or Apps areaS.
Currently I have 2 Server Classes. "Firewall Log" and "Security Log". 4DC's are in Firewall Log and 5 are in Security Log. It seems like I only really need 1 class with multiple apps but the whole thing is a bit confusing.
I have 17 "Apps". These are automatically created when I attempt to add new data. Nor was I able to find a way to change/view settings in an app aside from steps mentioned in paragraph 1 of this post.
I searched app.conf and inputs.conf in deployment-apps folder for the 2 apps I am using. Only thing there is:
I need to know how to change which windows logs the forwarder is harvesting as well as resolving the cause for the indexer dumping the incoming entries rather than assimilating them.
Any further assistance will be much appreciated.
One other thing I noticed a lot of packets like this incoming from the forwarders in the capture:
pathDC:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.logMetaData:Index internalevtresolveadobj0
I solved the event log issues! Your comment about the indexes was the clue. Just now I ran the Add Data wizarad again and created a new index for the existing forwarders and all logs are now appearing.
Remaining issue: ASA logs (udp 1025 bouncing off the forwarder running on server2012r2) fail to appear. Under Hosts the IP for the source firewall is not listed.
Packet captures and netstat on the DC reveal that logs are arriving but forwarder is not listening. .conf files are identical to other DC's which are forwarding. I will try to reinstall the forwarder.
I think this is an issue with that server. The forwarder service is not appearing in the list of services yet the process for the service is running. I will focus on resolving that issue first. Thanks for your help.