Getting Data In
Highlighted

Why are universal forwarders installed on domain controllers not sending all Windows security and Cisco ASA logs?

New Member

I have 4 domain controllers with Splunk Universal Forwarders installed on them. I'm trying to get the Windows Security logs and Cisco ASA logs sent to my Splunk Light server.

I get the ASA sys logs from all the forwarders except one and I get Windows Security logs from one of the forwards, but the other three I don't get them from.

Nothing makes sense. There are no firewall issues. All the domain controllers can ping one another. I don't have any of the Splunk ports blocked.

I checked the splunkd.txt log files and there are no errors. The inputs and outputs conf files are all set up exactly the same, but still only some forwarders send data while others don't.

I followed this article http://docs.splunk.com/Documentation/SplunkLight/6.4.1/GettingStarted/GettingdataintoSplunkLightusin... and still can't get every forwarded to communicate,

Under Forwarder Management -> Server Classes all of them are checking in, but there not all sending the data asked them to send.

Any help would be appreciated.

0 Karma
Highlighted

Re: Why are universal forwarders installed on domain controllers not sending all Windows security and Cisco ASA logs?

Splunk Employee
Splunk Employee

Possibly this Answers topic can help resolve your situation. I've also reached out to some others that might be able to offer some suggestions.
https://answers.splunk.com/answers/455323/splunk-unable-to-fetch-windows-security-eventlogs.html

0 Karma
Highlighted

Re: Why are universal forwarders installed on domain controllers not sending all Windows security and Cisco ASA logs?

New Member

Unfortunately this does not help. I forgot to mention one more thing. I have set these forwarders up and they were receiving the security logs fine. But for testing purposes I uninstall and reinstall these forwarders and it never just comes back up and gives me the logs. I always have to do some troubleshooting and more troubleshooting to finally get them communicating again.

Then I will reinstall them and again I have to troubleshoot them to get the logs.

How about the CISO ASA logs. Why are some of them being forwarded just fine and others not. Is there something special I need to do.

We also installed WireShark and we can see that the forwarders are seeing the data and even sending the data. And yet at the Splunk server only some of the data is getting through. It just doesn't make any sense. It seems very random. We have a mixture of 2008 R2 and Windows 2012 servers and some send and receive without an issue while others don't.

We can either get the Security log or the Cisco ASA logs from every server, And some of them we get both. So all the servers are communicating fine. It does not make sense that we don't get both logs. Is there something on the Splunk Light server that could be blocking some of the packets from coming through?

0 Karma
Highlighted

Re: Why are universal forwarders installed on domain controllers not sending all Windows security and Cisco ASA logs?

New Member

Just checking back on this.

I've tried reinstalling the forwarder several times, I'm getting my Cisco ASA logs but I'm not getting the Windows Security event logs.

Any other suggestions?

0 Karma
Highlighted

Re: Why are universal forwarders installed on domain controllers not sending all Windows security and Cisco ASA logs?

Splunk Employee
Splunk Employee

Hi,

So, to get this straight, you have:

  • 4 DCs that host Cisco ASA logs
  • Universal forwarder on each DC
  • 3 of 4 DCs send ASA logs
  • 1 of 4 DCs sends WinEventLog::Security
  • Firewall not an issue
  • Universal forwarder configuration on each DC not an issue

Believe it or not, the fact that only one DC is sending WinEventLog::Security sort of makes sense. Unless these DCs all operate for different domains/forests, it makes sense to use only one DC to send AD-related logs to Splunk to reduce network chatter. But, we'll still work to resolve this anyway.

The ASA problem is probably the easier one to diagnose based on the limited info we have. But first I need some more info:

  • What version of Windows Server?
  • Where are the ASA logs hosted on these machines?
  • Which DC is NOT sending the ASA logs?
  • Which DC is sending the WinEventLog::Security logs?
  • Can you do a search of all the DCs by host name in Splunk Light? What results return?
  • Can you provide the universal forwarder configuration files on the DCs that ARE sending data?

Give us whatever information you can. If you don't know, that's fine, we can find out more based on what you do know. Thanks for your assistance on sleuthing this.

0 Karma
Highlighted

Re: Why are universal forwarders installed on domain controllers not sending all Windows security and Cisco ASA logs?

New Member

Hello Malmoore and thank you for the response.

The DC's are a mix of 2008R2 and 2012R2

The ASA (Cisco Adaptive Security Appliance [Firewall]) is only able to send syslog messages to an IP on the inside interface. Therefore in 4 of the 5 sites the ASA directs the syslog messages to the local DC running Splunk Forwarder. Packet captures show the logs arrive and Netstat confirms Splunk.exe (forwarder) is listening on the port. This configuration is working for all servers running 2008R2. The one server running 2012R2 does not appear to forward the syslog events from the ASA (nor is it any longer sending Windows logs), however I suspect the problem lies with Splunk Light not logging them in the DB (ignoring them). The 2012R2 server was sending Windows logs at one time but these are no longer coming through.

We are only interested in the security logs on the DC's. Specifically we are interested to know when a user logs onto their local machine. Only the local DC logs this event so we need the events from all DC's.

DC's not sending Windows logs:

2 server running 2008R2 (both were sending logs at one time but stopped 1 week ago - no changes were implemented at that time - we are reinstalling forwarder again to try to bring it back online.)
1 server running 2012R2 (was sending logs at originally but stopped after a reinstall of the forwarder while troubleshooting ASA logs not coming through)

DC's not forwarding ASA logs
Same 2012R2 server no longer sending security logs. Splunk light has never registered the IP of the ASA as a source. The IP has appeared in the Hosts list automatically for the other firewalls. Packet capture shows the same syslog entries leaving the forwarder destined for the Splunk server.

Searching for DC's by hostname returns entries from the security logs from that host (results returned for all hosts - not all of them have recent events).

In troubleshooting we had tried to get all AD events to see if that would work and in many cases that allowed events to start flowing. As a result some servers are sending more information than desired. We attempted to change this under Home, Add Data, forward, Existing, (select server class for windows servers [all 5 DC's are listed as forwarders]), Next, Local Event Logs, Security, Review, Submit. However the Sources and Sourcetypes lists on the home page indicates last update for WinEventLog:Security 1/25. Additionally this seems to stop all Windows logs coming in including Security logs.

I suspect this to be a key part of the problem. Splunk seems to be ignoring logs due to some setting in the Add Data area or elsewhere. We have tried so many different combinations and are at a loss as to what may produce the expected result. We have run packet captures on one forwarder not showing Windows logs in Splunk and see what look like Windows event logs being sent from the forwarder yet not appearing in the DB.

Config from: C:\Program Files\SplunkUniversalForwarder\etc\system\local

deploymentclient

[target-broker:deploymentServer]
targetUri = (IP of Splunk Server):8089

inputs

[default]
host = B-DC01

[script://$SPLUNK_HOME\bin\scripts\splunk-wmi.path]
disabled = 0

outputs

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = (Splunk IP):9997

[tcpout-server://(Splunk IP):9997]

Server

[general]
serverName = B-DC01
pass4SymmKey = (redacted)

[sslConfig]
sslKeysfilePassword = (redacted)

[lmpool:autogeneratedpoolforwarder]
description = auto
generatedpoolforwarder
quota = MAX
slaves = *
stack_id = forwarder

[lmpool:autogeneratedpoolfree]
description = auto
generatedpoolfree
quota = MAX
slaves = *
stack_id = free

Thank you.

0 Karma
Highlighted

Re: Why are universal forwarders installed on domain controllers not sending all Windows security and Cisco ASA logs?

Splunk Employee
Splunk Employee

Well, it seems like you have done quite a bit of sleuthing on this.

When you say "we had tried to get all AD events" during the troubleshooting phase, did you mean changing the Group Policy on your DCs to collect all AD related events, or did you mean using the ADmon input? These are two separate things.

My apologies for not including a time constraint in your search for hosts previously. You can easily rectify that and see what data is coming in for each host currently by setting the time picker to a range that falls outside of where most events that have come in for that host, for example, "Last 4 hours" would be good as a time constraint for the host that has stopped sending ASA logs and EventLog data a while back.

Some additional questions:

  1. Does the failure to get any type of log survive a restart of the forwarder or reboot of the DC?
  2. Are you doing packet captures at the Splunk Light indexer?
  3. Have you reviewed splunkd.log on the Splunk Light indexer and found any anomalies such as data not being indexed correctly?
  4. Have you turned on debug logging on the indexer?

There seems to be multiple issues in play here. A network failure on the indexer would result in no data coming in from any host. I do not believe that it is a problem with the indexing of data either, as a problem with indexing should result in failure to find data for all hosts, not just one.

0 Karma
Highlighted

Re: Why are universal forwarders installed on domain controllers not sending all Windows security and Cisco ASA logs?

New Member

We have.

"All AD events" refers to the a setting set on the splunk light web portal under "Add Data". I referenced the exact area in Splunk where we changed this in the previous post: Home, Add Data, forward, Existing, (select server class for windows servers [all 5 DC's are listed as forwarders]), Next, Local Event Logs, Security, Review, Submit. No idea what admon is. We have the logging policy on the DCs functioning correctly.

Time constraint was not a problem. I only mentioned it to provide as much information as possible.

  1. Yes. I am convinced it is a configuration issue on the splunk light server.
  2. Yes. Log data is coming in on port 9997. I can read security log entries in the packets (wireshark) from several DC's. Splunk is not reporting it.
  3. I found the following notice in the web portal. Otherwise the log is incomprehensible to me.

Received event for unconfigured/disabled/deleted index=wineventlog with source="source::WinEventLog:Security" host="host::IC-DC01" sourcetype="sourcetype::WinEventLog:Security". So far received events from 1 missing index(es).
2/11/2017, 7:42:51 AM

  1. I am not aware of how this is accomplished.

I still feel the problem is in the configuration settings in the Add Data section of splunk. I have no documentation on how these settings should be. How does one configure within Splunk the data it should assimilate?

On Forwarder management all 5 forwarders appear and have phoned home within a minute. Yet no data now flows for windows logs - only firewall data. I suspect settings within the Server Class or Apps areaS.

Currently I have 2 Server Classes. "Firewall Log" and "Security Log". 4DC's are in Firewall Log and 5 are in Security Log. It seems like I only really need 1 class with multiple apps but the whole thing is a bit confusing.

I have 17 "Apps". These are automatically created when I attempt to add new data. Nor was I able to find a way to change/view settings in an app aside from steps mentioned in paragraph 1 of this post.

I searched app.conf and inputs.conf in deployment-apps folder for the 2 apps I am using. Only thing there is:
[udp://1025]
[WinEventLog://Forwarded Events]
[WinEventLog://Setup]
[WinEventLog://System]
[WinEventLog://Application]
[WinEventLog://Security]

I need to know how to change which windows logs the forwarder is harvesting as well as resolving the cause for the indexer dumping the incoming entries rather than assimilating them.

Any further assistance will be much appreciated.

One other thing I noticed a lot of packets like this incoming from the forwarders in the capture:
pathDC:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.logMetaData:Index internalevtresolveadobj0

0 Karma
Highlighted

Re: Why are universal forwarders installed on domain controllers not sending all Windows security and Cisco ASA logs?

New Member

I solved the event log issues! Your comment about the indexes was the clue. Just now I ran the Add Data wizarad again and created a new index for the existing forwarders and all logs are now appearing.

Remaining issue: ASA logs (udp 1025 bouncing off the forwarder running on server2012r2) fail to appear. Under Hosts the IP for the source firewall is not listed.

Packet captures and netstat on the DC reveal that logs are arriving but forwarder is not listening. .conf files are identical to other DC's which are forwarding. I will try to reinstall the forwarder.

0 Karma
Highlighted

Re: Why are universal forwarders installed on domain controllers not sending all Windows security and Cisco ASA logs?

New Member

I think this is an issue with that server. The forwarder service is not appearing in the list of services yet the process for the service is running. I will focus on resolving that issue first. Thanks for your help.

0 Karma