No search results from Windows Servers (DC and EXCH)- what to do next?

[MBIT2022]

I recently inherited a newly configured Splunk Enterprise 8 environment after the former admin left. I have a basic user level knowledge of Splunk so I will describe my issue the best I can.

When we try to search for a specific or wildcard event (ie: print logs) we only receive results from the Linux servers but not the Windows servers. I was suggested to check the .conf files for Windows TA, but I'm not quite sure what I should be looking for within the files. The Splunk documentation site has been helpful, however it doesn't explain why we aren't seeing events. Splunk is installed on RHEL8 and we have installed forwarders on all the servers. I do not know where to go from here. Any assistance is appreciated. 

*Note: Former admin claimed that the server was fully configured in accordance with DIA's required auditable event list. The server is receiving data however it is not being disseminated properly. 

[MBIT2022]

It appears that we need to install the universal forwarder on every workstation. Is there any easy way to deploy it remotely? We do not have SCCM nor a Altiris license. Thanks

[PickleRick]

I know that the official recommended approach is to install UFs everywhere but for the sake of manageability did you consider using Windows Event Forwarding? (provided your windows workstations are in an AD domain).

[MBIT2022]

We were trying not to manually install the forwarders so they are installed on just the DCs and Exchange servers (and other servers).  We are able to pull information with a generic search but cannot see workstation or user specific information. I feel that either a setting is incorrect on the server or there is something misconfigured in one of the .conf files.

@PickleRick 

[PickleRick]

You can't get events from the workstations if you don't have access to that workstation. So you'd have to either have UFs installed on your all workstations (which as you say is impossible in your organization) or configure a WMI-based event retrieval (which might be working for a small set of servers but is really not a good idea for a huge number of workstations).

The alternative is to use Windows Event Forwarding mechanism (a built-in services in AD) which will cause forwarding of the events from the workstations to a designated Windows Server which will store them in Forwarder Events event log. From ther you could just pull them with a single splunk UF.

The downside to this method is that again - you'd need to configure WEF mechanism company-wide (most probably using GPO).

There is no magical way to get the events from the workstations without "touching them".

[MBIT2022]

Ok, I'll try to install forwarders on some workstations today and see if anything changes. For reference, when installing the forwarder should I be choosing Local or Domain under Configuration Options? I updated the forwarder on the DCs last week and couldn't find any set answer on which to choose. Thanks

[PickleRick]

Well, it's a relatively complicated topic. In domain environment you'd probably want to run splunk forwarder using a managed service account but that's something you want to discuss with your local admins. The account splunk forwarder runs with has  to have certain privileges and permissions (for example, reading event logs). You can run it with Local System account but that might not land very well with your security team.

[MBIT2022]

@PickleRick  I was able to manually install the forwarder on 9 workstations. I am definitely receiving more data but I'm still not seeing the events I need (successful/failed logins, print activity, file/folder modifications). Is there anything that needs to be configured via GPO? I have all servers set to collect and forward event logs. I want to share my .conf files but the network with Splunk is isolated and classified, so it is very difficult to move over data. Thanks

[PickleRick]

You need to enable windows event log inputs. If I recall correctly ingesting event logs doesn't require Add on for Windows installation but if you'd want other kinds of data from the workstation, it'd be necessary. So it's good to have it anyway - https://splunkbase.splunk.com/app/742/

[MBIT2022]

This is the same as Windows TA correct? We have it installed but I don't feel it was configured correctly as it gives no results on the App screen..

[MBIT2022]

So this is strange... I went to the Splunk for Windows app and it brought me to the Overview page but it's a dashboard for AWS.. To my knowledge no one has changed any config files so I don't understand why its showing this. The server path is correct: splunk8/en-US/app/Splunk_TA_windows/overview

[MBIT2022]

Ok, I'll look more into it. Thanks

[richgalloway]

The UF installation manual has sections about that.

https://docs.splunk.com/Documentation/Forwarder/8.2.4/Forwarder/Installanixuniversalforwarderremotel...

https://docs.splunk.com/Documentation/Forwarder/8.2.4/Forwarder/InstallaWindowsuniversalforwarderrem...

[MBIT2022]

Unfortunately we currently do not have any software deployment tools. My server admin also informed me that GPOs are not working properly so we cannot deploy via GPO. Thank you for the links

[richgalloway]

Please clarify.  When you say you don't receive events from Windows servers are you referring to Splunk instances running on Windows or Windows data sources that are indexed in Splunk?

It would help if you could share a sanitized search query or tell us more about how you are searching for events.  Linux and Windows can produce very different logs so how you search may determine which logs appear in the results.

[MBIT2022]

What I mean is that when I attempt to search for events in the Splunk GUI, it's not returning any results. The only search that really gives me results is an error search, but all the errors trace back to only 3-4 of my servers.  At least one is a Linux server and the others are Windows.

[isoutamo]

There could be defined default index which you are use in unless you are adding index=xyz on you SPL. As @richgalloway said it helps us if you could write here your spl query.
r. Ismo

[MBIT2022]

I'm honestly struggling to understand SPL. But if I try wildcard entries such as *login or *error I receive some results but only from a handful of servers and it's not always what I'm looking for. For other searches, it shows "0 of 2,500,000 events matched" so I know that Splunk is receiving data but for some reason its not letting me search for it. If that makes sense

[PickleRick]

Try doing the introductory free trainings. They are quite well written and give a quick overview of what splunk is and does.

You might do a

| tstats count where index=* by index host

over a day or week back to see if you have any data and just can't find it.

Oh, and you might be using a user with limited access to indexes.

[MBIT2022]

I ran a "index=*" search for the last week to date and so far it's only returned 46 hosts and 90mil events. Many are duplicate events but it appears that not all the servers and workstations are reporting and/or the forwarder is nor installed or configured properly. I will look into this further. We are also using 2.8.1. Is it absolutely necessary to update to 2.8.4? I ask only because every forwarder will have to be manually updated for the workstations. Thanks

[PickleRick]

Whereas you should (must?) keep the version consistent within the clustered server environment, you don't have to be so strict about UF<->"server" consistency. The compatibility matrix is here https://docs.splunk.com/Documentation/VersionCompatibility/current/Matrix/Compatibilitybetweenforwar...