topic Re: No search results from Windows Servers (DC and EXCH) in Splunk Search

No search results from Windows Servers (DC and EXCH)- what to do next?

MBIT2022 — Thu, 10 Feb 2022 14:48:43 GMT

I recently inherited a newly configured Splunk Enterprise 8 environment after the former admin left. I have a basic user level knowledge of Splunk so I will describe my issue the best I can.

When we try to search for a specific or wildcard event (ie: print logs) we only receive results from the Linux servers but not the Windows servers. I was suggested to check the .conf files for Windows TA, but I'm not quite sure what I should be looking for within the files. The Splunk documentation site has been helpful, however it doesn't explain why we aren't seeing events. Splunk is installed on RHEL8 and we have installed forwarders on all the servers. I do not know where to go from here. Any assistance is appreciated.

*Note: Former admin claimed that the server was fully configured in accordance with DIA's required auditable event list. The server is receiving data however it is not being disseminated properly.

Re: No event results from Windows Servers (DC and EXCH)

richgalloway — Wed, 02 Feb 2022 16:09:58 GMT

Please clarify. When you say you don't receive events from Windows servers are you referring to Splunk instances running on Windows or Windows data sources that are indexed in Splunk?

It would help if you could share a sanitized search query or tell us more about how you are searching for events. Linux and Windows can produce very different logs so how you search may determine which logs appear in the results.

Re: No event results from Windows Servers (DC and EXCH)

MBIT2022 — Wed, 02 Feb 2022 17:14:18 GMT

What I mean is that when I attempt to search for events in the Splunk GUI, it's not returning any results. The only search that really gives me results is an error search, but all the errors trace back to only 3-4 of my servers. At least one is a Linux server and the others are Windows.

Re: No event results from Windows Servers (DC and EXCH)

isoutamo — Wed, 02 Feb 2022 21:37:04 GMT

There could be defined default index which you are use in unless you are adding index=xyz on you SPL. As @richgalloway said it helps us if you could write here your spl query.
r. Ismo

Re: No event results from Windows Servers (DC and EXCH)

MBIT2022 — Fri, 04 Feb 2022 12:04:51 GMT

I'm honestly struggling to understand SPL. But if I try wildcard entries such as *login or *error I receive some results but only from a handful of servers and it's not always what I'm looking for. For other searches, it shows "0 of 2,500,000 events matched" so I know that Splunk is receiving data but for some reason its not letting me search for it. If that makes sense

Re: No event results from Windows Servers (DC and EXCH)

richgalloway — Fri, 04 Feb 2022 12:44:25 GMT

Please share the full query, the (sanitized) results, and the expected results.

Re: No event results from Windows Servers (DC and EXCH)

PickleRick — Fri, 04 Feb 2022 12:58:43 GMT

Try doing the introductory free trainings. They are quite well written and give a quick overview of what splunk is and does.

You might do a

| tstats count where index=* by index host

over a day or week back to see if you have any data and just can't find it.

Oh, and you might be using a user with limited access to indexes.

Re: No event results from Windows Servers (DC and EXCH)

MBIT2022 — Fri, 04 Feb 2022 13:26:08 GMT

I ran a "index=*" search for the last week to date and so far it's only returned 46 hosts and 90mil events. Many are duplicate events but it appears that not all the servers and workstations are reporting and/or the forwarder is nor installed or configured properly. I will look into this further. We are also using 2.8.1. Is it absolutely necessary to update to 2.8.4? I ask only because every forwarder will have to be manually updated for the workstations. Thanks

Re: No event results from Windows Servers (DC and EXCH)

PickleRick — Fri, 04 Feb 2022 14:01:07 GMT

Whereas you should (must?) keep the version consistent within the clustered server environment, you don't have to be so strict about UF<->"server" consistency. The compatibility matrix is here https://docs.splunk.com/Documentation/VersionCompatibility/current/Matrix/Compatibilitybetweenforwardersandindexers

Re: No event results from Windows Servers (DC and EXCH)

MBIT2022 — Fri, 04 Feb 2022 14:41:36 GMT

ok thank you

Re: No search results from Windows Servers (DC and EXCH)

MBIT2022 — Mon, 07 Feb 2022 15:15:22 GMT

It appears that we need to install the universal forwarder on every workstation. Is there any easy way to deploy it remotely? We do not have SCCM nor a Altiris license. Thanks

Re: No search results from Windows Servers (DC and EXCH)

richgalloway — Mon, 07 Feb 2022 22:33:00 GMT

The UF installation manual has sections about that.

https://docs.splunk.com/Documentation/Forwarder/8.2.4/Forwarder/Installanixuniversalforwarderremotelywithastaticconfiguration

https://docs.splunk.com/Documentation/Forwarder/8.2.4/Forwarder/InstallaWindowsuniversalforwarderremotelywithastaticconfiguration

Re: No search results from Windows Servers (DC and EXCH)

PickleRick — Tue, 08 Feb 2022 07:38:09 GMT

I know that the official recommended approach is to install UFs everywhere but for the sake of manageability did you consider using Windows Event Forwarding? (provided your windows workstations are in an AD domain).

Re: No search results from Windows Servers (DC and EXCH)

MBIT2022 — Tue, 08 Feb 2022 11:28:45 GMT

We were trying not to manually install the forwarders so they are installed on just the DCs and Exchange servers (and other servers). We are able to pull information with a generic search but cannot see workstation or user specific information. I feel that either a setting is incorrect on the server or there is something misconfigured in one of the .conf files.

@PickleRick

Re: No search results from Windows Servers (DC and EXCH)

MBIT2022 — Tue, 08 Feb 2022 11:32:23 GMT

Unfortunately we currently do not have any software deployment tools. My server admin also informed me that GPOs are not working properly so we cannot deploy via GPO. Thank you for the links

Re: No search results from Windows Servers (DC and EXCH)

PickleRick — Tue, 08 Feb 2022 11:43:03 GMT

You can't get events from the workstations if you don't have access to that workstation. So you'd have to either have UFs installed on your all workstations (which as you say is impossible in your organization) or configure a WMI-based event retrieval (which might be working for a small set of servers but is really not a good idea for a huge number of workstations).

The alternative is to use Windows Event Forwarding mechanism (a built-in services in AD) which will cause forwarding of the events from the workstations to a designated Windows Server which will store them in Forwarder Events event log. From ther you could just pull them with a single splunk UF.

The downside to this method is that again - you'd need to configure WEF mechanism company-wide (most probably using GPO).

There is no magical way to get the events from the workstations without "touching them".

Re: No search results from Windows Servers (DC and EXCH)

MBIT2022 — Tue, 08 Feb 2022 11:48:04 GMT

Ok, I'll try to install forwarders on some workstations today and see if anything changes. For reference, when installing the forwarder should I be choosing Local or Domain under Configuration Options? I updated the forwarder on the DCs last week and couldn't find any set answer on which to choose. Thanks

Re: No search results from Windows Servers (DC and EXCH)

PickleRick — Tue, 08 Feb 2022 12:01:11 GMT

Well, it's a relatively complicated topic. In domain environment you'd probably want to run splunk forwarder using a managed service account but that's something you want to discuss with your local admins. The account splunk forwarder runs with has to have certain privileges and permissions (for example, reading event logs). You can run it with Local System account but that might not land very well with your security team.

Re: No search results from Windows Servers (DC and EXCH)

MBIT2022 — Tue, 08 Feb 2022 12:03:54 GMT

Ok, I'll look more into it. Thanks

Re: No search results from Windows Servers (DC and EXCH)

MBIT2022 — Thu, 10 Feb 2022 13:45:50 GMT

@PickleRick I was able to manually install the forwarder on 9 workstations. I am definitely receiving more data but I'm still not seeing the events I need (successful/failed logins, print activity, file/folder modifications). Is there anything that needs to be configured via GPO? I have all servers set to collect and forward event logs. I want to share my .conf files but the network with Splunk is isolated and classified, so it is very difficult to move over data. Thanks