Splunk Search

No search results from Windows Servers (DC and EXCH)- what to do next?

MBIT2022
Explorer

I recently inherited a newly configured Splunk Enterprise 8 environment after the former admin left. I have a basic user level knowledge of Splunk so I will describe my issue the best I can.

When we try to search for a specific or wildcard event (ie: print logs) we only receive results from the Linux servers but not the Windows servers. I was suggested to check the .conf files for Windows TA, but I'm not quite sure what I should be looking for within the files. The Splunk documentation site has been helpful, however it doesn't explain why we aren't seeing events. Splunk is installed on RHEL8 and we have installed forwarders on all the servers. I do not know where to go from here. Any assistance is appreciated. 

*Note: Former admin claimed that the server was fully configured in accordance with DIA's required auditable event list. The server is receiving data however it is not being disseminated properly. 

Labels (1)
0 Karma

MBIT2022
Explorer

It appears that we need to install the universal forwarder on every workstation. Is there any easy way to deploy it remotely? We do not have SCCM nor a Altiris license. Thanks

0 Karma

PickleRick
Ultra Champion

I know that the official recommended approach is to install UFs everywhere but for the sake of manageability did you consider using Windows Event Forwarding? (provided your windows workstations are in an AD domain).

0 Karma

MBIT2022
Explorer

We were trying not to manually install the forwarders so they are installed on just the DCs and Exchange servers (and other servers).  We are able to pull information with a generic search but cannot see workstation or user specific information. I feel that either a setting is incorrect on the server or there is something misconfigured in one of the .conf files.

@PickleRick 

0 Karma

PickleRick
Ultra Champion

You can't get events from the workstations if you don't have access to that workstation. So you'd have to either have UFs installed on your all workstations (which as you say is impossible in your organization) or configure a WMI-based event retrieval (which might be working for a small set of servers but is really not a good idea for a huge number of workstations).

The alternative is to use Windows Event Forwarding mechanism (a built-in services in AD) which will cause forwarding of the events from the workstations to a designated Windows Server which will store them in Forwarder Events event log. From ther you could just pull them with a single splunk UF.

The downside to this method is that again - you'd need to configure WEF mechanism company-wide (most probably using GPO).

There is no magical way to get the events from the workstations without "touching them".

0 Karma

MBIT2022
Explorer

Ok, I'll try to install forwarders on some workstations today and see if anything changes. For reference, when installing the forwarder should I be choosing Local or Domain under Configuration Options? I updated the forwarder on the DCs last week and couldn't find any set answer on which to choose. Thanks

0 Karma

PickleRick
Ultra Champion

Well, it's a relatively complicated topic. In domain environment you'd probably want to run splunk forwarder using a managed service account but that's something you want to discuss with your local admins. The account splunk forwarder runs with has  to have certain privileges and permissions (for example, reading event logs). You can run it with Local System account but that might not land very well with your security team.

0 Karma

MBIT2022
Explorer

@PickleRick  I was able to manually install the forwarder on 9 workstations. I am definitely receiving more data but I'm still not seeing the events I need (successful/failed logins, print activity, file/folder modifications). Is there anything that needs to be configured via GPO? I have all servers set to collect and forward event logs. I want to share my .conf files but the network with Splunk is isolated and classified, so it is very difficult to move over data. Thanks

0 Karma

PickleRick
Ultra Champion

You need to enable windows event log inputs. If I recall correctly ingesting event logs doesn't require Add on for Windows installation but if you'd want other kinds of data from the workstation, it'd be necessary. So it's good to have it anyway - https://splunkbase.splunk.com/app/742/

 

MBIT2022
Explorer

This is the same as Windows TA correct? We have it installed but I don't feel it was configured correctly as it gives no results on the App screen..

0 Karma

MBIT2022
Explorer

So this is strange... I went to the Splunk for Windows app and it brought me to the Overview page but it's a dashboard for AWS.. To my knowledge no one has changed any config files so I don't understand why its showing this. The server path is correct: splunk8/en-US/app/Splunk_TA_windows/overview

0 Karma

MBIT2022
Explorer

Ok, I'll look more into it. Thanks

0 Karma

richgalloway
SplunkTrust
SplunkTrust

MBIT2022
Explorer

Unfortunately we currently do not have any software deployment tools. My server admin also informed me that GPOs are not working properly so we cannot deploy via GPO. Thank you for the links

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Please clarify.  When you say you don't receive events from Windows servers are you referring to Splunk instances running on Windows or Windows data sources that are indexed in Splunk?

It would help if you could share a sanitized search query or tell us more about how you are searching for events.  Linux and Windows can produce very different logs so how you search may determine which logs appear in the results.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

MBIT2022
Explorer

What I mean is that when I attempt to search for events in the Splunk GUI, it's not returning any results. The only search that really gives me results is an error search, but all the errors trace back to only 3-4 of my servers.  At least one is a Linux server and the others are Windows.

0 Karma

isoutamo
SplunkTrust
SplunkTrust
There could be defined default index which you are use in unless you are adding index=xyz on you SPL. As @richgalloway said it helps us if you could write here your spl query.
r. Ismo
0 Karma

MBIT2022
Explorer

I'm honestly struggling to understand SPL. But if I try wildcard entries such as *login or *error I receive some results but only from a handful of servers and it's not always what I'm looking for. For other searches, it shows "0 of 2,500,000 events matched" so I know that Splunk is receiving data but for some reason its not letting me search for it. If that makes sense

0 Karma

PickleRick
Ultra Champion

Try doing the introductory free trainings. They are quite well written and give a quick overview of what splunk is and does.

You might do a

| tstats count where index=* by index host

over a day or week back to see if you have any data and just can't find it.

Oh, and you might be using a user with limited access to indexes.

0 Karma

MBIT2022
Explorer

I ran a "index=*" search for the last week to date and so far it's only returned 46 hosts and 90mil events. Many are duplicate events but it appears that not all the servers and workstations are reporting and/or the forwarder is nor installed or configured properly. I will look into this further. We are also using 2.8.1. Is it absolutely necessary to update to 2.8.4? I ask only because every forwarder will have to be manually updated for the workstations. Thanks

0 Karma

PickleRick
Ultra Champion

Whereas you should (must?) keep the version consistent within the clustered server environment, you don't have to be so strict about UF<->"server" consistency. The compatibility matrix is here https://docs.splunk.com/Documentation/VersionCompatibility/current/Matrix/Compatibilitybetweenforwar...

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...