Getting Data In

Excessive Windows Event Logs

michaeler
Communicator

I don't have much experience with Splunk but am starting to use it in a new role and have done a lot of research before asking this question. There are two parts and I cannot provide screenshots.

I'm running Splunk Enterprise with 3 workstations and 1 DC forwarding to the backup DC which holds the Splunk Server. We recently did a hardware update and began exceeding our license by 3-4x per day. The configuration didn't change and I cannot find what is causing this. I blacklisted the 10 event codes that were generating 80% of the logs and while they are no longer showing in my search, the server appears to continue to index them and by 8am today my index capacity was at 17500MB/5000MB for the day.

I've also noticed anywhere from 50-1500 event logs for a single "Record Number." It's my understanding that a record number is unique to a single event and this means one event is getting logged several times. The time stamp is the same down to the millisecond. This I would argue is the bigger issue.

WinEventLog://Security
disabled = 0
start_from = newest
blacklist = 4648,4701,.... <-- ... is not literal, just have 8 more

Labels (3)
0 Karma

isoutamo
SplunkTrust
SplunkTrust

Here is posting about windows event log with splunk if you haven’t found it yet? https://www.hurricanelabs.com/splunk-tutorials/windows-event-log-filtering-design-in-splunk
r. Ismo

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @michaeler,

Windows is very much verbose and only for a single accett to a machine you have 10-13 events 4624 (login) and 4634 (logout)!

You can easily check if the filter is running with a simple search:

index=wineventlog EventCode=4648

if you have events the filter isn't OK, if you haven't results it's OK.

The only hint I can give is to analyze your logs and filter one by one all the events you don't need.

Another hint: have you enabled perfmons? if yes, they probably they are the reason of your license consuption.

Ciao.

Giuseppe

0 Karma

michaeler
Communicator

The blacklist filters are working. I attempted to use crcSalt = <SOURCE> on inputs.conf to block the duplicate events but it did not work. 

I also checked the indexes page this morning and the searchable events. By 0900 this morning I indexed 60,000,000 events but could only find about 200,000 events in the search.

I'm not sure about perfmons but will check when I get back on that network.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @michaeler,

about perfmon, usually the are in an index called perfmon.

As I sai,

if your blacklists are running, you have to analyze your logs and identify the ones you really need and the ones you don't need: obviously, remember that if you filer a log you cannoy use it!

Then you could see the inputs.conf of the TA you're using (probably Splunk_TA_Windows), because maybe there's a too high frequence of the scipted inputs.

Anyway, it's always an analysis problem not a Splunk problem.

crcSalt is an option to use to reindex already indexed logs and it isn't useful for your need.

As I said, analyze your logs and identify the most relevant, then see if you can filter them (blacklists) or reduce frequency (scripted inputs).

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...