Excessive Windows Event Logs

[michaeler]

I don't have much experience with Splunk but am starting to use it in a new role and have done a lot of research before asking this question. There are two parts and I cannot provide screenshots.

I'm running Splunk Enterprise with 3 workstations and 1 DC forwarding to the backup DC which holds the Splunk Server. We recently did a hardware update and began exceeding our license by 3-4x per day. The configuration didn't change and I cannot find what is causing this. I blacklisted the 10 event codes that were generating 80% of the logs and while they are no longer showing in my search, the server appears to continue to index them and by 8am today my index capacity was at 17500MB/5000MB for the day.

I've also noticed anywhere from 50-1500 event logs for a single "Record Number." It's my understanding that a record number is unique to a single event and this means one event is getting logged several times. The time stamp is the same down to the millisecond. This I would argue is the bigger issue.

WinEventLog://Security
disabled = 0
start_from = newest
blacklist = 4648,4701,....    <-- ... is not literal, just have 8 more

[isoutamo]

Here is posting about windows event log with splunk if you haven’t found it yet? https://www.hurricanelabs.com/splunk-tutorials/windows-event-log-filtering-design-in-splunk
r. Ismo

[gcusello]

Hi @michaeler,

Windows is very much verbose and only for a single accett to a machine you have 10-13 events 4624 (login) and 4634 (logout)!

You can easily check if the filter is running with a simple search:

index=wineventlog EventCode=4648

if you have events the filter isn't OK, if you haven't results it's OK.

The only hint I can give is to analyze your logs and filter one by one all the events you don't need.

Another hint: have you enabled perfmons? if yes, they probably they are the reason of your license consuption.

Ciao.

Giuseppe

[michaeler]

The blacklist filters are working. I attempted to use crcSalt = <SOURCE> on inputs.conf to block the duplicate events but it did not work. 

I also checked the indexes page this morning and the searchable events. By 0900 this morning I indexed 60,000,000 events but could only find about 200,000 events in the search.

I'm not sure about perfmons but will check when I get back on that network.

[gcusello]

Hi @michaeler,

about perfmon, usually the are in an index called perfmon.

As I sai,

if your blacklists are running, you have to analyze your logs and identify the ones you really need and the ones you don't need: obviously, remember that if you filer a log you cannoy use it!

Then you could see the inputs.conf of the TA you're using (probably Splunk_TA_Windows), because maybe there's a too high frequence of the scipted inputs.

Anyway, it's always an analysis problem not a Splunk problem.

crcSalt is an option to use to reindex already indexed logs and it isn't useful for your need.

As I said, analyze your logs and identify the most relevant, then see if you can filter them (blacklists) or reduce frequency (scripted inputs).

Ciao.

Giuseppe