I don't have much experience with Splunk but am starting to use it in a new role and have done a lot of research before asking this question. There are two parts and I cannot provide screenshots.
I'm running Splunk Enterprise with 3 workstations and 1 DC forwarding to the backup DC which holds the Splunk Server. We recently did a hardware update and began exceeding our license by 3-4x per day. The configuration didn't change and I cannot find what is causing this. I blacklisted the 10 event codes that were generating 80% of the logs and while they are no longer showing in my search, the server appears to continue to index them and by 8am today my index capacity was at 17500MB/5000MB for the day.
I've also noticed anywhere from 50-1500 event logs for a single "Record Number." It's my understanding that a record number is unique to a single event and this means one event is getting logged several times. The time stamp is the same down to the millisecond. This I would argue is the bigger issue.
WinEventLog://Security
disabled = 0
start_from = newest
blacklist = 4648,4701,.... <-- ... is not literal, just have 8 more
Here is posting about windows event log with splunk if you haven’t found it yet? https://www.hurricanelabs.com/splunk-tutorials/windows-event-log-filtering-design-in-splunk
r. Ismo
Hi @michaeler,
Windows is very much verbose and only for a single accett to a machine you have 10-13 events 4624 (login) and 4634 (logout)!
You can easily check if the filter is running with a simple search:
index=wineventlog EventCode=4648
if you have events the filter isn't OK, if you haven't results it's OK.
The only hint I can give is to analyze your logs and filter one by one all the events you don't need.
Another hint: have you enabled perfmons? if yes, they probably they are the reason of your license consuption.
Ciao.
Giuseppe
The blacklist filters are working. I attempted to use crcSalt = <SOURCE> on inputs.conf to block the duplicate events but it did not work.
I also checked the indexes page this morning and the searchable events. By 0900 this morning I indexed 60,000,000 events but could only find about 200,000 events in the search.
I'm not sure about perfmons but will check when I get back on that network.
Hi @michaeler,
about perfmon, usually the are in an index called perfmon.
As I sai,
if your blacklists are running, you have to analyze your logs and identify the ones you really need and the ones you don't need: obviously, remember that if you filer a log you cannoy use it!
Then you could see the inputs.conf of the TA you're using (probably Splunk_TA_Windows), because maybe there's a too high frequence of the scipted inputs.
Anyway, it's always an analysis problem not a Splunk problem.
crcSalt is an option to use to reindex already indexed logs and it isn't useful for your need.
As I said, analyze your logs and identify the most relevant, then see if you can filter them (blacklists) or reduce frequency (scripted inputs).
Ciao.
Giuseppe