I recently took over as an admin for Splunk on one of my company's networks. We have 4 Forwarders and one enterprise instance. We recently updated our workstations and started getting large increases in events and exceeded our index by 8x everyday.
I recently monitored the data at different points in the day and realized every event is getting re-indexed every minute. I watched one time period grow from 2500 events to 250,000 by the end of the day. If i refreshed the search it would have an additional 1200 events every minute (roughly).
What could be causing Splunk to re-index the same events everytime a new one gets logged?
can you provide inputs.conf.
I'm not there and don't have it memorized but its something like this:
[WinEventLog://Security]
disabled = 0
start_from = newest
blacklist = 4648,7310
suppress_text = 1
[WinEventLog://Application]
disabled = 0
start_from = newest
blacklist = 4648,7310
suppress_text = 1
[WinEventLog://System]
disabled = 0
start_from = newest
blacklist = 4648,7310
suppress_text = 1
[perfmon]
disabled = 1
I've previously set "starts_from = oldest" and had the same issues.
Ignore the code numbers on the blacklist. I can't remember the specifics for each of those but I've blacklisted what contributes roughly 90% of all logs for each source.