High License Usage on Indexer/Search Head

[robertjollsdrs]

I have a Splunk Enterprise instance with a 1GB license set up to aggregate logs in a small Windows AD environment (Server 2016 DC, CentOS file server, and < 10 Win10 workstations). I currently have the DC, file server, and 3 workstations deployed. I keep getting license usage warnings. Upon investigation, the CentOS server where the Splunk server is installed is by far the largest license user (on average 200% usage). Furthermore, my linux_audit sourcetype is the main source of the usage. That sourcetype only monitors /var/log/audit/audit.log. On disk, /var/log/audit/audit.log is only 74MB, so I have no idea why I am using 2GB+ of license every single day!

Can anyone help?

[robertjollsdrs · Accepted solution]

Solved! I found this other post:

https://community.splunk.com/t5/Alerting/Why-am-I-receiving-too-many-Splunk-logs-on-audit-log/m-p/42...

Turns out that Splunk was doing its job properly, but the server hosting my Splunk indexer had audit settings that were logging every file action Splunk was doing. I disabled auditing inside the defaultdb, _metrics, and _introspection directories and the indexing volume dropped off. Everything works great now!

[robertjollsdrs]

I checked that out and it seems that the log file is just that big - I also checked the actual log file sizes and realized that with file rotation, the server is actually generating that much log data. I need to dive in and see what is going on.

[codebuilder]

With log rotation you'll  want to ensure that your aren't indexing the same log file more than once.
Splunk will see your_log_file.log and your_log_file.log.gz (or your_log_file.log.1) as two different files and ingest them both. 

You can avoid this by blacklisting everything and then whitelist .log files, or blacklist .gz files, etc.
To check where your events are coming from you can run something like:

|tstats count where index=your_index_name_here by source

[venkatasri]

Hi @robertjollsdrs 

Before deep dive checking the Splunk default provided report is where you can find first hand details. - https://docs.splunk.com/Documentation/Splunk/8.2.1/Admin/AboutSplunksLicenseUsageReportView

Splunk internal logs doesn't count under License , Have you installed any add-ons specific to CentOS?

You can issue following command  under $SPLUNK_HOME/bin and find out what files are being monitored.

Any file outside the location $SPLUNK_HOME could be adding to your quota, checkout how big they are.

./splunk list monitor

--

An upvote would be appreciated if it helps!