Splunk Indexing rate in extremely high

[Mescudi]

Hello,

We are using a Splunk enterprise license currently with 24 gb of license space. Our problem is that are indexing rate is above 1000kb/s and maxing out our license usage. We cannot upgrade our license usage due to policies. Our usage reports were not configured, so we cannot see anything through monitoring report. Is there something possible in the inputs or config files that is causing are machines to send such a large amount of info to splunk? 

Any help would be appreciated 

thanks

[inventsekar]

Hi @Mescudi ... the DMC console got some dashboards(i am not sure of the dashboards, as currently i dont have access to a DMC) that will tell you which UF's are sending high amounts of data/logs. you should find those UF's and troubleshoot the inputs.conf.. you may not need some logs i think, but mistakenly configured. 

for example, on UF inputs.conf, if /tmp/* is added for monitoring (lets assume the "*" was added by mistake), then every file under the tmp directory will be monitored. 

you can search for some search queries... Splunk license usage by UFs.

Let us know how it goes.. we will help you to troubleshoot this issue, thanks. 

[Mescudi]

Okay when searching through our DMC I See a couple machines and our Splunk server itself causing high amounts of events. I went to check one of the machines and they don't have an inputs.conf file in the universal forwarder file. Could this be the problem? 

[Mescudi]

It also seems to be coming from metric logs, I'm just confused as to what is causing such a high index rate when before it was low. We just hit 97% on our license usage.