topic Re: What are recommendations for reducing license usage? in Getting Data In

What are recommendations for reducing license usage?

Raghav2384 — Thu, 16 Oct 2014 12:55:57 GMT

Experts,

we have 100GB license and that data is being fed to Splunk. Out of that 100GB, 10% is what need to be retained forever and the 90% is needed only for 7 days max.
I know retention is not splunk's worry and what ever you want needs to be indexed.
Here is what i could think of...
1:For the remaining 90%, filter>filter>filter>filter using HF and feed it to splunk. Is there any other alternative you guys recommend? even a 30% saving on the usage would be a big deal.
2 : Parse the remaining 90%, save it to a storage and index it on demand?
oh, we filter the noise 90 by a specific keyword.

Thank you for your time.

Raghav

Re: What are recommendations for reducing license usage?

Raghav2384 — Thu, 16 Oct 2014 17:39:19 GMT

I Guess Filter>Filter>Filter is what i should be looking for

Re: What are recommendations for reducing license usage?

yannK — Thu, 16 Oct 2014 17:58:03 GMT

The retention managed by index in splunk.
So you can have an index with short retention (7 days in your case), and an index with long retention (6 years default, or size limit)

If you want to reduce the license usage, then you should keep only monitors enabled for your critical logs.
and have all the others inputs disabled.
By example with special inputs, or with an app. (put all temp inputs in an app to turn them, on/off altogether)

The problem with keeping the useless inputs disabled, and enable them at will, is that they will likely pick up all the events available and may blow up your license usage for that day.

A better approach is to use methods like :

ignore older modified files : ignoreOlderThan = 2d will skip files modified more than 2 days ago
use nullQueue filters to drop some events (example fro the DEBUG events)
monitor a different folder, and simply copy the logs to monitor to this folder. (example the batch input or the splunk spooler)
on windows, wineventlog use the option current_only=1 to collect only current events, not historical ones

see http://docs.splunk.com/Documentation/Splunk/6.1.4/admin/Inputsconf