Hi Mata
The options are simple :
reduce the indexed volume.
or get a license volume upgrade (contact splunk sales)
For the first option here are the steps :
1 - Analyze you data, to identify where the volume it is coming from.
in 4.2+ you can use those searches on the license-master
see http://wiki.splunk.com/Community:TroubleshootingIndexedDataVolume
if you prefer detail, you can add details on the source "s", host "h", sourcetype "st", indexer "i".
total per pool index=_internal source=*license_usage.log type=Usage | eval GB=b/1024/1024/1024 | timechart span=1d sum(GB) by pool
detail per sourcetype
index=_internal source=*license_usage.log type=Usage | eval GB=b/1024/1024/1024 | timechart span=1d sum(GB) by st useother=false
detail per source
index=_internal source=*license_usage.log type=Usage | eval GB=b/1024/1024/1024 | timechart span=1d sum(GB) by s useother=false
detail per host
index=_internal source=*license_usage.log type=Usage | eval GB=b/1024/1024/1024 | timechart span=1d sum(GB) by h useother=false
2- If some forwarders are not necessary, turn splunk forwarder off on those boxes.
Why did you deployed a forwarder on every single box in the first place !!!
3- If some useless files are being indexed, be more selective.
Disable the inputs, or use whitelist/blacklists to limit the scope
example to drop the core files, or to index only .log files:
`[montitor:///var/log]
blacklist=.core$
[monitor:///mypath/.log]
`
4 - If some servers are sending to much data (syslog by example)
disable the routing to splunk, or select the components to send.
example on syslog.conf (send only critical and errors, and every event from my application)
*.CRITICAL splunk.mydomain.com
*.ERROR splunk.mydomain.com
myapplication.* splunk.mydomain.com
5 - If some log files contains too much data, change the verbosity level of your applications (by example, avoid the DEBUG mode)
6- Search for duplicates events in the logs, please check they exists in the original logs, or if the same log file is being indexed several times (some log rotation may cause that)
here are searches to find duplicates in splunk :
* | eval raw=_raw | convert ctime(_indextime) as indextime
| stats count first(indextime) as first last(indextime) as last by raw | where count > 1 | table count first last raw
Then drilldown to the source to figure.
7 - If your cannot disable an input but don't need all the events, you can setup a NULLQUEUE filtering of the events.
This has to be setup on the indexers (or heavy forwarders)
(with windows eventlogs, we usually use filtering on the eventcode)
see examples http://docs.splunk.com/Documentation/Splunk/4.3/Deploy/Routeandfilterdatad#Discard_specific_events_and_keep_the_rest:
Discard specific events and keep the rest
or Keep specific events and discard the rest
... View more