Is there an option to drop older events from the pipeline? Older events can cause frequent bucket rolling and most likely not useful.
Latest splunk release has new added capability that allows older events to be routed or dropped without configuring any regex expression(e.g. INGEST_EVAL).
With this new capability, filtering happens during date time extraction, very early stage in the pipeline by AggregatorProcessor, it's least expensive way to dop/route older events.
New config ROUTE_EVENTS_OLDER_THAN in props.conf can be added to any stanza along with Timestamp extraction configurations. Right after date/time extraction, by default events can be routed to nulllQueue by setting ROUTE_EVENTS_OLDER_THAN.
ROUTE_EVENTS_OLDER_THAN = <non-negative integer>[s|m|h|d] * If set, AggregatorProcessor routes events older than 'ROUTE_EVENTS_OLDER_THAN' to nullQueue after timestamp extraction. * Default: no default
Example to drop (route to nullQueue) data that is older than 30 days, set ROUTE_EVENTS_OLDER_THAN=30d
[source::/Applications/splunk/var/spool/splunk]
TIME_PREFIX = \d{4}\/\d{2}\/\d{2} \d{2}:\d{2}:\d{2} \w+\s
MAX_TIMESTAMP_LOOKAHEAD = 21
ROUTE_EVENTS_OLDER_THAN = 30d
[host::foo] TIME_PREFIX = \d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} \w+\s TIME_FORMAT = %b %d %H:%M:%S %Y ROUTE_EVENTS_OLDER_THAN = 30d
Routing data based on dest routing key. ( index/queue/_TCP_ROUTING /_SYSLOG_ROUTING)
Route data 30 days and older straight to index queue and avoid regex extraction.
[host::foo] TIME_PREFIX = \d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} \w+\s TIME_FORMAT = %b %d %H:%M:%S %Y ROUTE_EVENTS_OLDER_THAN = 30d queue = indexqueue
Route data 30 days and older to another index
[host::foo] TIME_PREFIX = \d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} \w+\s TIME_FORMAT = %b %d %H:%M:%S %Y ROUTE_EVENTS_OLDER_THAN = 30d index = <30-days-and-older-index>
Index and forward OR HWF specific settings
Route data 30 days and older to another tcpout group
[host::foo] TIME_PREFIX = \d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} \w+\s TIME_FORMAT = %b %d %H:%M:%S %Y ROUTE_EVENTS_OLDER_THAN = 30d _TCP_ROUTING = <send to another cluster - OR - another tcpoutgroup>
Route data 30 days and older to another syslog output group
[host::foo] TIME_PREFIX = \d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} \w+\s TIME_FORMAT = %b %d %H:%M:%S %Y ROUTE_EVENTS_OLDER_THAN = 30d _SYSLOG_ROUTING = <send to another cluster - OR - another tcpoutgroup>
Latest splunk release has new added capability that allows older events to be routed or dropped without configuring any regex expression(e.g. INGEST_EVAL).
With this new capability, filtering happens during date time extraction, very early stage in the pipeline by AggregatorProcessor, it's least expensive way to dop/route older events.
New config ROUTE_EVENTS_OLDER_THAN in props.conf can be added to any stanza along with Timestamp extraction configurations. Right after date/time extraction, by default events can be routed to nulllQueue by setting ROUTE_EVENTS_OLDER_THAN.
ROUTE_EVENTS_OLDER_THAN = <non-negative integer>[s|m|h|d] * If set, AggregatorProcessor routes events older than 'ROUTE_EVENTS_OLDER_THAN' to nullQueue after timestamp extraction. * Default: no default
Example to drop (route to nullQueue) data that is older than 30 days, set ROUTE_EVENTS_OLDER_THAN=30d
[source::/Applications/splunk/var/spool/splunk]
TIME_PREFIX = \d{4}\/\d{2}\/\d{2} \d{2}:\d{2}:\d{2} \w+\s
MAX_TIMESTAMP_LOOKAHEAD = 21
ROUTE_EVENTS_OLDER_THAN = 30d
[host::foo] TIME_PREFIX = \d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} \w+\s TIME_FORMAT = %b %d %H:%M:%S %Y ROUTE_EVENTS_OLDER_THAN = 30d
Routing data based on dest routing key. ( index/queue/_TCP_ROUTING /_SYSLOG_ROUTING)
Route data 30 days and older straight to index queue and avoid regex extraction.
[host::foo] TIME_PREFIX = \d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} \w+\s TIME_FORMAT = %b %d %H:%M:%S %Y ROUTE_EVENTS_OLDER_THAN = 30d queue = indexqueue
Route data 30 days and older to another index
[host::foo] TIME_PREFIX = \d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} \w+\s TIME_FORMAT = %b %d %H:%M:%S %Y ROUTE_EVENTS_OLDER_THAN = 30d index = <30-days-and-older-index>
Index and forward OR HWF specific settings
Route data 30 days and older to another tcpout group
[host::foo] TIME_PREFIX = \d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} \w+\s TIME_FORMAT = %b %d %H:%M:%S %Y ROUTE_EVENTS_OLDER_THAN = 30d _TCP_ROUTING = <send to another cluster - OR - another tcpoutgroup>
Route data 30 days and older to another syslog output group
[host::foo] TIME_PREFIX = \d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} \w+\s TIME_FORMAT = %b %d %H:%M:%S %Y ROUTE_EVENTS_OLDER_THAN = 30d _SYSLOG_ROUTING = <send to another cluster - OR - another tcpoutgroup>