Latest splunk release has new added capability that allows older events to be routed or dropped without configuring any regex expression(e.g. INGEST_EVAL). With this new capability, filtering happens during date time extraction, very early stage in the pipeline by AggregatorProcessor, it's least expensive way to dop/route older events. New config ROUTE_EVENTS_OLDER_THAN in props.conf can be added to any stanza along with Timestamp extraction configurations. Right after date/time extraction, by default events can be routed to nulllQueue by setting ROUTE_EVENTS_OLDER_THAN. ROUTE_EVENTS_OLDER_THAN = <non-negative integer>[s|m|h|d]
* If set, AggregatorProcessor routes events older than 'ROUTE_EVENTS_OLDER_THAN'
to nullQueue after timestamp extraction.
* Default: no default
Example to drop (route to nullQueue) data that is older than 30 days, set ROUTE_EVENTS_OLDER_THAN=30d [source::/Applications/splunk/var/spool/splunk]
TIME_PREFIX = \d{4}\/\d{2}\/\d{2} \d{2}:\d{2}:\d{2} \w+\s
MAX_TIMESTAMP_LOOKAHEAD = 21
ROUTE_EVENTS_OLDER_THAN = 30d [host::foo]
TIME_PREFIX = \d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} \w+\s
TIME_FORMAT = %b %d %H:%M:%S %Y
ROUTE_EVENTS_OLDER_THAN = 30d Routing data based on dest routing key. ( index/queue/_TCP_ROUTING /_SYSLOG_ROUTING) Route data 30 days and older straight to index queue and avoid regex extraction. [host::foo]
TIME_PREFIX = \d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} \w+\s
TIME_FORMAT = %b %d %H:%M:%S %Y
ROUTE_EVENTS_OLDER_THAN = 30d
queue = indexqueue Route data 30 days and older to another index [host::foo]
TIME_PREFIX = \d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} \w+\s
TIME_FORMAT = %b %d %H:%M:%S %Y
ROUTE_EVENTS_OLDER_THAN = 30d
index = <30-days-and-older-index> Index and forward OR HWF specific settings Route data 30 days and older to another tcpout group [host::foo]
TIME_PREFIX = \d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} \w+\s
TIME_FORMAT = %b %d %H:%M:%S %Y
ROUTE_EVENTS_OLDER_THAN = 30d
_TCP_ROUTING = <send to another cluster - OR - another tcpoutgroup> Route data 30 days and older to another syslog output group [host::foo]
TIME_PREFIX = \d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} \w+\s
TIME_FORMAT = %b %d %H:%M:%S %Y
ROUTE_EVENTS_OLDER_THAN = 30d
_SYSLOG_ROUTING = <send to another cluster - OR - another tcpoutgroup> Note : Above configs are not applicable for HEC event endpoint as these events don't go through date/time extraction. In order to apply above routing rules on HEC events, one of the following option is required. 1. Use endpoint - services/collector/raw from the source end to undergo parsing and timestamp extraction, 2. You can also extract timestamp using /event endpoint by adding the field :auto_extract_timestamp=true . /services/collector/event?auto_extract_timestamp=true. Example: http://localhost:8088/services/collector/event?auto_extract_timestamp=true
... View more
