Can I configure Heavy Forwards to Reject *Events* older than a certain age?

[briancronrath]

I should clarify I am not talking about the "ignoreOlderThan" setting which by my understanding only looks at the file date, what I'm interested in here is ignoring events themselves that are older than a certain date.  We have a lot of events come in through methods other than file ingestion, and a lot of the time we get events that are older than the configured retention age on an index, which cause early eviction of other events lumped into the bucket.  I'm looking for a way if possible, to look at events on the heavy forwarders and reject them if they are older than a certain age, ideally per index if possible as well.

[woodcock · Accepted solution]

props.conf:

# For now, do it for ALL sourcetypes.
[(::)?...]
TRANSFORMS-oldevents=delete-older-than-January-1-2022

transforms.conf:

[delete-older-than-January-1-2021]
# 01 Jan 2021 01:01:01 GMT = 1609462861
INGEST_EVAL = queue=if(substr(tostring(1609462861-_time),1,1)=”-“, “indexQueue”, “nullQueue”)

[delete-older-than-January-1-2022]
# 01 Jan 2022 01:01:01 GMT = 1640998861
INGEST_EVAL = queue=if(substr(tostring(1640998861-_time),1,1)=”-“, “indexQueue”, “nullQueue”)

[delete-older-than-January-1-2023]
# 01 Jan 2023 01:01:01 GMT = 1672534861
INGEST_EVAL = queue=if(substr(tostring(1672534861-_time),1,1)=”-“, “indexQueue”, “nullQueue”)

[briancronrath]

Awesome thank you so much!  I modified it a bit to use a relative time check:

INGEST_EVAL = queue=if(substr(tostring(relative_time(now(),"-10d")-_time),1,1)="-", "indexQueue", "nullQueue")