topic Re: Exclude Old data in Splunk Enterprise

Exclude Old data

Sree — Tue, 20 Jul 2021 13:50:59 GMT

Hi,

I'm trying to exclude events that have an old timestamp in a url which look like this -

{"timestamp": 1626739199.964, "c-ip": "178.245.92.14", "time-to-first-byte": 0.002, "sc-status": 404, "sc-bytes": 467, "cs-method": "GET", "cs-protocol": "https", "cs-host": "xxxxxxx", "cs-uri-stem": "/out/v1/bac5ea7d5e06476598d34ba48b3f1bd1/index_8_0.m3u8?start=2021-07-16T16:40:07+00:00",

here timestamp and the start date are different.So is there any way to index the events have start date as current day.

Thanks in advance

Re: Exclude Old data

richgalloway — Tue, 20 Jul 2021 17:37:25 GMT

Convert the start date to epoch form then filter on it.

... | eval startDate=strptime(start,"%Y-%m-%dT%H:%M:%S%:z") | where startDate>=relative_time(now(), "@d")

Re: Exclude Old data

Sree — Thu, 22 Jul 2021 14:26:16 GMT

Hi @richgalloway

Thanks for the reply but your answer is for the splunk events that got already indexed.

I'm trying to filter the data at the sourcetype level before it gets indexed.

Re: Exclude Old data

richgalloway — Thu, 22 Jul 2021 15:26:57 GMT

Sorry, misunderstood the question. Filtering at index time requires a transform that sends data to nullQueue. The trick (for which I don't have an answer) is writing a regex that matches the current day without having to change it every day. I'm not sure it's possible. Perhaps Cribl can do it.