Hey Splunk Community,
One of my biggest challenges now is trying to figure out how to get old stuff out and focus on the new data coming in. I am wondering if there is a way to use the ( | delete) command or something on a schedule to delete/purge a file from a monitored directory once a new file is added. I have begun playing around with the dedup command to focus on the relevant data, but I think I will run into problems in the future if I can't remove old data from my index (because of changing unique identifiers and using too much storage space in the splunk cluster).
Example: Day 1 I upload a file with 10,000 events into a monitored directory. Day 2, I pull from the data source and the relevant data is now 9,500 events because 1,000 machines were removed and 500 new ones were added (9,000 is the same, 500 are completely new, 1,000 are no longer relevant).
How can I delete the file from day 1 from my host and only look at the events captured on day 2, etc? Is it possible to run a scheduled task to remove old data sources and only focus on the most recent? ( I also don't want to host/index to show 19,500 events, just the 9,500 from day 2). Is it possible to do this in Splunk Light?
Any thoughts on this issue would be greatly appreciated!
Thanks
... View more