If you want to delete an already indexed data and just view the latest, you can try to set a retention period with it.
And if you want to delete the files you are monitoring after indexing it, you can use batch instead of monitor.
Consider this as a work-around since this doesn't directly addresses your concern.
Hope it helps.
You can monitor files in a folder using inputs.conf
When one file is updated Splunk indexes only the new events.
If you delete a file and then you copy a new different version of it in the folder, it will be reindexed all.
What is the problem: in the new version of your file there are also old events?
let me better understand:
you put files in a folder and they are indexed by Splunk
after you delete these files and put new ones in this folder,
Splunk indexes the new files.
What are the "old files data" that you say, you deleted the old ones?
If you put new files containing old data they will be reindexed again.
old files i meant here is first file which is indexed.
if i keep the second file in the monitoring folder. i want the results of second file data only. the indexer should not have first file data
a file is indexed once in Splunk, why you say that you don't want to reindex it? do your splunk reindex the old file?
when a new files are copied in a folder only the new ones are indexed the old one isn't indexed again.
If instead the old file is updated (events are added), only the new events are indexed.
In other words, do you want to delete old file after indexing?
Splunk doesn't delete files after indexing but iyou could create a batch script that deletes all files older that a period (e.g. one hour).
If you want to do a more scientific deletion, you could generate a report that lists all the sources already indexed (
index=your_index, sourcetype=yoursourcetype) | dedup source | table source | outputcsv deletion_list.csv) and use it as input in your deletion batch.
Maybe I'm starting to understand: do you want to remove old files on file system and also old events in the index?
to do this, set the retantion for the index you're using: e.g. if you want to load data on an index every day and delete all older events set a retention (frozenTimePeriodInSecs = 86400).
to be more sure you could use in your searches as period -24h and delete events ater 2 days.
Otherwise (better!) if you haven't too logs, you could load them in a lookup and nightly rebuild it.
i have a batch script to remove the old file in filesystem. i want to understand is there any way to remove the old file data from indexer when a new data gets indexed.
retention policy we can set in cold bucket but it will be help us here. may be two new files will be generated in some intervals of twice a day
The only correct way to delete old events is to set retention in indexes.
there is another way but not correct, you could schedule deletion of old events, but remebering that in Splunk deletion is only logical and deleted events remain in indexes until retention period expires.
I don't know how many events you daily index, if they aren't too many, You could index new events, create a lookup with a scheduled search and setting retention to 24 hours delete events earlier that eretention period.