i was monitor a folder continuously in which if a new file comes in the folder i want to index and show the latest file data only

[DataOrg]

i am monitor a folder continuously after indexing a file the old file will be removed and a new file will placed but dont to the old files data. i want the latest file data to displyed

[lloydknight]

If you want to delete an already indexed data and just view the latest, you can try to set a retention period with it.

https://answers.splunk.com/answers/107910/determine-index-retention-time.html
https://docs.splunk.com/Documentation/Splunk/6.6.1/Indexer/Setaretirementandarchivingpolicy

And if you want to delete the files you are monitoring after indexing it, you can use batch instead of monitor.

https://docs.splunk.com/Documentation/Splunk/6.6.2/Admin/Inputsconf

Consider this as a work-around since this doesn't directly addresses your concern.

Hope it helps.

[gcusello]

Hi premranjithj,
You can monitor files in a folder using inputs.conf

[monitor:///tmp/your_folder/*]
...

When one file is updated Splunk indexes only the new events.
If you delete a file and then you copy a new different version of it in the folder, it will be reindexed all.

What is the problem: in the new version of your file there are also old events?

Bye.
Giuseppe

[DataOrg]

hi @cusello
when new file is placed on the folder. i dont want the data indexed previously . i want only the new files data to be indexed and want the results of new file

[gcusello]

Hi premranjithj,
let me better understand:
you put files in a folder and they are indexed by Splunk
after you delete these files and put new ones in this folder,
Splunk indexes the new files.

What are the "old files data" that you say, you deleted the old ones?
If you put new files containing old data they will be reindexed again.

Bye.
Giuseppe

[DataOrg]

@cusello
old files i meant here is first file which is indexed.
if i keep the second file in the monitoring folder. i want the results of second file data only. the indexer should not have first file data

[gcusello]

Hi premranjithj,
a file is indexed once in Splunk, why you say that you don't want to reindex it? do your splunk reindex the old file?
when a new files are copied in a folder only the new ones are indexed the old one isn't indexed again.
If instead the old file is updated (events are added), only the new events are indexed.

Bye.
Giuseppe

[DataOrg]

sorry. its not reindexing but i dont want the file 1 data anymore when a second file is placed. so i dont want my index to retain the first file data

[gcusello]

Hi premranjithj,
In other words, do you want to delete old file after indexing?

Splunk doesn't delete files after indexing but iyou could create a batch script that deletes all files older that a period (e.g. one hour).
If you want to do a more scientific deletion, you could generate a report that lists all the sources already indexed (index=your_index, sourcetype=yoursourcetype) | dedup source | table source | outputcsv deletion_list.csv) and use it as input in your deletion batch.

Bye.
Giuseppe

[DataOrg]

yes i will delete the file using batch script. how could we delete or remove the old data in indexer.

[gcusello]

Maybe I'm starting to understand: do you want to remove old files on file system and also old events in the index?
to do this, set the retantion for the index you're using: e.g. if you want to load data on an index every day and delete all older events set a retention (frozenTimePeriodInSecs = 86400).
to be more sure you could use in your searches as period -24h and delete events ater 2 days.

Otherwise (better!) if you haven't too logs, you could load them in a lookup and nightly rebuild it.

Bye.
Giuseppe

[DataOrg]

i have a batch script to remove the old file in filesystem. i want to understand is there any way to remove the old file data from indexer when a new data gets indexed.

retention policy we can set in cold bucket but it will be help us here. may be two new files will be generated in some intervals of twice a day

[gcusello]

Hi premranjithj,
The only correct way to delete old events is to set retention in indexes.
there is another way but not correct, you could schedule deletion of old events, but remebering that in Splunk deletion is only logical and deleted events remain in indexes until retention period expires.

I don't know how many events you daily index, if they aren't too many, You could index new events, create a lookup with a scheduled search and setting retention to 24 hours delete events earlier that eretention period.

Bye.

Giuseppe