Overwrite index on each local csv file data pulling

[cpm003]

I have a need to overwrite an index every time a continously monitored local csv file is modified.

This index should only have the total number of entries in the csv file at any given time, without being incremental.

How could I achieve it?

Thanks!

[aasabatini · Accepted solution]

Hi @cpm003 

I think the easier way to store the CSV data is:

• Create a lookup based on this csv file
• If the csv file is very huge consider to use a KV_STORE

Anyway, If for you is mandatory collect csv events on one Index, you need to consider to delete the data on the index everytime (you can set retention 1 day, or schedule a search with delete options) and schedule a scripted input (run scheduling 1 time per day) to recollect the data day by day.

[cpm003]

Thank you @aasabatini 

Is it in this window where to set the retention? Any implications that I should consider when establishing retention? (not counting the obvious removal of the data from the index?)

[aasabatini]

Hi @cpm003 

the retention is configured on the indexes.conf file

example:

[main]
frozenTimePeriodInSecs = 15552000

 you need to set the data retention in seconds

https://docs.splunk.com/Documentation/Splunk/8.2.0/Indexer/Setaretirementandarchivingpolicy

if this reply help, please confirm solution or add karma points

Regards

[cpm003]

Hello @aasabatini 

After setting the retention for the index to 20 seconds I get the index to empty but only once, after reloading data in it they remain as you see in the capture since yesterday.

[aasabatini]

Hi @cpm003 

Can I know whats inputs method you used?

script or monitor?

[cpm003]

I´m monitoring an csv file continously.

[aasabatini]

Hi @cpm003 

as I suggested, you need to create a scripted input 

script example

#!/bin/bash
# read csv file
cat  /[path]/[name].csv

set to run the script only once time per day and set the retention only for one day

https://helgeklein.com/blog/2015/03/splunk-scripted-input-secrects/

[cpm003]

Thanks @aasabatini 

I´ll check right now

[cpm003]

Hello Again @aasabatini, 

Created an script to read local csv and index their content on an index called "cduinventario" executed each 10seconds

Configured retention at the end of /opt/splunk/etc/system/default/indexes.conf with:

[cduinventario]
frozenTimePeriodInSecs = 5

Result:

Index doesn´t stop growing each script reads csv file, retention is not working as expected. 

[aasabatini]

Hi @cpm003 

I wrote:

"run the script only once time per day"

if you scheduling the script run each 10 seconds is normal the index increase continuously.

• script run once time per day
• retention set on 8400 seconds (1 day)

if you need to monitor continuously your csv, I suggested to use lookup or kvstore method.

hope can help

Alessandro