topic Re: Overwrite index on each local csv file data pulling in Getting Data In

Overwrite index on each local csv file data pulling

cpm003 — Thu, 03 Jun 2021 13:20:36 GMT

I have a need to overwrite an index every time a continously monitored local csv file is modified.

This index should only have the total number of entries in the csv file at any given time, without being incremental.

How could I achieve it?

Thanks!

Re: Overwrite index on each local csv file data pulling

aasabatini — Thu, 03 Jun 2021 13:59:33 GMT

Hi @cpm003

I think the easier way to store the CSV data is:

Create a lookup based on this csv file
If the csv file is very huge consider to use a KV_STORE

Anyway, If for you is mandatory collect csv events on one Index, you need to consider to delete the data on the index everytime (you can set retention 1 day, or schedule a search with delete options) and schedule a scripted input (run scheduling 1 time per day) to recollect the data day by day.

Re: Overwrite index on each local csv file data pulling

cpm003 — Thu, 03 Jun 2021 14:09:49 GMT

Thank you @aasabatini

Is it in this window where to set the retention? Any implications that I should consider when establishing retention? (not counting the obvious removal of the data from the index?)

Re: Overwrite index on each local csv file data pulling

aasabatini — Thu, 03 Jun 2021 15:11:16 GMT

Hi @cpm003

the retention is configured on the indexes.conf file

example:

[main] frozenTimePeriodInSecs = 15552000

you need to set the data retention in seconds

https://docs.splunk.com/Documentation/Splunk/8.2.0/Indexer/Setaretirementandarchivingpolicy

if this reply help, please confirm solution or add karma points

Regards

Re: Overwrite index on each local csv file data pulling

cpm003 — Fri, 04 Jun 2021 05:49:15 GMT

Hello @aasabatini

After setting the retention for the index to 20 seconds I get the index to empty but only once, after reloading data in it they remain as you see in the capture since yesterday.

Re: Overwrite index on each local csv file data pulling

aasabatini — Fri, 04 Jun 2021 06:12:16 GMT

Hi @cpm003

Can I know whats inputs method you used?

script or monitor?

Re: Overwrite index on each local csv file data pulling

cpm003 — Fri, 04 Jun 2021 06:13:45 GMT

I´m monitoring an csv file continously.

Re: Overwrite index on each local csv file data pulling

aasabatini — Fri, 04 Jun 2021 06:45:43 GMT

Hi @cpm003

as I suggested, you need to create a scripted input

script example

#!/bin/bash # read csv file cat /[path]/[name].csv

set to run the script only once time per day and set the retention only for one day

https://helgeklein.com/blog/2015/03/splunk-scripted-input-secrects/

Re: Overwrite index on each local csv file data pulling

cpm003 — Fri, 04 Jun 2021 06:50:05 GMT

Thanks @aasabatini

I´ll check right now

Re: Overwrite index on each local csv file data pulling

cpm003 — Fri, 04 Jun 2021 07:04:38 GMT

Hello Again @aasabatini,

Created an script to read local csv and index their content on an index called "cduinventario" executed each 10seconds

Configured retention at the end of /opt/splunk/etc/system/default/indexes.conf with:

[cduinventario]
frozenTimePeriodInSecs = 5

Result:

Index doesn´t stop growing each script reads csv file, retention is not working as expected.

Re: Overwrite index on each local csv file data pulling

aasabatini — Fri, 04 Jun 2021 07:10:12 GMT

Hi @cpm003

I wrote:

"run the script only once time per day"

if you scheduling the script run each 10 seconds is normal the index increase continuously.

script run once time per day
retention set on 8400 seconds (1 day)

if you need to monitor continuously your csv, I suggested to use lookup or kvstore method.

hope can help

Alessandro