Getting Data In

Overwrite index on each local csv file data pulling

cpm003
Path Finder

I have a need to overwrite an index every time a continously monitored local csv file is modified.

This index should only have the total number of entries in the csv file at any given time, without being incremental.

How could I achieve it?

Thanks!

0 Karma
1 Solution

aasabatini
Motivator

Hi @cpm003 

I think the easier way to store the CSV data is:

  • Create a lookup based on this csv file
  • If the csv file is very huge consider to use a KV_STORE

Anyway, If for you is mandatory collect csv events on one Index, you need to consider to delete the data on the index everytime (you can set retention 1 day, or schedule a search with delete options) and schedule a scripted input (run scheduling 1 time per day) to recollect the data day by day.

“The answer is out there, Neo, and it’s looking for you, and it will find you if you want it to.”

View solution in original post

aasabatini
Motivator

Hi @cpm003 

I think the easier way to store the CSV data is:

  • Create a lookup based on this csv file
  • If the csv file is very huge consider to use a KV_STORE

Anyway, If for you is mandatory collect csv events on one Index, you need to consider to delete the data on the index everytime (you can set retention 1 day, or schedule a search with delete options) and schedule a scripted input (run scheduling 1 time per day) to recollect the data day by day.

“The answer is out there, Neo, and it’s looking for you, and it will find you if you want it to.”

cpm003
Path Finder

Thank you @aasabatini 

Is it in this window where to set the retention? Any implications that I should consider when establishing retention? (not counting the obvious removal of the data from the index?)

cpm003_0-1622729195784.png

 

0 Karma

aasabatini
Motivator

Hi @cpm003 

the retention is configured on the indexes.conf file

example:

[main]
frozenTimePeriodInSecs = 15552000

 you need to set the data retention in seconds

https://docs.splunk.com/Documentation/Splunk/8.2.0/Indexer/Setaretirementandarchivingpolicy

if this reply help, please confirm solution or add karma points

Regards

“The answer is out there, Neo, and it’s looking for you, and it will find you if you want it to.”
0 Karma

cpm003
Path Finder

Hello @aasabatini 

After setting the retention for the index to 20 seconds I get the index to empty but only once, after reloading data in it they remain as you see in the capture since yesterday.

cpm003_0-1622785503160.png

 

0 Karma

aasabatini
Motivator

Hi @cpm003 

Can I know whats inputs method you used?

script or monitor?

 

“The answer is out there, Neo, and it’s looking for you, and it will find you if you want it to.”
0 Karma

cpm003
Path Finder

I´m monitoring an csv file continously.

0 Karma

aasabatini
Motivator

Hi @cpm003 

as I suggested, you need to create a scripted input 

script example

#!/bin/bash
# read csv file
cat  /[path]/[name].csv

set to run the script only once time per day and set the retention only for one day

https://helgeklein.com/blog/2015/03/splunk-scripted-input-secrects/

“The answer is out there, Neo, and it’s looking for you, and it will find you if you want it to.”
0 Karma

cpm003
Path Finder

Thanks @aasabatini 

I´ll check right now

0 Karma

cpm003
Path Finder

Hello Again @aasabatini

Created an script to read local csv and index their content on an index called "cduinventario" executed each 10seconds

Configured retention at the end of /opt/splunk/etc/system/default/indexes.conf with:

[cduinventario]
frozenTimePeriodInSecs = 5

Result:

Index doesn´t stop growing each script reads csv file, retention is not working as expected. 

0 Karma

aasabatini
Motivator

Hi @cpm003 

I wrote:

"run the script only once time per day"

if you scheduling the script run each 10 seconds is normal the index increase continuously.

  • script run once time per day
  • retention set on 8400 seconds (1 day)

if you need to monitor continuously your csv, I suggested to use lookup or kvstore method.

hope can help

Alessandro

“The answer is out there, Neo, and it’s looking for you, and it will find you if you want it to.”
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...