topic Re: Update existing index with CSV files containing changes in Getting Data In

Update existing index with CSV files containing changes

wredny125 — Thu, 09 Jul 2015 21:36:13 GMT

Hi Guys,

I have a case where I'm importing every week a new dump of a data base to Splunk index using CSV files (I'm deleting the index with old data, and then recreating it back with new once). Right now I will receive not the whole dump of the DB in CSV but only CSV files with the items that have recently changed or have been added.

What would be the best way to update my existing index with this incremental csv files in Splunk and how could I do that?

Many thanks in advance

Re: Update existing index with CSV files containing changes

martin_mueller — Thu, 09 Jul 2015 21:59:49 GMT

Splunk doesn't do updates on index data, it's insert-only.

You could insert a new version of an event, and select the most recent version in your search, but that can be complicated and expensive.
You could store the entire set in a lookup file or Splunk's Key-Value-Store which both support updates but may or may not be suitable for your use case.
Have you considered leaving the data in an RDBMS and querying that from Splunk at search time using DB Connect?