How to load a CSV file into Splunk & Change the csv file dynamically, so that data is refreshed.

[alenseb]

Hi All,

I am trying to load a .csv file into splunk, using sourcetype(csv). Upload of data is working fine but the only issue if i change the data in this .csv file, new data get appended to the old data. what i need is to replace the old with new.
Example: if i have 4 lines in the csv initially & if i delete one of its line and add 2 new lines.
My new csv data should contain only 4 lines.
Right now when i am doing this, i am getting 9 lines(4-Old & 5-New) of data.

Please help me with the configurations.

[Paolo_Prigione · Accepted solution]

CSVs the way you mean them are treated in a different way than regular log files.
There are 2 basic kinds:

1. "just CSVs", which are only accessed via "| inputcsv" and "| outputcsv"
2. lookup CSVs, which are accessed with commands "| lookup", "| inputlookup" and "| outputlookup"

In the former case you can:

1. load your CSV in $SPLUNK_HOME/var/run/splunk/filename.csv

2. execute the search:

   | inputcsv filename.csv

In the latter you can:

1. configure it from the Manager and upload the csv file
2. you will be able to replace the file under this directory $SPLUNK_HOME/etc/apps/APPNAME/lookups/filename.csv.

3. execute searches like:

   .... | lookup configname inputfield OUTPUT outputfield

[Paolo_Prigione]

It will probably will be slower than with the "| lookup" command and limited to some 50k results, but | join joinfield [|inputcsv ... | fields + joinfield otherfield] might do

[alenseb]

Thanks!! Appending worked Just fine!

Can i use this "| inputcsv" command to Join with another log file?

[MHibbin]

Do you not want to do this as a lookup?... http://docs.splunk.com/Documentation/Splunk/latest/User/CreateAndConfigureFieldLookups

If not you should be able to use the dedup command to show the latest event (i.e. the latest csv) depending on how you are indexing it.

MHIbbin

[alenseb]

Thanks in advance!!

[alenseb]

I tried the lookup but it didn't serve my purpose since the replacement of data wasn't happening.
Also i need to join this .csv file with another log file.
Therefore wen the data in csv file change, the result of the Join Query also should change.
Any Configuration files i need to look into. I am basically trying to automate it.

[jpmackl]

Any luck completing this effort? I'm trying to do the same thing. I have tool the produces a CSV report every hour, I would like to pull the data into Splunk in an automated fashion and then build real time dashboards from the data. Thank you for any help!

[colinmchugo]

Any luck with this jpmackl ? I want to do something similar