How to automatically upload a CSV file I receive daily from my Mac Desktop to Splunk and either refresh the data or add deltas?

[kgreat]

I've installed the universal forwarder on my mac now I want to automatically send a csv file from a folder on my Mac to my Splunk instance. I just want to add any deltas.

Thanks!

[kgreat · Accepted solution]

Right now the daily csv files are about 13MB in size. I receive one per day via email where I can save locally on my computer in a folder called "DailyUserStatus". I've only uploaded at least two of these files manually to Splunk.

[musskopf]

Ok, if you receive via e-mail you'll need to manually (or create a script) to get the file out from there. Once you save it on your computer, just try the command:

$SPLUNK_HOME/bin/splunk add oneshot yourfile.csv  -index yourindex -sourcetype csv  -hostname yourmachine.yourdomain -auth "admin:changeme"

It should replicate the same process as uploading the file using the GUI. If that works you should be able to configure the command to run via Crontab every day.

ps.: 13 Mb is very small and shouldn't be a problem loading it every day.

[kgreat]

It's a list of all users in the system going back let's say three years to date that are either active or terminated. So new active users will be added to the list and existing users can change from "active" to "terminated". If there is a way to just replace the older file with the newer file that would work too.

[musskopf]

How big is this file? My first approach would be importing the whole file every day, using, lets say a crontab to call the "one shot" method.

In Splunk you would have the historical changes, a snapshot per day. You should be able to easily plot timecharts, or simply create a view for the last imported data.

[musskopf]

is this file incremental? Like, is only adding new data to the end of it? Or things might change everywhere?