Lookups and .csv files

[tkwaller]

A somewhat basic question as I have not done this often. I have many .cvs files I would like to get indexed in Splunk. I have them stored on a nas storage on a box we will call box1. I want the data searchable so I believe the lookup should go into the $SPLUNKHOME/etc/apps/app/lookups path on the search head and the necessary info into the props and transforms configs on the search head.

The .csv files however are NOT accessible to the search head as the nas storage is not shared there.

My question is, how would I create the app and where should I deploy it to solve for this issue?
Do I create the app and just the monitor stanza for the .cvs files and deploy it to box1 and add the lookup, props and transforms configs to the search head or is there a better solution?

[jbjerke_splunk]

Hi tkwaller

The recommended approach would be to install a Splunk Universal Forwarder (UF) on box1.
Configure the UF to send its data to the Splunk Indexer (which can also be the Splunk search head on single instance installs). This is done by modifying the outputs.conf file or by specifying the IP of the Splunk Indexer when you install the Splunk UF (windows only).
http://docs.splunk.com/Documentation/Splunk/latest/admin/outputsconf

The stanza should look something like this:

[tcpout]
defaultGroup=my_indexers

[tcpout:my_indexers]
server=mysplunk_indexer1:9997, mysplunk_indexer2:9996

[tcpout-server://mysplunk_indexer1:9997]

You also need to enable receiving on the Splunk Indexer. This can be done through the GUI under Settings->Forwarding and Receiving

After this is done you should configure a file monitoring input on the UF that you point to the folder where you store the csv files. This is done by creating an inputs.conf file.

http://docs.splunk.com/Documentation/Splunk/latest/admin/inputsconf

The stanza should look something like this:

[monitor:///var/log/httpd]
sourcetype = csv

[sjohnson_splunk · Accepted solution]

The reason for having csv files in a lookups directory is so that you can use the contents of the csv to provide data enrichment (usually to some other data source). If all you want is to make the csv data searchable, then all you have to do is index the csv files.

If you want to turn them into lookup tables then you will need to do a couple more steps (assuming you can't get them to your search head directly).

I'll give you an outline of the steps you need to go thru:

1. create an app to monitor the .csv files on the NAS
2. create a search to return the data in a table format
3. use the ouputlookup command to create a lookup table inside of some app

[monitor:///mountpoint/my_data.csv]
sourcetype = some_csv
index = test

| index=test sourcetype=some_csv | table | outputlookup lookup name