topic Re: Trying to index a csv lookup table in Splunk Search

Trying to index a csv lookup table

zsizemore — Wed, 20 Jul 2016 14:12:50 GMT

I have a .csv file that is just over 2GB, I noticed that the lookup table could only handle 500MB or less, so I reduced the amount of rows and was able to upload it.

Now, I'm trying to figure out how to show that data in a Splunk search and I don't really have any idea where to start.

Any guidance or help is much appreciated.

Re: Trying to index a csv lookup table

skoelpin — Wed, 20 Jul 2016 14:18:55 GMT

I'm not sure I understand the question fully. You uploaded data into Splunk, you then uploaded a lookup table. You then need to set up the field lookup definition so you can correlate fields to the lookup table, then you need to make the lookup table automatic. Can you give us more information as to what your trying to accomplish?

http://docs.splunk.com/Documentation/Splunk/6.1.3/SearchTutorial/Usefieldlookups

Re: Trying to index a csv lookup table

sundareshr — Wed, 20 Jul 2016 14:24:11 GMT

Try for a listing of everything in the lookup file

| inputlookup yourcsvfile.csv

Re: Trying to index a csv lookup table

zsizemore — Wed, 20 Jul 2016 14:27:34 GMT

The first thing I did was upload the .csv as a Lookup table, there wasn't anywhere to "upload" the data into Splunk otherwise.

When I go to the Lookup definitions page, and try to select the App Context that I created the Lookup Table in -- it gives me a 500 Internal Server error.

Sorry i'm not very experienced with Splunk, I was given a .csv file by someone and told to import it to Splunk so I could write a query for it, etc.

Re: Trying to index a csv lookup table

skoelpin — Wed, 20 Jul 2016 14:35:15 GMT

Ok, I think I understand what you mean. So you had a CSV file that you wanted to upload into Splunk so you could run queries against it and return data. I'm assuming that you didn't want a lookup table but rather a way to upload data into Splunk.. If this is the case then there's a few ways of doing this..

You could upload the data once by going to the GUI Settings>Add Data or you could run a simple oneshoot command to get it uploaded (See below).. If you have to contineously upload the data then I would recommend installing a Splunk forwarder and have it monitor that data so it can index it in real time

Windows

Open PowerShell or CMD as Admin

cd Splunk_Home\bin

.\splunk add oneshot C:\Program Files\AppLog\log.txt

Linux

cd Splunk_home/bin
./splunk add oneshot /var/log/applog

The oneshoot command will tell this that it should not monitor the file and that it should only upload it one time which is when you run the command

Re: Trying to index a csv lookup table

zsizemore — Wed, 20 Jul 2016 14:35:33 GMT

When I uploaded my .csv I named it ViaTest, and it shows it in the Lookup Table Files settings, but when I tried running that command, it says "The lookup table 'ViaTest' is invalid

Re: Trying to index a csv lookup table

zsizemore — Wed, 20 Jul 2016 14:40:10 GMT

What is the difference in uploading the data vs. creating a lookup table if you don't mind me asking?

I tried uploading it via the GUI first off, but the "Add Data" option isn't available to me under settings, something to do with what permissions I have access to, i'll have to ask someone higher up to do it for me I suppose. In that case they would have to be the one to install the Splunk forwarder too then?

I'm on OS X so the oneshoot cmd isn't an option I guess.

Re: Trying to index a csv lookup table

sundareshr — Wed, 20 Jul 2016 14:46:56 GMT

Try ViaTest.csv

Re: Trying to index a csv lookup table

skoelpin — Wed, 20 Jul 2016 14:47:04 GMT

OSX shell is the same as Linux so you can run the Linux command to do the oneshoot. If you have access to the host machine then you can install a forwarder and point it to the indexer and it should get the data flowing in

As for the lookup, I will give you an example.. Say you have a process that has 2 return codes 3 AND 4.. So say you have no idea what these return codes mean, but you also don't want to change the logging style but you also want a clear way to communicate what these return codes are. You can upload a lookup table in Splunk to correlate the return values to actual names.. So say 3 represents pass and 4 represents fail.. You can have the fields return Pass and Fail rather than the numbers without changing your logs..

Re: Trying to index a csv lookup table

zsizemore — Wed, 20 Jul 2016 18:33:41 GMT

Okay interesting...thanks a bunch for explaining that.

I've got someone else with permissions to upload the entire csv for me (~2GB), so now I need to add new Lookup definitions defining the fields I want?

Re: Trying to index a csv lookup table

skoelpin — Wed, 20 Jul 2016 18:38:52 GMT

If you want to add a lookup table to your logs then yes. What exactly are you trying to accomplish?

This article walks you through step by step

http://docs.splunk.com/Documentation/Splunk/6.1.3/SearchTutorial/Usefieldlookups

Re: Trying to index a csv lookup table

woodcock — Wed, 20 Jul 2016 19:56:21 GMT

Send it in as data with an ad-hoc upload instead of as a lookup like this:

Settings -> Add Data (the large icon in the upper left corner of the menu dialog) -> Upload