I have a .csv file that is just over 2GB, I noticed that the lookup table could only handle 500MB or less, so I reduced the amount of rows and was able to upload it.
Now, I'm trying to figure out how to show that data in a Splunk search and I don't really have any idea where to start.
Any guidance or help is much appreciated.
I'm not sure I understand the question fully. You uploaded data into Splunk, you then uploaded a lookup table. You then need to set up the field lookup definition
so you can correlate fields to the lookup table, then you need to make the lookup table automatic. Can you give us more information as to what your trying to accomplish?
http://docs.splunk.com/Documentation/Splunk/6.1.3/SearchTutorial/Usefieldlookups
Send it in as data
with an ad-hoc upload instead of as a lookup
like this:
Settings
-> Add Data
(the large icon in the upper left corner of the menu dialog) -> Upload
Try for a listing of everything in the lookup file
| inputlookup yourcsvfile.csv
When I uploaded my .csv I named it ViaTest, and it shows it in the Lookup Table Files settings, but when I tried running that command, it says "The lookup table 'ViaTest' is invalid
Try ViaTest.csv
I'm not sure I understand the question fully. You uploaded data into Splunk, you then uploaded a lookup table. You then need to set up the field lookup definition
so you can correlate fields to the lookup table, then you need to make the lookup table automatic. Can you give us more information as to what your trying to accomplish?
http://docs.splunk.com/Documentation/Splunk/6.1.3/SearchTutorial/Usefieldlookups
The first thing I did was upload the .csv as a Lookup table, there wasn't anywhere to "upload" the data into Splunk otherwise.
When I go to the Lookup definitions page, and try to select the App Context that I created the Lookup Table in -- it gives me a 500 Internal Server error.
Sorry i'm not very experienced with Splunk, I was given a .csv file by someone and told to import it to Splunk so I could write a query for it, etc.
Ok, I think I understand what you mean. So you had a CSV file that you wanted to upload into Splunk so you could run queries against it and return data. I'm assuming that you didn't want a lookup table but rather a way to upload data into Splunk.. If this is the case then there's a few ways of doing this..
You could upload the data once by going to the GUI Settings>Add Data
or you could run a simple oneshoot command to get it uploaded (See below).. If you have to contineously upload the data then I would recommend installing a Splunk forwarder and have it monitor that data so it can index it in real time
Windows
Open PowerShell or CMD as Admin
cd Splunk_Home\bin
.\splunk add oneshot C:\Program Files\AppLog\log.txt
Linux
cd Splunk_home/bin
./splunk add oneshot /var/log/applog
The oneshoot command will tell this that it should not monitor the file and that it should only upload it one time which is when you run the command
What is the difference in uploading the data vs. creating a lookup table if you don't mind me asking?
I tried uploading it via the GUI first off, but the "Add Data" option isn't available to me under settings, something to do with what permissions I have access to, i'll have to ask someone higher up to do it for me I suppose. In that case they would have to be the one to install the Splunk forwarder too then?
I'm on OS X so the oneshoot cmd isn't an option I guess.
OSX shell is the same as Linux so you can run the Linux command to do the oneshoot. If you have access to the host machine then you can install a forwarder and point it to the indexer and it should get the data flowing in
As for the lookup, I will give you an example.. Say you have a process that has 2 return codes 3
AND 4
.. So say you have no idea what these return codes mean, but you also don't want to change the logging style but you also want a clear way to communicate what these return codes are. You can upload a lookup table in Splunk to correlate the return values to actual names.. So say 3
represents pass
and 4
represents fail
.. You can have the fields return Pass
and Fail
rather than the numbers without changing your logs..
Okay interesting...thanks a bunch for explaining that.
I've got someone else with permissions to upload the entire csv for me (~2GB), so now I need to add new Lookup definitions defining the fields I want?
If you want to add a lookup table to your logs then yes. What exactly are you trying to accomplish?
This article walks you through step by step
http://docs.splunk.com/Documentation/Splunk/6.1.3/SearchTutorial/Usefieldlookups