Hi, I wish to import data from a folder structure and cannot find or understand how to do this.
I have over a hundred folders with five distinct .gz files in each. I wish to import the contents of one of these files from each folder into SPLUNK for analyses. I will not need to monitor these folders again.
I have gone into the Data Inputs >> Files & Folders and created a DataInput, I chose "Once Only".
In the Files & directories listing my new input shows up and has Number of files = 130 . I am unable to find how do I import and index them. I had expected them to show up somewhere in the Data Summary screen, am I missing something?
Creating a folder input will index all the files in that folder. If you only wish to import a single file from each folder, I suggest writing a script around the oneshot command
(/opt/splunkforwarder/bin/splunk add oneshot filename -index foo -sourcetype bar -hostname localhost.localdomain -auth "admin:changeme"). See the docs at http://docs.splunk.com/Documentation/Splunk/latest/Data/MonitorfilesanddirectoriesusingtheCLI.
/opt/splunkforwarder/bin/splunk add oneshot filename -index foo -sourcetype bar -hostname localhost.localdomain -auth "admin:changeme"
So for my case I replace, am I correct to say
filename - will be the file name I wish to import
foo - create an index e.g.. my index
bar - unique identifier (can be anything)
localhost.localdomain - route to the folder structure I wish to interrogate.
Am I right with these comments?
As a newbe I appreciate your assistance
Almost right. Sourcetype can be just about anything, but you should use a built-in sourcetype if one matches your data. See http://docs.splunk.com/Documentation/Splunk/latest/Data/Listofpretrainedsourcetypes. The hostname parameter should be the name of the system the files are on.
I think it is the hostname that stopping the command running, I don't fully understand it,. If my folder is for example
c:\folders\test\ what would my hostname be?
Hostname can be just about anything. DNS name (or part of it) is one option. In your case you might use "Scan001-PC" or whatever name Windows calls your computer. You can even omit the -hostname option.
The script is giving a "No results found"
my complete script is:
/opt/splunkforwarder/bin/splunk add oneshot auth-detail.gz -index wifi -sourcetype My-PC -hostname /Users/Philip/Dropbox/Projects/Access_Logs -auth "admin:changeme"
Did I misinterpret something along the way?
Allow me to suggest a few changes
/opt/splunkforwarder/bin/splunk add oneshot auth-detail.gz -index wifi -sourcetype gzip -hostname philip -auth "admin:changeme"
Depending on the nature of the data within the .gz files, you may want to consider unzipping the files within your script and then indexing the uncompressed data. You could then specify a sourcetype that better describes the contents.
If you need the folder names, try specifying the full path of the input file (/Users/Philip/Dropbox/Projects/Access_Logs/auth-detail.gz).