importing all the files available in a directory

[abhayneilam]

Hi,

Can any body tell me how to import all the files of a particular directory in splunk at one go ?
next time if I keep any other file in the same directory it should be automatically imported in the splunk, no need to import it manually..

Kindly get me the solution asap, as I am in urgent need of this..

Thanks in Advance,
Abhay

[sdaniels]

Please start a new question in the future. I assume you are getting multiple files indexed correctly?

props.conf

[USB]
SHOULD_LINEMERGE = false
KV_MODE = none
REPORT-my_fields = my_fields

transforms.conf:

[my_fields]
DELIMS="|"
FIELDS = "name", "age", "sex", "location"

Here's a previous splunk answers on this:

http://splunk-base.splunk.com/answers/3000/using-delims-to-extract-fix-data

[abhayneilam]

I am going to this option and giving the input as following :

D:\TEST

and clicking on SAVE button..

In this case only the first file in TEST directory is taking.. other file it is not taking..

Please suggest !!

[sdaniels · Accepted solution]

Look at the docs here:

http://docs.splunk.com/Documentation/Splunk/5.0/Data/UseSplunkWeb

You'll want -> Continuously index data from a file or directory this Splunk instance can access

[abhayneilam]

my transforms.conf contains :

[transform_usb_data]
delims = "|"
fields = "name","age","sex","location"

[abhayneilam]

I am trying to extract the fields of my files :

I have two files which is of same type means : file_one contains: name|age|sex|location
xyz|45|M|kol
mno|50|F|mum

and file_two contains:
name|age|sex|location
abc|60|M|hyd
lkg|100|M|ker

these two files are in the same directory, and I am extracting the fields: name age sex location by the following method:

index="usb_data" | extract transform_usb_data

when I am giving this I am getting all the fields are getting extracted but can you suggest me how to automated this process from transfoms.conf file

[sdaniels]

First one...yes. Nothing in props.conf

[abhayneilam]

you meand to say crcSalt=
or

SOURCEtype = USB

and Do I need to configure props.conf for this also ?

please suggest !!

Thanks
Abhay

[sdaniels]

Change the file to this and restart Splunk. I assume the index has been properly created?

[monitor:://c:\Test\New_Folder\USB_Data]
disabled=0
crcSalt=
index = usb_data
sourcetype = USB

-->(the word source should be in caps)

[abhayneilam]

[monitor:://c:\Test\New_Folder\USB_Data]
index = usb_data
sourcetype = USB

[sdaniels]

Look in the directory in my comment and look at the inputs.conf file that was created for you. Post the stanza in your question

[abhayneilam]

I have not written any thing in the inputs.conf..kindly suggest me what to write ? but i was clicking on "Continuously index data from a file or directory this Splunk instance can access" this optioin while importing the entire directory. I am giving my directory name as c:\Test\New_Folder\USB_Data

I have created an Index manuaally called "usb_data" and creating source type at the time of importing data manually.. Kindly suggest me how it can be done through configuration file or through any other way...

Please Help !!

Thanks,
Abhay

[sdaniels]

Trial version is not the problem. Can you post the inputs.conf settings in:

$SPLUNK_HOME/etc/system/local/

You can see the settings and options for file monitoring here:

http://docs.splunk.com/Documentation/Splunk/latest/admin/inputsconf

[abhayneilam]

I am using a trial version ? Is there any limitation for this ?

[abhayneilam]

Hi,

This is a file which is "|" separated which contains 14 columns. The first line is header and rest of the lines are the values. I have five files in a directory: 104KB, 18KB, 69KB, 63KB and 8KB size of files...It is taking only the first file..please suggest how to get this task done...

Thanks in advance,
Abhay

[sdaniels]

What type of file is it that's not getting picked up from that directory? What is the size of the file?

[abhayneilam]

I am going to this option and giving the input as following : D:\TESTand clicking on SAVE button..In this case only the first file in TEST directory is taking.. other file it is not taking..Please suggest !!