topic Re: importing all the files available in a directory in Getting Data In

importing all the files available in a directory

abhayneilam — Mon, 08 Oct 2012 13:34:20 GMT

Hi,

Can any body tell me how to import all the files of a particular directory in splunk at one go ?
next time if I keep any other file in the same directory it should be automatically imported in the splunk, no need to import it manually..

Kindly get me the solution asap, as I am in urgent need of this..

Thanks in Advance,
Abhay

Re: importing all the files available in a directory

sdaniels — Mon, 08 Oct 2012 13:37:49 GMT

Look at the docs here:

http://docs.splunk.com/Documentation/Splunk/5.0/Data/UseSplunkWeb

You'll want -> Continuously index data from a file or directory this Splunk instance can access

Re: importing all the files available in a directory

abhayneilam — Mon, 08 Oct 2012 13:42:00 GMT

I am going to this option and giving the input as following :

D:\TEST

and clicking on SAVE button..

In this case only the first file in TEST directory is taking.. other file it is not taking..

Please suggest !!

Re: importing all the files available in a directory

abhayneilam — Mon, 08 Oct 2012 14:49:13 GMT

I am going to this option and giving the input as following : D:\TESTand clicking on SAVE button..In this case only the first file in TEST directory is taking.. other file it is not taking..Please suggest !!

Re: importing all the files available in a directory

sdaniels — Mon, 08 Oct 2012 16:57:01 GMT

What type of file is it that's not getting picked up from that directory? What is the size of the file?

Re: importing all the files available in a directory

abhayneilam — Tue, 09 Oct 2012 04:41:27 GMT

Hi,

This is a file which is "|" separated which contains 14 columns. The first line is header and rest of the lines are the values. I have five files in a directory: 104KB, 18KB, 69KB, 63KB and 8KB size of files...It is taking only the first file..please suggest how to get this task done...

Thanks in advance,
Abhay

Re: importing all the files available in a directory

abhayneilam — Tue, 09 Oct 2012 07:44:05 GMT

I am using a trial version ? Is there any limitation for this ?

Re: importing all the files available in a directory

sdaniels — Tue, 09 Oct 2012 12:14:32 GMT

Trial version is not the problem. Can you post the inputs.conf settings in:

$SPLUNK_HOME/etc/system/local/

You can see the settings and options for file monitoring here:

http://docs.splunk.com/Documentation/Splunk/latest/admin/inputsconf

Re: importing all the files available in a directory

abhayneilam — Mon, 28 Sep 2020 12:35:43 GMT

I have not written any thing in the inputs.conf..kindly suggest me what to write ? but i was clicking on "Continuously index data from a file or directory this Splunk instance can access" this optioin while importing the entire directory. I am giving my directory name as c:\Test\New_Folder\USB_Data

I have created an Index manuaally called "usb_data" and creating source type at the time of importing data manually.. Kindly suggest me how it can be done through configuration file or through any other way...

Please Help !!

Thanks,
Abhay

Re: importing all the files available in a directory

sdaniels — Tue, 09 Oct 2012 18:18:50 GMT

Look in the directory in my comment and look at the inputs.conf file that was created for you. Post the stanza in your question

Re: importing all the files available in a directory

abhayneilam — Mon, 28 Sep 2020 12:35:46 GMT

[monitor:://c:\Test\New_Folder\USB_Data]
index = usb_data
sourcetype = USB

Re: importing all the files available in a directory

sdaniels — Mon, 28 Sep 2020 12:35:49 GMT

Change the file to this and restart Splunk. I assume the index has been properly created?

[monitor:://c:\Test\New_Folder\USB_Data]
disabled=0
crcSalt=
index = usb_data
sourcetype = USB

-->(the word source should be in caps)

Re: importing all the files available in a directory

abhayneilam — Tue, 09 Oct 2012 18:52:43 GMT

you meand to say crcSalt=
or

SOURCEtype = USB

and Do I need to configure props.conf for this also ?

please suggest !!

Thanks
Abhay

Re: importing all the files available in a directory

sdaniels — Tue, 09 Oct 2012 18:54:34 GMT

First one...yes. Nothing in props.conf

Re: importing all the files available in a directory

abhayneilam — Mon, 28 Sep 2020 12:35:51 GMT

I am trying to extract the fields of my files :

I have two files which is of same type means : file_one contains: name|age|sex|location
xyz|45|M|kol
mno|50|F|mum

and file_two contains:
name|age|sex|location
abc|60|M|hyd
lkg|100|M|ker

these two files are in the same directory, and I am extracting the fields: name age sex location by the following method:

index="usb_data" | extract transform_usb_data

when I am giving this I am getting all the fields are getting extracted but can you suggest me how to automated this process from transfoms.conf file

Re: importing all the files available in a directory

abhayneilam — Mon, 28 Sep 2020 12:35:54 GMT

my transforms.conf contains :

[transform_usb_data]
delims = "|"
fields = "name","age","sex","location"

Re: importing all the files available in a directory

sdaniels — Tue, 09 Oct 2012 19:15:44 GMT

Please start a new question in the future. I assume you are getting multiple files indexed correctly?

props.conf

[USB]
SHOULD_LINEMERGE = false
KV_MODE = none
REPORT-my_fields = my_fields

transforms.conf:

[my_fields]
DELIMS="|"
FIELDS = "name", "age", "sex", "location"

Here's a previous splunk answers on this:

http://splunk-base.splunk.com/answers/3000/using-delims-to-extract-fix-data