topic Re: How to index multiple CSV file from a local machine? in Getting Data In

How to index multiple CSV file from a local machine?

ibmrakesh — Wed, 08 Feb 2017 17:03:52 GMT

Hi All,

I have multiple CSV files which are on the local machine under the same directory. I would like to add these files and index them.
I have multiple CSV files with fields let's say.

table_1.csv with fields 'Bus No', 'Booking ID', 'start Time', 'End Time', 'Source', 'Destination'
table_2.csv with fields 'Bus No', 'Booking ID', 'Via','Halting Timings','Passenger counts', 'Failures', etcs.
table_3.csv with fields ....

Can someone guide me the best way to do it?

Thanks In Adv.

Re: How to index multiple CSV file from a local machine?

felipesewaybric — Wed, 08 Feb 2017 17:40:42 GMT

Maybe monitoring the folder? Are the files dynamic or static?

Re: How to index multiple CSV file from a local machine?

ibmrakesh — Thu, 09 Feb 2017 13:16:10 GMT

@felipesewaybricker: Thanks for responding my query! In both the cases i.e if CSV files are 1. static and 2. dynamic.

Re: How to index multiple CSV file from a local machine?

felipesewaybric — Thu, 09 Feb 2017 14:15:35 GMT

Nice, you can monitor the folder, send to the same index and perform the searches as follows: index = nnn source = file.csv

Re: How to index multiple CSV file from a local machine?

DalJeanis — Thu, 09 Feb 2017 17:10:04 GMT

If an answer to your question has solved your issue, please accept the answer.

Re: How to index multiple CSV file from a local machine?

ibmrakesh — Fri, 10 Feb 2017 19:46:20 GMT

@felipesewaybricker: This is just for clarification, Are you saying we need to use "Monitoring" option instead of "Upload" option while uploading files. If this is Yes then how will i be creating Index for that ?

what about if csv files are dynamic ?

Re: How to index multiple CSV file from a local machine?

dbcase — Fri, 10 Feb 2017 19:57:45 GMT

Create the index ahead of time (settings->index->new)

Then in your inputs.conf file on the universal forwarder put in something like this

[batch:///var/nfs/SAT_SplunkLogs/ts/beta/*.csv]
move_policy = sinkhole
host_segment=5
sourcetype=csv
index=betats

then restart splunk on the universal forwarder

FYI . you can substitute monitor for batch, the move_policy = sinkhole will delete the CSV file after it is indexed

Re: How to index multiple CSV file from a local machine?

ibmrakesh — Fri, 17 Feb 2017 17:17:04 GMT

@dbcase: Is this for dynamic CSV file?

Re: How to index multiple CSV file from a local machine?

dbcase — Fri, 17 Feb 2017 17:18:41 GMT

for dynamic (meaning the file gets added to) I'd use something like this

[monitor:///var/nfs/SAT_SplunkLogs/version/*.csv]
crcSalt = defprof
sourcetype=csv
index=allmsos

Re: How to index multiple CSV file from a local machine?

dbcase — Fri, 17 Feb 2017 17:23:12 GMT

If the csv file gets created, then indexed then a new file gets created I'd use the batch method.

The batch method with the move_policy sinkhole parameter will index the csv file, then delete it so a new csv file can be written.