Getting Data In

How to index multiple CSV file from a local machine?

Explorer

Hi All,

I have multiple CSV files which are on the local machine under the same directory. I would like to add these files and index them.
I have multiple CSV files with fields let's say.

  1. table_1.csv with fields 'Bus No', 'Booking ID', 'start Time', 'End Time', 'Source', 'Destination'
  2. table_2.csv with fields 'Bus No', 'Booking ID', 'Via','Halting Timings','Passenger counts', 'Failures', etcs.
  3. table_3.csv with fields ....

Can someone guide me the best way to do it?

Thanks In Adv.

0 Karma
1 Solution

Contributor

Maybe monitoring the folder? Are the files dynamic or static?

View solution in original post

Motivator

Create the index ahead of time (settings->index->new)

Then in your inputs.conf file on the universal forwarder put in something like this

[batch:///var/nfs/SAT_SplunkLogs/ts/beta/*.csv]
move_policy = sinkhole
host_segment=5
sourcetype=csv
index=betats

then restart splunk on the universal forwarder

FYI . you can substitute monitor for batch, the move_policy = sinkhole will delete the CSV file after it is indexed

0 Karma

Explorer

@dbcase: Is this for dynamic CSV file?

0 Karma

Motivator

for dynamic (meaning the file gets added to) I'd use something like this

[monitor:///var/nfs/SAT_SplunkLogs/version/*.csv]
crcSalt = defprof
sourcetype=csv
index=allmsos
0 Karma

Motivator

If the csv file gets created, then indexed then a new file gets created I'd use the batch method.

The batch method with the move_policy sinkhole parameter will index the csv file, then delete it so a new csv file can be written.

0 Karma

SplunkTrust
SplunkTrust

If an answer to your question has solved your issue, please accept the answer.

0 Karma

Contributor

Maybe monitoring the folder? Are the files dynamic or static?

View solution in original post

Explorer

@felipesewaybricker: Thanks for responding my query! In both the cases i.e if CSV files are 1. static and 2. dynamic.

0 Karma

Contributor

Nice, you can monitor the folder, send to the same index and perform the searches as follows: index = nnn source = file.csv

0 Karma

Explorer

@felipesewaybricker: This is just for clarification, Are you saying we need to use "Monitoring" option instead of "Upload" option while uploading files. If this is Yes then how will i be creating Index for that ?

  1. what about if csv files are dynamic ?
0 Karma