Getting Data In

How to index multiple CSV file from a local machine?

ibmrakesh
Explorer

Hi All,

I have multiple CSV files which are on the local machine under the same directory. I would like to add these files and index them.
I have multiple CSV files with fields let's say.

  1. table_1.csv with fields 'Bus No', 'Booking ID', 'start Time', 'End Time', 'Source', 'Destination'
  2. table_2.csv with fields 'Bus No', 'Booking ID', 'Via','Halting Timings','Passenger counts', 'Failures', etcs.
  3. table_3.csv with fields ....

Can someone guide me the best way to do it?

Thanks In Adv.

0 Karma
1 Solution

felipesewaybric
Contributor

Maybe monitoring the folder? Are the files dynamic or static?

View solution in original post

dbcase
Motivator

Create the index ahead of time (settings->index->new)

Then in your inputs.conf file on the universal forwarder put in something like this

[batch:///var/nfs/SAT_SplunkLogs/ts/beta/*.csv]
move_policy = sinkhole
host_segment=5
sourcetype=csv
index=betats

then restart splunk on the universal forwarder

FYI . you can substitute monitor for batch, the move_policy = sinkhole will delete the CSV file after it is indexed

0 Karma

ibmrakesh
Explorer

@dbcase: Is this for dynamic CSV file?

0 Karma

dbcase
Motivator

for dynamic (meaning the file gets added to) I'd use something like this

[monitor:///var/nfs/SAT_SplunkLogs/version/*.csv]
crcSalt = defprof
sourcetype=csv
index=allmsos
0 Karma

dbcase
Motivator

If the csv file gets created, then indexed then a new file gets created I'd use the batch method.

The batch method with the move_policy sinkhole parameter will index the csv file, then delete it so a new csv file can be written.

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

If an answer to your question has solved your issue, please accept the answer.

0 Karma

felipesewaybric
Contributor

Maybe monitoring the folder? Are the files dynamic or static?

ibmrakesh
Explorer

@felipesewaybricker: Thanks for responding my query! In both the cases i.e if CSV files are 1. static and 2. dynamic.

0 Karma

felipesewaybric
Contributor

Nice, you can monitor the folder, send to the same index and perform the searches as follows: index = nnn source = file.csv

0 Karma

ibmrakesh
Explorer

@felipesewaybricker: This is just for clarification, Are you saying we need to use "Monitoring" option instead of "Upload" option while uploading files. If this is Yes then how will i be creating Index for that ?

  1. what about if csv files are dynamic ?
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...