Getting Data In

How to index multiple CSV file from a local machine?

ibmrakesh
Explorer

Hi All,

I have multiple CSV files which are on the local machine under the same directory. I would like to add these files and index them.
I have multiple CSV files with fields let's say.

  1. table_1.csv with fields 'Bus No', 'Booking ID', 'start Time', 'End Time', 'Source', 'Destination'
  2. table_2.csv with fields 'Bus No', 'Booking ID', 'Via','Halting Timings','Passenger counts', 'Failures', etcs.
  3. table_3.csv with fields ....

Can someone guide me the best way to do it?

Thanks In Adv.

0 Karma
1 Solution

felipesewaybric
Contributor

Maybe monitoring the folder? Are the files dynamic or static?

View solution in original post

dbcase
Motivator

Create the index ahead of time (settings->index->new)

Then in your inputs.conf file on the universal forwarder put in something like this

[batch:///var/nfs/SAT_SplunkLogs/ts/beta/*.csv]
move_policy = sinkhole
host_segment=5
sourcetype=csv
index=betats

then restart splunk on the universal forwarder

FYI . you can substitute monitor for batch, the move_policy = sinkhole will delete the CSV file after it is indexed

0 Karma

ibmrakesh
Explorer

@dbcase: Is this for dynamic CSV file?

0 Karma

dbcase
Motivator

for dynamic (meaning the file gets added to) I'd use something like this

[monitor:///var/nfs/SAT_SplunkLogs/version/*.csv]
crcSalt = defprof
sourcetype=csv
index=allmsos
0 Karma

dbcase
Motivator

If the csv file gets created, then indexed then a new file gets created I'd use the batch method.

The batch method with the move_policy sinkhole parameter will index the csv file, then delete it so a new csv file can be written.

0 Karma

DalJeanis
Legend

If an answer to your question has solved your issue, please accept the answer.

0 Karma

felipesewaybric
Contributor

Maybe monitoring the folder? Are the files dynamic or static?

ibmrakesh
Explorer

@felipesewaybricker: Thanks for responding my query! In both the cases i.e if CSV files are 1. static and 2. dynamic.

0 Karma

felipesewaybric
Contributor

Nice, you can monitor the folder, send to the same index and perform the searches as follows: index = nnn source = file.csv

0 Karma

ibmrakesh
Explorer

@felipesewaybricker: This is just for clarification, Are you saying we need to use "Monitoring" option instead of "Upload" option while uploading files. If this is Yes then how will i be creating Index for that ?

  1. what about if csv files are dynamic ?
0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...