Getting Data In

Best way to index .csv files with some common fields

simonbuskens
Engager

Hi
I have a series of .csv files (1 for each month) where the first 100 fields are the same, but after that there are about 4 or 5 fields that are specific to that month only.

What is the best way to add that data into Splunk?

Will I be able to add them all to the same index, or do they need their own index (so then I have to search across them all)

Tags (3)
1 Solution

esix_splunk
Splunk Employee
Splunk Employee

For your sourcetype, define the fieldnames in the header in all of your csv's in props.conf.

Then, define additional fields in that header for the sourcetype. E.g.,

fields = alwaysherefield1, alwaysherefield2, alwaysherefield...30, sometimesherefield31, sometimesherefield32, sometimesherefield33

If those fields dont exists, there wont be a value set. In the CSVs that have the fields, the values will be set. Then in your search, you can rename the fields to whatever you want them to be.

Refer to here for props.conf examples :
http://docs.splunk.com/Documentation/Splunk/6.2.1/Data/Extractfieldsfromfileheadersatindextime

View solution in original post

esix_splunk
Splunk Employee
Splunk Employee

For your sourcetype, define the fieldnames in the header in all of your csv's in props.conf.

Then, define additional fields in that header for the sourcetype. E.g.,

fields = alwaysherefield1, alwaysherefield2, alwaysherefield...30, sometimesherefield31, sometimesherefield32, sometimesherefield33

If those fields dont exists, there wont be a value set. In the CSVs that have the fields, the values will be set. Then in your search, you can rename the fields to whatever you want them to be.

Refer to here for props.conf examples :
http://docs.splunk.com/Documentation/Splunk/6.2.1/Data/Extractfieldsfromfileheadersatindextime

asifhj
Path Finder

Files name are same or different? if the file name are different you can put in same index and can write queries according to source field.

0 Karma

kml_uvce
Builder

you can add in same index and source name will be different and you can search based on source name

0 Karma

simonbuskens
Engager

won't the source name be the same (if they are saved in the same folder)

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Source name will be path+filename, sourcetype will be whatever you defined it to be.

0 Karma

asifhj
Path Finder

Nope, folders cannot have two files with same name.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...