csv file header (1st tow) field extraction

[sim_tcr]

Hello,

I have a csv file in below format,

date,time,rundate
02/09/2016,00:00.0,02/07/2016
02/09/2016,00:00.0,02/07/2016
02/09/2016,00:00.0,02/07/2016
02/09/2016,00:00.0,02/07/2016

What should be the props.conf on indexers look like, so that each item in first row (separated by ,) should come as fields and items under that get indexed under that fields name?

Already tried,

[sourcetype-name]
INDEXED_EXTRACTIONS = csv
HEADER_FIELD_LINE_NUMBER = 1
HEADER_FIELD_DELIMITER = ,

Thanks,
Simon Mandy

[woodcock · Accepted solution]

Note that this props.conf must be deployed TO THE FORWARDER, not to the indexers because INDEXED_EXTRACTIONS is a very special case.

Perhaps you are misunderstanding how to verify if "it works". Once this goes to the forwarders and all of the splunk instances there have been bounced, ONLY events forwarded/indexed after the restart will be using the new configurations; previous data will stay wrong forever.

[lukejadamec]

I tested this log file content:

date,time,rundate
02/09/2016,00:00.0,2/07/2016
02/09/2016,00:00.1,2/07/2016
02/09/2016,00:00.2,2/07/2016
02/09/2016,00:00.3,2/07/2016
02/09/2016,00:00.4,2/07/2016
02/09/2016,00:00.5,2/07/2016
02/09/2016,00:00.6,2/07/2016
02/09/2016,00:00.7,2/07/2016

With these config files:

inputs.conf
[monitor://C:\temp\Splunk\test\csv-test\csv-test3.csv]
disabled = false
index = test
sourcetype = csvtest3

props.conf
[csvtest3]
NO_BINARY_CHECK = true
category = Custom
disabled = false
pulldown_type = true
REPORT-csvtest3 = REPORT-csvtest3

transforms.conf
[REPORT-csvtest3]
DELIMS = ","
FIELDS = "Date","Time","runDate"

Everything works fine, with the exception of the fractional minutes - strptime cannot compute HH:MM.M so you will get HH:MM:SS.SSS truncated to MM as _time for each event, i.e. log time 12:00.9 will equal event time 12:00:00.000.

If seconds are important, then you should ask another question on how to convert the Time field (string value extracted above) in a search to a time value that includes accurate seconds for sorting purposes.

[lukejadamec]

Is the time really HH:MM.M? There is no strptime variable for MM.M, so the best you're going to get is HH:MM

ex: strptime cannot compute 00:00.5 = 00:00:30, you can get 00:00.5 to represent as 00:00:05 but that is not accurate.

[PPape]

It should work like this:

[ mycsv ]
CHARSET=AUTO
INDEXED_EXTRACTIONS=csv
KV_MODE=none
SHOULD_LINEMERGE=false
category=Structured
disabled=false
pulldown_type=true
HEADER_FIELD_LINE_NUMBER=1
FIELD_DELIMITER=,

Can you give an example of your csv, inputs.conf and props.conf?

[sim_tcr]

It did not work

inputs.conf

[monitor://L:\Logs\csv\*.csv]
sourcetype = uow_fitcap_csv
index = sandbox
disabled = false
crcSalt=<SOURCE>

outputs.conf

[ uow_fitcap_csv  ]
CHARSET=AUTO
INDEXED_EXTRACTIONS=csv
KV_MODE=none
SHOULD_LINEMERGE=false
category=Structured
disabled=false
pulldown_type=true
HEADER_FIELD_LINE_NUMBER=1
FIELD_DELIMITER=,

[PPape]

wich version of splunk are you running?

and is it your outputs.conf or props.conf?

[sim_tcr]

we are on 6.3.3
mistakenly said outputs.conf, it is props.conf

[sundareshr]

Try this

[ csv ]
SHOULD_LINEMERGE=false
INDEXED_EXTRACTIONS=csv
TIMESTAMP_FIELDS=rundate,time