I have set up a directory on a Windows system to be monitored by a UF. Two csv files are created every night and are getting indexed. However, the timestamp is the time the file is created, not the time that is in the "Timestamp Fields" parameter.
The first line of my csv file is -
Event,Door,Side,First name,Last name,Picture,Credential,Supplemental credential,Event timestamp,Credential code,Card format
Event timestamp is in this format 4/15/2017 3:45:15 PM
The defined parameters under source type are Catagory - Structured, Indexed Extractions -csv, Extraction - Advanced, Timestamp fields
props.conf contains -
category = Structured
pulldowntype = 1
INDEXEDEXTRACTIONS = csv
NOBINARYCHECK = true
TIMESTAMPFIELDS = Event timestamp
description = Door log
disabled = false
FIELDQUOTE = '
The second problem is that not all lines of the file not be indexed. I cannot find any parameter that would restrict the size of a file to be indexed.
in TIMESTAMP_FIELDS I see a space between words, so Splunk cannot recognize the field name, use double quotes or change the field name (if possible it's better!).
Remember that to index csv files, you have to put props.conf file both on Indexers and Forwarders.
It looks like that is pulling in data now. However, the time in the "Event timestamp" field is not being indexed correctly. The entry in the "Event timestamp" field data is in this format - 4/17/2017 12:05:28 PM or 4/17/2017 2:27:43 PM. When I run a query against the record, the indexed data shows as correct but the _time field is incorrect. Spunk shows
csv entry Indexed entry _time
4/13/2017 5:57 4/13/2017 5:57:00 AM 2017-04-13T05:57:00.000-05:00
4/13/2017 15:01 4/13/2017 3:01:10 PM 2017-04-13T03:01:00.000-05:00
What is happening is that it is not converting the 24 hour clock correctly. I tried to modifying the timestamp format in the prop.conf file to %m/%d/%Y %H:%M or %m/%d%Y %I:%M:%S %p but nothing changed.
Any help would be appreciated.
the easiest way to proceed is to download a copy of your csv and try to load it using the web interface [Settings -- Add Data].
In this way you can verify on the fly if your props.conf correctly runs and all the configurations: you can set time format and all the parameters that will enter in your props.conf.
At the end you can load the file in a test index or not to load, but the important thing is to define and save the correct props.conf.
Remember that once defined props.conf, you have to copy it both on Indexers and Forwarders.
I had the group that issues the report that is uploaded to Splunk move the timestamp column to be the first column. So far, Splunk is seeing the correct date/time.