Getting Data In
Highlighted

Data Input problem, csv files are not seeing time field and sometimes not indexing entire file.

Contributor

I have set up a directory on a Windows system to be monitored by a UF. Two csv files are created every night and are getting indexed. However, the timestamp is the time the file is created, not the time that is in the "Timestamp Fields" parameter.

The first line of my csv file is -
Event,Door,Side,First name,Last name,Picture,Credential,Supplemental credential,Event timestamp,Credential code,Card format

Event timestamp is in this format 4/15/2017 3:45:15 PM

The defined parameters under source type are Catagory - Structured, Indexed Extractions -csv, Extraction - Advanced, Timestamp fields

- Event timestamp. All others are set to default.

props.conf contains -
[logs]
category = Structured
pulldowntype = 1
DATETIME
CONFIG =
HEADERFIELDLINENUMBER =
INDEXED
EXTRACTIONS = csv
NOBINARYCHECK = true
TIMESTAMPFIELDS = Event timestamp
description = Door log
disabled = false
FIELD
QUOTE = '

The second problem is that not all lines of the file not be indexed. I cannot find any parameter that would restrict the size of a file to be indexed.

0 Karma
Highlighted

Re: Data Input problem, csv files are not seeing time field and sometimes not indexing entire file.

Legend

Hi scottrunyon,
in TIMESTAMP_FIELDS I see a space between words, so Splunk cannot recognize the field name, use double quotes or change the field name (if possible it's better!).
Remember that to index csv files, you have to put props.conf file both on Indexers and Forwarders.
Bye.
Giuseppe

View solution in original post

0 Karma
Highlighted

Re: Data Input problem, csv files are not seeing time field and sometimes not indexing entire file.

Contributor

Giuseppe,

It looks like that is pulling in data now. However, the time in the "Event timestamp" field is not being indexed correctly. The entry in the "Event timestamp" field data is in this format - 4/17/2017 12:05:28 PM or 4/17/2017 2:27:43 PM. When I run a query against the record, the indexed data shows as correct but the _time field is incorrect. Spunk shows

csv entry Indexed entry _time

4/13/2017 5:57 4/13/2017 5:57:00 AM 2017-04-13T05:57:00.000-05:00
4/13/2017 15:01 4/13/2017 3:01:10 PM 2017-04-13T03:01:00.000-05:00

What is happening is that it is not converting the 24 hour clock correctly. I tried to modifying the timestamp format in the prop.conf file to %m/%d/%Y %H:%M or %m/%d%Y %I:%M:%S %p but nothing changed.

Any help would be appreciated.

Scott

0 Karma
Highlighted

Re: Data Input problem, csv files are not seeing time field and sometimes not indexing entire file.

Legend

the easiest way to proceed is to download a copy of your csv and try to load it using the web interface [Settings -- Add Data].
In this way you can verify on the fly if your props.conf correctly runs and all the configurations: you can set time format and all the parameters that will enter in your props.conf.
At the end you can load the file in a test index or not to load, but the important thing is to define and save the correct props.conf.
Remember that once defined props.conf, you have to copy it both on Indexers and Forwarders.
Bye.
Giuseppe

0 Karma
Highlighted

Re: Data Input problem, csv files are not seeing time field and sometimes not indexing entire file.

Contributor

I had the group that issues the report that is uploaded to Splunk move the timestamp column to be the first column. So far, Splunk is seeing the correct date/time.

0 Karma