Getting Data In

Data Input problem, csv files are not seeing time field and sometimes not indexing entire file.

scottrunyon
Contributor

I have set up a directory on a Windows system to be monitored by a UF. Two csv files are created every night and are getting indexed. However, the timestamp is the time the file is created, not the time that is in the "Timestamp Fields" parameter.

The first line of my csv file is -
Event,Door,Side,First name,Last name,Picture,Credential,Supplemental credential,Event timestamp,Credential code,Card format

Event timestamp is in this format 4/15/2017 3:45:15 PM

The defined parameters under source type are Catagory - Structured, Indexed Extractions -csv, Extraction - Advanced, Timestamp fields

- Event timestamp. All others are set to default.

props.conf contains -
[logs]
category = Structured
pulldown_type = 1
DATETIME_CONFIG =
HEADER_FIELD_LINE_NUMBER =
INDEXED_EXTRACTIONS = csv
NO_BINARY_CHECK = true
TIMESTAMP_FIELDS = Event timestamp
description = Door log
disabled = false
FIELD_QUOTE = '

The second problem is that not all lines of the file not be indexed. I cannot find any parameter that would restrict the size of a file to be indexed.

0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi scottrunyon,
in TIMESTAMP_FIELDS I see a space between words, so Splunk cannot recognize the field name, use double quotes or change the field name (if possible it's better!).
Remember that to index csv files, you have to put props.conf file both on Indexers and Forwarders.
Bye.
Giuseppe

View solution in original post

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi scottrunyon,
in TIMESTAMP_FIELDS I see a space between words, so Splunk cannot recognize the field name, use double quotes or change the field name (if possible it's better!).
Remember that to index csv files, you have to put props.conf file both on Indexers and Forwarders.
Bye.
Giuseppe

0 Karma

scottrunyon
Contributor

Giuseppe,

It looks like that is pulling in data now. However, the time in the "Event timestamp" field is not being indexed correctly. The entry in the "Event timestamp" field data is in this format - 4/17/2017 12:05:28 PM or 4/17/2017 2:27:43 PM. When I run a query against the record, the indexed data shows as correct but the _time field is incorrect. Spunk shows

csv entry Indexed entry _time

4/13/2017 5:57 4/13/2017 5:57:00 AM 2017-04-13T05:57:00.000-05:00
4/13/2017 15:01 4/13/2017 3:01:10 PM 2017-04-13T03:01:00.000-05:00

What is happening is that it is not converting the 24 hour clock correctly. I tried to modifying the timestamp format in the prop.conf file to %m/%d/%Y %H:%M or %m/%d%Y %I:%M:%S %p but nothing changed.

Any help would be appreciated.

Scott

0 Karma

gcusello
SplunkTrust
SplunkTrust

the easiest way to proceed is to download a copy of your csv and try to load it using the web interface [Settings -- Add Data].
In this way you can verify on the fly if your props.conf correctly runs and all the configurations: you can set time format and all the parameters that will enter in your props.conf.
At the end you can load the file in a test index or not to load, but the important thing is to define and save the correct props.conf.
Remember that once defined props.conf, you have to copy it both on Indexers and Forwarders.
Bye.
Giuseppe

0 Karma

scottrunyon
Contributor

I had the group that issues the report that is uploaded to Splunk move the timestamp column to be the first column. So far, Splunk is seeing the correct date/time.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...