Getting Data In

Problems with CSV timestamps

New Member

Hi, I have some csv files on my Splunk index. The files are named with a date like xxxxx20180703.csv . In the csv files there is a field with a time in 12:30:45 PM format. The timestamp is able to pickup the date and time. However I have an issues where on some of the files(not all) it detects 11pm properly but then it treats 12 AM as the next day and any time after that will be labeled as the next day as well.

0 Karma

Splunk Employee
Splunk Employee

You could use a custom datetime.xml file and reference it from your props.conf file. Below is an example I used where I had a similar issue. The data I was working with had a time in the data, but the date was in the file name. My filenames looked something like the following:

filename20180703.txt

I just copied and pasted an existing definition that was similar and tweaked it in. You need to change the name, the extract as well if the order is different, and the regex to extract the values.

<define name="_masheddate3" extract="year, month, day">
    <text><![CDATA[source::.*?/sampledata/\w+(\d{4})_(\d{2})_(\d{2})\.txt]]></text>
</define>
0 Karma

New Member

Splunk have no problem reading the date and time. My problem is that for one file, Splunk reads 11pm and treats any events pass midnight as the next day and not the same day. It only mysteriously happens for one csv file and the rest are read perfectly.

0 Karma

SplunkTrust
SplunkTrust

Would the following help?
https://answers.splunk.com/answers/557841/how-to-extract-date-from-filename-and-add-it-with.html

Have you defined TIME_FORMAT as per your time field in the csv file field?

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

SplunkTrust
SplunkTrust

It would be helpful to see some sample events.
What are the props.conf settings for that sourcetype?

---
If this reply helps you, an upvote would be appreciated.
0 Karma

New Member

I've used the csv sourcetype. It is able to correctly read the date and time for most of the files. It is just the first file I am having issue with for some reason.

0 Karma

SplunkTrust
SplunkTrust

What is different about the first file?

---
If this reply helps you, an upvote would be appreciated.
0 Karma