topic Search Timestamp in Splunk Search

Search Timestamp

simon_pytches — Tue, 02 Oct 2012 17:24:38 GMT

I'm having problems with a remote file import using a forwarder, where the file time date stamp is in UK format dd/mm/yyyy 17:00:00 and the first field in the CSV is also dd/mm/yyy 17:00:00 but the first entry for all items in splunk is in mm/dd/yyyy format.

Problem is that when I do a search on this csv in splunk, the file date stamps are not indexing correctly. See below. Todays entries are coming out of the splunk index as February. (csv file only now contains entries for today in it)

10/02/2012 17:32:14.000 02/10/2012 17:32:14,fred.blogs,,,,,,,,,,,

First column in the search is incorrect and the date splunk seems to be indexing on. Second date is correct and as per the CSV file.

I've seen some posts that talk about changing the prop.conf file here's what i've added to the CSV section of e:\Program File\Splunk\etc\system\default\prop.conf

--------------------cut-----------------------

# NON-LOG FILES

[source::....(jar)(.\d+)?]
sourcetype = source_archive

[source::....csv]
sourcetype = csv
TIME_FORMAT=%m/%d/%Y %H:%M:%S

--------------------cut-----------------------

This hasn't fixed my problem. So I've either edited the wrong file or I've added the wrong format info or both.

Any body know how I can fix this issue? I'm stumped.

The confusing thing is that this was working when the csv had loads of data in it, going back several months. I was about to go into production so I flattened all the logs to start with clean data using

.\splunk clean

All my files now only have from today in them.

My other log files from syslog and ais are working fine it's just this csv that's causing problems. I guess there isn't enough data in the new file for the system to auto detect correct date format.

Cheers

Simon

Re: Search Timestamp

emiller42 — Tue, 02 Oct 2012 19:44:25 GMT

That TIME_FORMAT setting should have been placed in props.conf on your indexer. Is that the case?

Re: Search Timestamp

lguinn2 — Tue, 02 Oct 2012 22:40:18 GMT

Your TIME_FORMAT says that the CSV is in month-day-year format! I think you want

TIME_FORMAT=%d/%m/%Y %H:%M:%S

Re: Search Timestamp

simon_pytches — Wed, 03 Oct 2012 10:28:13 GMT

Oops thanks, good spot that's what happens when you spend too long staring at something. I've changed it as above but still get same thing. Have tried restarting splunk. Still appears the same in the search (thinks today transactions are March)

I'm not sure this is the correct file or section in the prop.conf or even if I've edited the correct prop.conf instance. The file I changed was on the Splunk server (not the forwarder) under

e:\Program Files\Splunk\etc\system\default\prop.conf

Unless I need to re-index somehow :s

Cheers

Simon

Re: Search Timestamp

Ayn — Wed, 03 Oct 2012 11:02:28 GMT

The file should be called "props.conf", not "prop.conf".

Also these settings are applied at index-time, so any changes you make will NOT have any effect on data that has already been indexed.