Timestamp not being recognized in CSV

[skgbanga]

Searching for the other answer, I believe this is one of the most common questions, but I couldn't figure out my answer after reading all the others. So here it goes:

My CSV looks like this:

trig.hwts,delta                    
1517492760549648185,58445

And after setting the options via splunk web interface, here is what my props.conf looks like:

[csv]
DATETIME_CONFIG = 
NO_BINARY_CHECK = true
TIMESTAMP_FIELDS = trig_hwts               
disabled = false                           
TIME_FORMAT = %s%9N                        
TZ = America/Detroit

Note that I have specified TIMESTAMP_FIELDS as trig_hwts (_ instead of the .) because that is how splunk recognized this field. (I have tried changing it to . as well, but that also didn't work)

The problem is splunk is not picking trig.hwts at all. It keeps showing the time when I uploaded my file to splunk, and not the time in the file. I have restarted my splunk after editing file.

Any help is appreciated.

[harsmarvania57]

Have you applied this configuration on Splunk Indexer or Splunk Search Head ? This configuration should go to Splunk Indexer or Heavy Forwarder whichever comes first.

I am trying to upload data which you have provided in my lab in standalone instance and it is working fine apart from TZ & DATETIME_CONFIG setting. Can you please remove DATETIME_CONFIG = from your configuration and restart splunk ?

[skgbanga]

Hey,

I have applied the configuration via splunk web interface. (Settings -> Source Types -> csv -> TimeStamp). Could you point me where I can change the configurations you specified.

I am on mac os. So I went to /Applications/Splunk/etc/system/local/props.conf and removed the DATETIME_CONFIG =, but that didn't change anything.

[skgbanga]

Also note that I followed these instructions to uploaded the data:
http://docs.splunk.com/Documentation/Splunk/7.0.2/SearchTutorial/GetthetutorialdataintoSplunk

[harsmarvania57]

After changes have you restarted splunk ?

[skgbanga]

Yes. How are you adding the data and changing the configuration? Maybe I can try to follow your exact steps and then report back?

[harsmarvania57]

Steps are:

1.) Create test1.csv file on desktop.
2.) Goto Settings - Add Data -> Upload
3.) Select test.csv from your desktop then Click Next
4.) Now you can see that Sourcetype is selected as csv
5.) Click on Timestamp on left hand side which will expand timestamp setting, then select Extraction as Advanced. Now Timezone select timezone America/Detroit , Timestamp format %s%9N , Timestamp fields trig_hwts
6.) After this settings you can see that on right side window, time is recognized correctly.
Then click Save As and then Next, Next ....

[skgbanga]

Hey,

So any new data that I am adding is being correctly interpreted now. (I tried two ways: .csv and .zip (consisting of a lot of csvs)

However changing the settings does NOT affect the previous data. Maybe I need to reindex the data or something (but I don't know how to do that)

[harsmarvania57]

Data which is already indexed will not alter. So you need to reindex data, easiet way is remove whole fishbucket if you are monitoring only these CSV files and you want to reindex them all.

Then
1.) Stop splunk on universal forwarder.
2.) Remove $SPLUNK_HOME/var/lib/splunk/fishbucket directory.
3.) Start splunk on universal forwarder.

This will reindex all data which you have configured on your universal forwarder.

If you want to reindex specific file then follow below steps.
1.) Stop splunk on universal forwarder.
2.) Run below command to remove file from fishbucket so that it will reindex again. In below example I have used /var/tmp/test1.csv but you need to change this with your actual file name with absolute file path.

$SPLUNK_HOME/bin/splunk cmd btprobe -d $SPLUNK_HOME/var/lib/splunk/fishbucket/splunk_private_db --file /var/tmp/test1.csv --reset

3.) Start splunk on universal forwarder.

[493669]

what is your field name in csvtrig.hwts or trig_hwts

[skgbanga]

The name in my csv is trig.hwts, but splunks changes that to trig_hwts as the field name it recognized.