Hello Team,
I am using a python script input method to extract data (meta-data and some raw data) from binary and txt files. My script is working fine, I can search extracted data from Splunk's search app.
Issue:- This script has 'ctime' field, I am trying to match this field to splunk timestamp or _time field but the matching is not working. I am getting timestamp field at Splunk end with NONE value but this suppose to be either ctime field value.
Following is the detail of configurations and data-set:-
1) props.conf:-
[waterwise]
SHOULD_LINEMERGE = false
CHARSET = UTF-8
LINE_BREAKER = ([\r\n])
TIMESTAMP_FIELDS = ctime
TIME_FORMAT = %m-%d-%Y %H:%M:%S %z
2) inputs.conf:-
[script://$SPLUNK_HOME\bin\scripts\waterwise-data.py]
disabled = false
interval = 300
source = waterwise
sourcetype = waterwise
index = main
3) Script out-put data-set (multi key value pair):-
sensor_data=361, sensor_data=361, sensor_data=360,location=Z:\Documents\rdms-data-test\waterwise\FC_PH\stn_11\data\2011\001\BTRY\STN_11_2011001232949_BTRY_600_15.txt, size=164, atime=06-04-2014 04:51:32, mtime=01-01-2011 15:40:14, ctime=01-01-2011 15:40:14 China Standard Time, project=waterwise, system=FC_PH, station=stn_11, year=2011, day_of_year=001, sensor=BTRY
sensor_data=361, sensor_data=361, sensor_data=361, location=Z:\Documents\rdms-data-test\waterwise\FC_PH\stn_11\data\2011\001\BTRY\STN_11_2011001234012_BTRY_600_15.txt, size=164, atime=06-04-2014 04:51:32, mtime=01-01-2011 15:50:42, ctime=01-01-2011 15:50:42 China Standard Time, project=waterwise, system=FC_PH, station=stn_11, year=2011, day_of_year=001, sensor=BTRY
sensor_data=359, sensor_data=359, location=Z:\Documents\rdms-data-test\waterwise\FC_PH\stn_11\data\2011\001\BTRY\STN_11_2011001235034_BTRY_600_15.txt, size=164, atime=06-04-2014 04:51:32, mtime=01-01-2011 16:01:01, ctime=01-01-2011 16:01:01 China Standard Time, project=waterwise, system=FC_PH, station=stn_11, year=2011, day_of_year=001, sensor=BTRY
location=Z:\Documents\rdms-data-test\waterwise\FC_PH\stn_11\data\2011\001\ORP\STN_11_2011001000056_ORP_SGT.raw, size=52, atime=06-03-2014 07:31:55, mtime=12-31-2010 16:01:32, ctime=12-31-2010 16:01:32 China Standard Time, project=waterwise, system=FC_PH, station=stn_11, year=2011, day_of_year=001, sensor=ORP
location=Z:\Documents\rdms-data-test\waterwise\FC_PH\stn_11\data\2011\001\ORP\STN_11_2011001000156_ORP_SGT.raw, size=52, atime=06-03-2014 07:31:55, mtime=12-31-2010 16:02:32, ctime=12-31-2010 16:02:32 China Standard Time, project=waterwise, system=FC_PH, station=stn_11, year=2011, day_of_year=001, sensor=ORP
Thanks in advance. I look forward to hearing from you team.
--
Rupesh
Based on the props.conf spec it doesn't look like the TIMESTAMP_FIELDS is the correct declaration for you to use based on the sample output.
Further, it looks like the time stamp (crime) in the output is in the middle of the event. Again by the spec, Splunk will only examine the first 150 characters by default.
I would try adding TIME_PREFIX and MAX_TIMESTAMP_LOOKAHEAD values to props.conf.
http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf
Hope that helps
Thank you Alterdego for your reply. There was problem with time zone.
I think the problem is time zone, "China Standard Time" is not the correct format of TZ , you can set TZ=CN or TZ=Asia/Shanghai in props.conf
for example :
TIME_FORMAT=%m-%d-%Y %H:%M:%S
TIME_PREFIX=ctime=
TZ=CN
Thank you Dmlee, yes 'TZ=CN' works for me.