Script Input - Timestamp field extraction from multi key value pair data-set (props.conf, inputs.conf)

[rupesh_kumar]

Hello Team,

I am using a python script input method to extract data (meta-data and some raw data) from binary and txt files. My script is working fine, I can search extracted data from Splunk's search app.

Issue:- This script has 'ctime' field, I am trying to match this field to splunk timestamp or _time field but the matching is not working. I am getting timestamp field at Splunk end with NONE value but this suppose to be either ctime field value.

Following is the detail of configurations and data-set:-

1) props.conf:-

[waterwise]
SHOULD_LINEMERGE = false
CHARSET = UTF-8
LINE_BREAKER = ([\r\n])
TIMESTAMP_FIELDS = ctime
TIME_FORMAT = %m-%d-%Y %H:%M:%S %z

2) inputs.conf:-

[script://$SPLUNK_HOME\bin\scripts\waterwise-data.py]
disabled = false
interval = 300
source = waterwise
sourcetype = waterwise
index = main

3) Script out-put data-set (multi key value pair):-

sensor_data=361, sensor_data=361, sensor_data=360,location=Z:\Documents\rdms-data-test\waterwise\FC_PH\stn_11\data\2011\001\BTRY\STN_11_2011001232949_BTRY_600_15.txt, size=164, atime=06-04-2014 04:51:32, mtime=01-01-2011 15:40:14, ctime=01-01-2011 15:40:14 China Standard Time, project=waterwise, system=FC_PH, station=stn_11, year=2011, day_of_year=001, sensor=BTRY

sensor_data=361, sensor_data=361, sensor_data=361, location=Z:\Documents\rdms-data-test\waterwise\FC_PH\stn_11\data\2011\001\BTRY\STN_11_2011001234012_BTRY_600_15.txt, size=164, atime=06-04-2014 04:51:32, mtime=01-01-2011 15:50:42, ctime=01-01-2011 15:50:42 China Standard Time, project=waterwise, system=FC_PH, station=stn_11, year=2011, day_of_year=001, sensor=BTRY

sensor_data=359, sensor_data=359, location=Z:\Documents\rdms-data-test\waterwise\FC_PH\stn_11\data\2011\001\BTRY\STN_11_2011001235034_BTRY_600_15.txt, size=164, atime=06-04-2014 04:51:32, mtime=01-01-2011 16:01:01, ctime=01-01-2011 16:01:01 China Standard Time, project=waterwise, system=FC_PH, station=stn_11, year=2011, day_of_year=001, sensor=BTRY

location=Z:\Documents\rdms-data-test\waterwise\FC_PH\stn_11\data\2011\001\ORP\STN_11_2011001000056_ORP_SGT.raw, size=52, atime=06-03-2014 07:31:55, mtime=12-31-2010 16:01:32, ctime=12-31-2010 16:01:32 China Standard Time, project=waterwise, system=FC_PH, station=stn_11, year=2011, day_of_year=001, sensor=ORP

location=Z:\Documents\rdms-data-test\waterwise\FC_PH\stn_11\data\2011\001\ORP\STN_11_2011001000156_ORP_SGT.raw, size=52, atime=06-03-2014 07:31:55, mtime=12-31-2010 16:02:32, ctime=12-31-2010 16:02:32 China Standard Time, project=waterwise, system=FC_PH, station=stn_11, year=2011, day_of_year=001, sensor=ORP

Thanks in advance. I look forward to hearing from you team.

--

Rupesh

[alterdego]

Based on the props.conf spec it doesn't look like the TIMESTAMP_FIELDS is the correct declaration for you to use based on the sample output.
Further, it looks like the time stamp (crime) in the output is in the middle of the event. Again by the spec, Splunk will only examine the first 150 characters by default.
I would try adding TIME_PREFIX and MAX_TIMESTAMP_LOOKAHEAD values to props.conf.

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf

Hope that helps

[rupesh_kumar]

Thank you Alterdego for your reply. There was problem with time zone.

[dmlee]

I think the problem is time zone, "China Standard Time" is not the correct format of TZ , you can set TZ=CN or TZ=Asia/Shanghai in props.conf

for example :

TIME_FORMAT=%m-%d-%Y %H:%M:%S 
TIME_PREFIX=ctime= 
TZ=CN

[rupesh_kumar]

Thank you Dmlee, yes 'TZ=CN' works for me.